Preface
Prev Up PostgreSQL 14.4 Documentation Home Next

Preface

Table of Contents

1. What Is PostgreSQL?
2. A Brief History of PostgreSQL
2.1. The Berkeley POSTGRES Project
2.2. Postgres95
2.3. PostgreSQL
3. Conventions
4. Further Information
5. Bug Reporting Guidelines
5.1. Identifying Bugs
5.2. What to Report
5.3. Where to Report Bugs

This book is the official documentation of PostgreSQL. It has been written by the PostgreSQL developers and other volunteers in parallel to the development of the PostgreSQL software. It describes all the functionality that the current version of PostgreSQL officially supports.

To make the large amount of information about PostgreSQL manageable, this book has been organized in several parts. Each part is targeted at a different class of users, or at users in different stages of their PostgreSQL experience:

  • Part I is an informal introduction for new users.

  • Part II documents the SQL query language environment, including data types and functions, as well as user-level performance tuning. Every PostgreSQL user should read this.

  • Part III describes the installation and administration of the server. Everyone who runs a PostgreSQL server, be it for private use or for others, should read this part.

  • Part IV describes the programming interfaces for PostgreSQL client programs.

  • Part V contains information for advanced users about the extensibility capabilities of the server. Topics include user-defined data types and functions.

  • Part VI contains reference information about SQL commands, client and server programs. This part supports the other parts with structured information sorted by command or program.

  • Part VII contains assorted information that might be of use to PostgreSQL developers.

Prev | Up | Next
PostgreSQL 14.4 Documentation | Home | 1. What Is PostgreSQL?


1. What Is PostgreSQL?
Prev Up Preface Home Next

1. What Is PostgreSQL?

PostgreSQL is an object-relational database management system (ORDBMS) based on POSTGRES, Version 4.2, developed at the University of California at Berkeley Computer Science Department. POSTGRES pioneered many concepts that only became available in some commercial database systems much later.

PostgreSQL is an open-source descendant of this original Berkeley code. It supports a large part of the SQL standard and offers many modern features:

  • complex queries
  • foreign keys
  • triggers
  • updatable views
  • transactional integrity
  • multiversion concurrency control

Also, PostgreSQL can be extended by the user in many ways, for example by adding new

  • data types
  • functions
  • operators
  • aggregate functions
  • index methods
  • procedural languages

And because of the liberal license, PostgreSQL can be used, modified, and distributed by anyone free of charge for any purpose, be it private, commercial, or academic.

Prev | Up | Next
Preface | Home | 2. A Brief History of PostgreSQL


2. A Brief History of PostgreSQL
Prev Up Preface Home Next

2. A Brief History of PostgreSQL

2.1. The Berkeley POSTGRES Project
2.2. Postgres95
2.3. PostgreSQL

The object-relational database management system now known as PostgreSQL is derived from the POSTGRES package written at the University of California at Berkeley. With over two decades of development behind it, PostgreSQL is now the most advanced open-source database available anywhere.

2.1. The Berkeley POSTGRES Project

The POSTGRES project, led by Professor Michael Stonebraker, was sponsored by the Defense Advanced Research Projects Agency (DARPA), the Army Research Office (ARO), the National Science Foundation (NSF), and ESL, Inc. The implementation of POSTGRES began in 1986. The initial concepts for the system were presented in [ston86], and the definition of the initial data model appeared in [rowe87]. The design of the rule system at that time was described in [ston87a]. The rationale and architecture of the storage manager were detailed in [ston87b].

POSTGRES has undergone several major releases since then. The first “demoware” system became operational in 1987 and was shown at the 1988 ACM-SIGMOD Conference. Version 1, described in [ston90a], was released to a few external users in June 1989. In response to a critique of the first rule system ([ston89]), the rule system was redesigned ([ston90b]), and Version 2 was released in June 1990 with the new rule system. Version 3 appeared in 1991 and added support for multiple storage managers, an improved query executor, and a rewritten rule system. For the most part, subsequent releases until Postgres95 (see below) focused on portability and reliability.

POSTGRES has been used to implement many different research and production applications. These include: a financial data analysis system, a jet engine performance monitoring package, an asteroid tracking database, a medical information database, and several geographic information systems. POSTGRES has also been used as an educational tool at several universities. Finally, Illustra Information Technologies (later merged into Informix, which is now owned by IBM) picked up the code and commercialized it. In late 1992, POSTGRES became the primary data manager for the Sequoia 2000 scientific computing project.

The size of the external user community nearly doubled during 1993. It became increasingly obvious that maintenance of the prototype code and support was taking up large amounts of time that should have been devoted to database research. In an effort to reduce this support burden, the Berkeley POSTGRES project officially ended with Version 4.2.

2.2. Postgres95

In 1994, Andrew Yu and Jolly Chen added an SQL language interpreter to POSTGRES. Under a new name, Postgres95 was subsequently released to the web to find its own way in the world as an open-source descendant of the original POSTGRES Berkeley code.

Postgres95 code was completely ANSI C and trimmed in size by 25%. Many internal changes improved performance and maintainability. Postgres95 release 1.0.x ran about 30–50% faster on the Wisconsin Benchmark compared to POSTGRES, Version 4.2. Apart from bug fixes, the following were the major enhancements:

  • The query language PostQUEL was replaced with SQL (implemented in the server). (Interface library libpq was named after PostQUEL.) Subqueries were not supported until PostgreSQL (see below), but they could be imitated in Postgres95 with user-defined SQL functions. Aggregate functions were re-implemented. Support for the GROUP BY query clause was also added.

  • A new program (psql) was provided for interactive SQL queries, which used GNU Readline. This largely superseded the old monitor program.

  • A new front-end library, libpgtcl, supported Tcl-based clients. A sample shell, pgtclsh, provided new Tcl commands to interface Tcl programs with the Postgres95 server.

  • The large-object interface was overhauled. The inversion large objects were the only mechanism for storing large objects. (The inversion file system was removed.)

  • The instance-level rule system was removed. Rules were still available as rewrite rules.

  • A short tutorial introducing regular SQL features as well as those of Postgres95 was distributed with the source code

  • GNU make (instead of BSD make) was used for the build. Also, Postgres95 could be compiled with an unpatched GCC (data alignment of doubles was fixed).

2.3. PostgreSQL

By 1996, it became clear that the name “Postgres95” would not stand the test of time. We chose a new name, PostgreSQL, to reflect the relationship between the original POSTGRES and the more recent versions with SQL capability. At the same time, we set the version numbering to start at 6.0, putting the numbers back into the sequence originally begun by the Berkeley POSTGRES project.

Many people continue to refer to PostgreSQL as “Postgres” (now rarely in all capital letters) because of tradition or because it is easier to pronounce. This usage is widely accepted as a nickname or alias.

The emphasis during development of Postgres95 was on identifying and understanding existing problems in the server code. With PostgreSQL, the emphasis has shifted to augmenting features and capabilities, although work continues in all areas.

Details about what has happened in PostgreSQL since then can be found in Appendix E.

Prev | Up | Next
1. What Is PostgreSQL? | Home | 3. Conventions


3. Conventions
Prev Up Preface Home Next

3. Conventions

The following conventions are used in the synopsis of a command: brackets ([ and ]) indicate optional parts. Braces ({ and }) and vertical lines (|) indicate that you must choose one alternative. Dots (...) mean that the preceding element can be repeated. All other symbols, including parentheses, should be taken literally.

Where it enhances the clarity, SQL commands are preceded by the prompt =>, and shell commands are preceded by the prompt $. Normally, prompts are not shown, though.

An administrator is generally a person who is in charge of installing and running the server. A user could be anyone who is using, or wants to use, any part of the PostgreSQL system. These terms should not be interpreted too narrowly; this book does not have fixed presumptions about system administration procedures.

Prev | Up | Next
2. A Brief History of PostgreSQL | Home | 4. Further Information


5. Bug Reporting Guidelines
Prev Up Preface Home Next

5. Bug Reporting Guidelines

5.1. Identifying Bugs
5.2. What to Report
5.3. Where to Report Bugs

When you find a bug in PostgreSQL we want to hear about it. Your bug reports play an important part in making PostgreSQL more reliable because even the utmost care cannot guarantee that every part of PostgreSQL will work on every platform under every circumstance.

The following suggestions are intended to assist you in forming bug reports that can be handled in an effective fashion. No one is required to follow them but doing so tends to be to everyone's advantage.

We cannot promise to fix every bug right away. If the bug is obvious, critical, or affects a lot of users, chances are good that someone will look into it. It could also happen that we tell you to update to a newer version to see if the bug happens there. Or we might decide that the bug cannot be fixed before some major rewrite we might be planning is done. Or perhaps it is simply too hard and there are more important things on the agenda. If you need help immediately, consider obtaining a commercial support contract.

5.1. Identifying Bugs

Before you report a bug, please read and re-read the documentation to verify that you can really do whatever it is you are trying. If it is not clear from the documentation whether you can do something or not, please report that too; it is a bug in the documentation. If it turns out that a program does something different from what the documentation says, that is a bug. That might include, but is not limited to, the following circumstances:

  • A program terminates with a fatal signal or an operating system error message that would point to a problem in the program. (A counterexample might be a “disk full” message, since you have to fix that yourself.)

  • A program produces the wrong output for any given input.

  • A program refuses to accept valid input (as defined in the documentation).

  • A program accepts invalid input without a notice or error message. But keep in mind that your idea of invalid input might be our idea of an extension or compatibility with traditional practice.

  • PostgreSQL fails to compile, build, or install according to the instructions on supported platforms.

Here “program” refers to any executable, not only the backend process.

Being slow or resource-hogging is not necessarily a bug. Read the documentation or ask on one of the mailing lists for help in tuning your applications. Failing to comply to the SQL standard is not necessarily a bug either, unless compliance for the specific feature is explicitly claimed.

Before you continue, check on the TODO list and in the FAQ to see if your bug is already known. If you cannot decode the information on the TODO list, report your problem. The least we can do is make the TODO list clearer.

5.2. What to Report

The most important thing to remember about bug reporting is to state all the facts and only facts. Do not speculate what you think went wrong, what “it seemed to do”, or which part of the program has a fault. If you are not familiar with the implementation you would probably guess wrong and not help us a bit. And even if you are, educated explanations are a great supplement to but no substitute for facts. If we are going to fix the bug we still have to see it happen for ourselves first. Reporting the bare facts is relatively straightforward (you can probably copy and paste them from the screen) but all too often important details are left out because someone thought it does not matter or the report would be understood anyway.

The following items should be contained in every bug report:

  • The exact sequence of steps from program start-up necessary to reproduce the problem. This should be self-contained; it is not enough to send in a bare SELECT statement without the preceding CREATE TABLE and INSERT statements, if the output should depend on the data in the tables. We do not have the time to reverse-engineer your database schema, and if we are supposed to make up our own data we would probably miss the problem.

    The best format for a test case for SQL-related problems is a file that can be run through the psql frontend that shows the problem. (Be sure to not have anything in your ~/.psqlrc start-up file.) An easy way to create this file is to use pg_dump to dump out the table declarations and data needed to set the scene, then add the problem query. You are encouraged to minimize the size of your example, but this is not absolutely necessary. If the bug is reproducible, we will find it either way.

    If your application uses some other client interface, such as PHP, then please try to isolate the offending queries. We will probably not set up a web server to reproduce your problem. In any case remember to provide the exact input files; do not guess that the problem happens for “large files” or “midsize databases”, etc. since this information is too inexact to be of use.

  • The output you got. Please do not say that it “didn't work” or “crashed”. If there is an error message, show it, even if you do not understand it. If the program terminates with an operating system error, say which. If nothing at all happens, say so. Even if the result of your test case is a program crash or otherwise obvious it might not happen on our platform. The easiest thing is to copy the output from the terminal, if possible.

    Note

    If you are reporting an error message, please obtain the most verbose form of the message. In psql, say \set VERBOSITY verbose beforehand. If you are extracting the message from the server log, set the run-time parameter log_error_verbosity to verbose so that all details are logged.

    Note

    In case of fatal errors, the error message reported by the client might not contain all the information available. Please also look at the log output of the database server. If you do not keep your server's log output, this would be a good time to start doing so.

  • The output you expected is very important to state. If you just write “This command gives me that output.” or “This is not what I expected.”, we might run it ourselves, scan the output, and think it looks OK and is exactly what we expected. We should not have to spend the time to decode the exact semantics behind your commands. Especially refrain from merely saying that “This is not what SQL says/Oracle does.” Digging out the correct behavior from SQL is not a fun undertaking, nor do we all know how all the other relational databases out there behave. (If your problem is a program crash, you can obviously omit this item.)

  • Any command line options and other start-up options, including any relevant environment variables or configuration files that you changed from the default. Again, please provide exact information. If you are using a prepackaged distribution that starts the database server at boot time, you should try to find out how that is done.

  • Anything you did at all differently from the installation instructions.

  • The PostgreSQL version. You can run the command SELECT version(); to find out the version of the server you are connected to. Most executable programs also support a --version option; at least postgres --version and psql --version should work. If the function or the options do not exist then your version is more than old enough to warrant an upgrade. If you run a prepackaged version, such as RPMs, say so, including any subversion the package might have. If you are talking about a Git snapshot, mention that, including the commit hash.

    If your version is older than 14.4 we will almost certainly tell you to upgrade. There are many bug fixes and improvements in each new release, so it is quite possible that a bug you have encountered in an older release of PostgreSQL has already been fixed. We can only provide limited support for sites using older releases of PostgreSQL; if you require more than we can provide, consider acquiring a commercial support contract.

  • Platform information. This includes the kernel name and version, C library, processor, memory information, and so on. In most cases it is sufficient to report the vendor and version, but do not assume everyone knows what exactly “Debian” contains or that everyone runs on x86_64. If you have installation problems then information about the toolchain on your machine (compiler, make, and so on) is also necessary.

Do not be afraid if your bug report becomes rather lengthy. That is a fact of life. It is better to report everything the first time than us having to squeeze the facts out of you. On the other hand, if your input files are huge, it is fair to ask first whether somebody is interested in looking into it. Here is an article that outlines some more tips on reporting bugs.

Do not spend all your time to figure out which changes in the input make the problem go away. This will probably not help solving it. If it turns out that the bug cannot be fixed right away, you will still have time to find and share your work-around. Also, once again, do not waste your time guessing why the bug exists. We will find that out soon enough.

When writing a bug report, please avoid confusing terminology. The software package in total is called “PostgreSQL”, sometimes “Postgres” for short. If you are specifically talking about the backend process, mention that, do not just say “PostgreSQL crashes”. A crash of a single backend process is quite different from crash of the parent “postgres” process; please don't say “the server crashed” when you mean a single backend process went down, nor vice versa. Also, client programs such as the interactive frontend “psql” are completely separate from the backend. Please try to be specific about whether the problem is on the client or server side.

5.3. Where to Report Bugs

In general, send bug reports to the bug report mailing list at <pgsql-bugs@lists.postgresql.org>. You are requested to use a descriptive subject for your email message, perhaps parts of the error message.

Another method is to fill in the bug report web-form available at the project's web site. Entering a bug report this way causes it to be mailed to the <pgsql-bugs@lists.postgresql.org> mailing list.

If your bug report has security implications and you'd prefer that it not become immediately visible in public archives, don't send it to pgsql-bugs. Security issues can be reported privately to <security@postgresql.org>.

Do not send bug reports to any of the user mailing lists, such as <pgsql-sql@lists.postgresql.org> or <pgsql-general@lists.postgresql.org>. These mailing lists are for answering user questions, and their subscribers normally do not wish to receive bug reports. More importantly, they are unlikely to fix them.

Also, please do not send reports to the developers' mailing list <pgsql-hackers@lists.postgresql.org>. This list is for discussing the development of PostgreSQL, and it would be nice if we could keep the bug reports separate. We might choose to take up a discussion about your bug report on pgsql-hackers, if the problem needs more review.

If you have a problem with the documentation, the best place to report it is the documentation mailing list <pgsql-docs@lists.postgresql.org>. Please be specific about what part of the documentation you are unhappy with.

If your bug is a portability problem on a non-supported platform, send mail to <pgsql-hackers@lists.postgresql.org>, so we (and you) can work on porting PostgreSQL to your platform.

Note

Due to the unfortunate amount of spam going around, all of the above lists will be moderated unless you are subscribed. That means there will be some delay before the email is delivered. If you wish to subscribe to the lists, please visit https://lists.postgresql.org/ for instructions.

Prev | Up | Next
4. Further Information | Home | Part I. Tutorial


Chapter 5. Data Definition
Prev Up Part II. The SQL Language Home Next

Chapter 5. Data Definition

Table of Contents

5.1. Table Basics
5.2. Default Values
5.3. Generated Columns
5.4. Constraints
5.4.1. Check Constraints
5.4.2. Not-Null Constraints
5.4.3. Unique Constraints
5.4.4. Primary Keys
5.4.5. Foreign Keys
5.4.6. Exclusion Constraints
5.5. System Columns
5.6. Modifying Tables
5.6.1. Adding a Column
5.6.2. Removing a Column
5.6.3. Adding a Constraint
5.6.4. Removing a Constraint
5.6.5. Changing a Column's Default Value
5.6.6. Changing a Column's Data Type
5.6.7. Renaming a Column
5.6.8. Renaming a Table
5.7. Privileges
5.8. Row Security Policies
5.9. Schemas
5.9.1. Creating a Schema
5.9.2. The Public Schema
5.9.3. The Schema Search Path
5.9.4. Schemas and Privileges
5.9.5. The System Catalog Schema
5.9.6. Usage Patterns
5.9.7. Portability
5.10. Inheritance
5.10.1. Caveats
5.11. Table Partitioning
5.11.1. Overview
5.11.2. Declarative Partitioning
5.11.3. Partitioning Using Inheritance
5.11.4. Partition Pruning
5.11.5. Partitioning and Constraint Exclusion
5.11.6. Best Practices for Declarative Partitioning
5.12. Foreign Data
5.13. Other Database Objects
5.14. Dependency Tracking

This chapter covers how one creates the database structures that will hold one's data. In a relational database, the raw data is stored in tables, so the majority of this chapter is devoted to explaining how tables are created and modified and what features are available to control what data is stored in the tables. Subsequently, we discuss how tables can be organized into schemas, and how privileges can be assigned to tables. Finally, we will briefly look at other features that affect the data storage, such as inheritance, table partitioning, views, functions, and triggers.

Prev | Up | Next
4.3. Calling Functions | Home | 5.1. Table Basics


5.1. Table Basics
Prev Up Chapter 5. Data Definition Home Next

5.1. Table Basics

A table in a relational database is much like a table on paper: It consists of rows and columns. The number and order of the columns is fixed, and each column has a name. The number of rows is variable — it reflects how much data is stored at a given moment. SQL does not make any guarantees about the order of the rows in a table. When a table is read, the rows will appear in an unspecified order, unless sorting is explicitly requested. This is covered in Chapter 7. Furthermore, SQL does not assign unique identifiers to rows, so it is possible to have several completely identical rows in a table. This is a consequence of the mathematical model that underlies SQL but is usually not desirable. Later in this chapter we will see how to deal with this issue.

Each column has a data type. The data type constrains the set of possible values that can be assigned to a column and assigns semantics to the data stored in the column so that it can be used for computations. For instance, a column declared to be of a numerical type will not accept arbitrary text strings, and the data stored in such a column can be used for mathematical computations. By contrast, a column declared to be of a character string type will accept almost any kind of data but it does not lend itself to mathematical calculations, although other operations such as string concatenation are available.

PostgreSQL includes a sizable set of built-in data types that fit many applications. Users can also define their own data types. Most built-in data types have obvious names and semantics, so we defer a detailed explanation to Chapter 8. Some of the frequently used data types are integer for whole numbers, numeric for possibly fractional numbers, text for character strings, date for dates, time for time-of-day values, and timestamp for values containing both date and time.

To create a table, you use the aptly named CREATE TABLE command. In this command you specify at least a name for the new table, the names of the columns and the data type of each column. For example:

CREATE TABLE my_first_table (
    first_column text,
    second_column integer
);

This creates a table named my_first_table with two columns. The first column is named first_column and has a data type of text; the second column has the name second_column and the type integer. The table and column names follow the identifier syntax explained in Section 4.1.1. The type names are usually also identifiers, but there are some exceptions. Note that the column list is comma-separated and surrounded by parentheses.

Of course, the previous example was heavily contrived. Normally, you would give names to your tables and columns that convey what kind of data they store. So let's look at a more realistic example:

CREATE TABLE products (
    product_no integer,
    name text,
    price numeric
);

(The numeric type can store fractional components, as would be typical of monetary amounts.)

Tip

When you create many interrelated tables it is wise to choose a consistent naming pattern for the tables and columns. For instance, there is a choice of using singular or plural nouns for table names, both of which are favored by some theorist or other.

There is a limit on how many columns a table can contain. Depending on the column types, it is between 250 and 1600. However, defining a table with anywhere near this many columns is highly unusual and often a questionable design.

If you no longer need a table, you can remove it using the DROP TABLE command. For example:

DROP TABLE my_first_table;
DROP TABLE products;

Attempting to drop a table that does not exist is an error. Nevertheless, it is common in SQL script files to unconditionally try to drop each table before creating it, ignoring any error messages, so that the script works whether or not the table exists. (If you like, you can use the DROP TABLE IF EXISTS variant to avoid the error messages, but this is not standard SQL.)

If you need to modify a table that already exists, see Section 5.6 later in this chapter.

With the tools discussed so far you can create fully functional tables. The remainder of this chapter is concerned with adding features to the table definition to ensure data integrity, security, or convenience. If you are eager to fill your tables with data now you can skip ahead to Chapter 6 and read the rest of this chapter later.

Prev | Up | Next
Chapter 5. Data Definition | Home | 5.2. Default Values


5.2. Default Values
Prev Up Chapter 5. Data Definition Home Next

5.2. Default Values

A column can be assigned a default value. When a new row is created and no values are specified for some of the columns, those columns will be filled with their respective default values. A data manipulation command can also request explicitly that a column be set to its default value, without having to know what that value is. (Details about data manipulation commands are in Chapter 6.)

If no default value is declared explicitly, the default value is the null value. This usually makes sense because a null value can be considered to represent unknown data.

In a table definition, default values are listed after the column data type. For example:

CREATE TABLE products (
    product_no integer,
    name text,
    price numeric DEFAULT 9.99
);

The default value can be an expression, which will be evaluated whenever the default value is inserted (not when the table is created). A common example is for a timestamp column to have a default of CURRENT_TIMESTAMP, so that it gets set to the time of row insertion. Another common example is generating a “serial number” for each row. In PostgreSQL this is typically done by something like:

CREATE TABLE products (
    product_no integer DEFAULT nextval('products_product_no_seq'),
    ...
);

where the nextval() function supplies successive values from a sequence object (see Section 9.17). This arrangement is sufficiently common that there's a special shorthand for it:

CREATE TABLE products (
    product_no SERIAL,
    ...
);

The SERIAL shorthand is discussed further in Section 8.1.4.

Prev | Up | Next
5.1. Table Basics | Home | 5.3. Generated Columns


5.3. Generated Columns
Prev Up Chapter 5. Data Definition Home Next

5.3. Generated Columns

A generated column is a special column that is always computed from other columns. Thus, it is for columns what a view is for tables. There are two kinds of generated columns: stored and virtual. A stored generated column is computed when it is written (inserted or updated) and occupies storage as if it were a normal column. A virtual generated column occupies no storage and is computed when it is read. Thus, a virtual generated column is similar to a view and a stored generated column is similar to a materialized view (except that it is always updated automatically). PostgreSQL currently implements only stored generated columns.

To create a generated column, use the GENERATED ALWAYS AS clause in CREATE TABLE, for example:

CREATE TABLE people (
    ...,
    height_cm numeric,
    height_in numeric GENERATED ALWAYS AS (height_cm / 2.54) STORED
);

The keyword STORED must be specified to choose the stored kind of generated column. See CREATE TABLE for more details.

A generated column cannot be written to directly. In INSERT or UPDATE commands, a value cannot be specified for a generated column, but the keyword DEFAULT may be specified.

Consider the differences between a column with a default and a generated column. The column default is evaluated once when the row is first inserted if no other value was provided; a generated column is updated whenever the row changes and cannot be overridden. A column default may not refer to other columns of the table; a generation expression would normally do so. A column default can use volatile functions, for example random() or functions referring to the current time; this is not allowed for generated columns.

Several restrictions apply to the definition of generated columns and tables involving generated columns:

  • The generation expression can only use immutable functions and cannot use subqueries or reference anything other than the current row in any way.

  • A generation expression cannot reference another generated column.

  • A generation expression cannot reference a system column, except tableoid.

  • A generated column cannot have a column default or an identity definition.

  • A generated column cannot be part of a partition key.

  • Foreign tables can have generated columns. See CREATE FOREIGN TABLE for details.

  • For inheritance:

    • If a parent column is a generated column, a child column must also be a generated column using the same expression. In the definition of the child column, leave off the GENERATED clause, as it will be copied from the parent.

    • In case of multiple inheritance, if one parent column is a generated column, then all parent columns must be generated columns and with the same expression.

    • If a parent column is not a generated column, a child column may be defined to be a generated column or not.

Additional considerations apply to the use of generated columns.

  • Generated columns maintain access privileges separately from their underlying base columns. So, it is possible to arrange it so that a particular role can read from a generated column but not from the underlying base columns.

  • Generated columns are, conceptually, updated after BEFORE triggers have run. Therefore, changes made to base columns in a BEFORE trigger will be reflected in generated columns. But conversely, it is not allowed to access generated columns in BEFORE triggers.

Prev | Up | Next
5.2. Default Values | Home | 5.4. Constraints


5.4. Constraints
Prev Up Chapter 5. Data Definition Home Next

5.4. Constraints

5.4.1. Check Constraints
5.4.2. Not-Null Constraints
5.4.3. Unique Constraints
5.4.4. Primary Keys
5.4.5. Foreign Keys
5.4.6. Exclusion Constraints

Data types are a way to limit the kind of data that can be stored in a table. For many applications, however, the constraint they provide is too coarse. For example, a column containing a product price should probably only accept positive values. But there is no standard data type that accepts only positive numbers. Another issue is that you might want to constrain column data with respect to other columns or rows. For example, in a table containing product information, there should be only one row for each product number.

To that end, SQL allows you to define constraints on columns and tables. Constraints give you as much control over the data in your tables as you wish. If a user attempts to store data in a column that would violate a constraint, an error is raised. This applies even if the value came from the default value definition.

5.4.1. Check Constraints

A check constraint is the most generic constraint type. It allows you to specify that the value in a certain column must satisfy a Boolean (truth-value) expression. For instance, to require positive product prices, you could use:

CREATE TABLE products (
    product_no integer,
    name text,
    price numeric CHECK (price > 0)
);

As you see, the constraint definition comes after the data type, just like default value definitions. Default values and constraints can be listed in any order. A check constraint consists of the key word CHECK followed by an expression in parentheses. The check constraint expression should involve the column thus constrained, otherwise the constraint would not make too much sense.

You can also give the constraint a separate name. This clarifies error messages and allows you to refer to the constraint when you need to change it. The syntax is:

CREATE TABLE products (
    product_no integer,
    name text,
    price numeric CONSTRAINT positive_price CHECK (price > 0)
);

So, to specify a named constraint, use the key word CONSTRAINT followed by an identifier followed by the constraint definition. (If you don't specify a constraint name in this way, the system chooses a name for you.)

A check constraint can also refer to several columns. Say you store a regular price and a discounted price, and you want to ensure that the discounted price is lower than the regular price:

CREATE TABLE products (
    product_no integer,
    name text,
    price numeric CHECK (price > 0),
    discounted_price numeric CHECK (discounted_price > 0),
    CHECK (price > discounted_price)
);

The first two constraints should look familiar. The third one uses a new syntax. It is not attached to a particular column, instead it appears as a separate item in the comma-separated column list. Column definitions and these constraint definitions can be listed in mixed order.

We say that the first two constraints are column constraints, whereas the third one is a table constraint because it is written separately from any one column definition. Column constraints can also be written as table constraints, while the reverse is not necessarily possible, since a column constraint is supposed to refer to only the column it is attached to. (PostgreSQL doesn't enforce that rule, but you should follow it if you want your table definitions to work with other database systems.) The above example could also be written as:

CREATE TABLE products (
    product_no integer,
    name text,
    price numeric,
    CHECK (price > 0),
    discounted_price numeric,
    CHECK (discounted_price > 0),
    CHECK (price > discounted_price)
);

or even:

CREATE TABLE products (
    product_no integer,
    name text,
    price numeric CHECK (price > 0),
    discounted_price numeric,
    CHECK (discounted_price > 0 AND price > discounted_price)
);

It's a matter of taste.

Names can be assigned to table constraints in the same way as column constraints:

CREATE TABLE products (
    product_no integer,
    name text,
    price numeric,
    CHECK (price > 0),
    discounted_price numeric,
    CHECK (discounted_price > 0),
    CONSTRAINT valid_discount CHECK (price > discounted_price)
);

It should be noted that a check constraint is satisfied if the check expression evaluates to true or the null value. Since most expressions will evaluate to the null value if any operand is null, they will not prevent null values in the constrained columns. To ensure that a column does not contain null values, the not-null constraint described in the next section can be used.

Note

PostgreSQL does not support CHECK constraints that reference table data other than the new or updated row being checked. While a CHECK constraint that violates this rule may appear to work in simple tests, it cannot guarantee that the database will not reach a state in which the constraint condition is false (due to subsequent changes of the other row(s) involved). This would cause a database dump and reload to fail. The reload could fail even when the complete database state is consistent with the constraint, due to rows not being loaded in an order that will satisfy the constraint. If possible, use UNIQUE, EXCLUDE, or FOREIGN KEY constraints to express cross-row and cross-table restrictions.

If what you desire is a one-time check against other rows at row insertion, rather than a continuously-maintained consistency guarantee, a custom trigger can be used to implement that. (This approach avoids the dump/reload problem because pg_dump does not reinstall triggers until after reloading data, so that the check will not be enforced during a dump/reload.)

Note

PostgreSQL assumes that CHECK constraints' conditions are immutable, that is, they will always give the same result for the same input row. This assumption is what justifies examining CHECK constraints only when rows are inserted or updated, and not at other times. (The warning above about not referencing other table data is really a special case of this restriction.)

An example of a common way to break this assumption is to reference a user-defined function in a CHECK expression, and then change the behavior of that function. PostgreSQL does not disallow that, but it will not notice if there are rows in the table that now violate the CHECK constraint. That would cause a subsequent database dump and reload to fail. The recommended way to handle such a change is to drop the constraint (using ALTER TABLE), adjust the function definition, and re-add the constraint, thereby rechecking it against all table rows.

5.4.2. Not-Null Constraints

A not-null constraint simply specifies that a column must not assume the null value. A syntax example:

CREATE TABLE products (
    product_no integer NOT NULL,
    name text NOT NULL,
    price numeric
);

A not-null constraint is always written as a column constraint. A not-null constraint is functionally equivalent to creating a check constraint CHECK (column_name IS NOT NULL), but in PostgreSQL creating an explicit not-null constraint is more efficient. The drawback is that you cannot give explicit names to not-null constraints created this way.

Of course, a column can have more than one constraint. Just write the constraints one after another:

CREATE TABLE products (
    product_no integer NOT NULL,
    name text NOT NULL,
    price numeric NOT NULL CHECK (price > 0)
);

The order doesn't matter. It does not necessarily determine in which order the constraints are checked.

The NOT NULL constraint has an inverse: the NULL constraint. This does not mean that the column must be null, which would surely be useless. Instead, this simply selects the default behavior that the column might be null. The NULL constraint is not present in the SQL standard and should not be used in portable applications. (It was only added to PostgreSQL to be compatible with some other database systems.) Some users, however, like it because it makes it easy to toggle the constraint in a script file. For example, you could start with:

CREATE TABLE products (
    product_no integer NULL,
    name text NULL,
    price numeric NULL
);

and then insert the NOT key word where desired.

Tip

In most database designs the majority of columns should be marked not null.

5.4.3. Unique Constraints

Unique constraints ensure that the data contained in a column, or a group of columns, is unique among all the rows in the table. The syntax is:

CREATE TABLE products (
    product_no integer UNIQUE,
    name text,
    price numeric
);

when written as a column constraint, and:

CREATE TABLE products (
    product_no integer,
    name text,
    price numeric,
    UNIQUE (product_no)
);

when written as a table constraint.

To define a unique constraint for a group of columns, write it as a table constraint with the column names separated by commas:

CREATE TABLE example (
    a integer,
    b integer,
    c integer,
    UNIQUE (a, c)
);

This specifies that the combination of values in the indicated columns is unique across the whole table, though any one of the columns need not be (and ordinarily isn't) unique.

You can assign your own name for a unique constraint, in the usual way:

CREATE TABLE products (
    product_no integer CONSTRAINT must_be_different UNIQUE,
    name text,
    price numeric
);

Adding a unique constraint will automatically create a unique B-tree index on the column or group of columns listed in the constraint. A uniqueness restriction covering only some rows cannot be written as a unique constraint, but it is possible to enforce such a restriction by creating a unique partial index.

In general, a unique constraint is violated if there is more than one row in the table where the values of all of the columns included in the constraint are equal. However, two null values are never considered equal in this comparison. That means even in the presence of a unique constraint it is possible to store duplicate rows that contain a null value in at least one of the constrained columns. This behavior conforms to the SQL standard, but we have heard that other SQL databases might not follow this rule. So be careful when developing applications that are intended to be portable.

5.4.4. Primary Keys

A primary key constraint indicates that a column, or group of columns, can be used as a unique identifier for rows in the table. This requires that the values be both unique and not null. So, the following two table definitions accept the same data:

CREATE TABLE products (
    product_no integer UNIQUE NOT NULL,
    name text,
    price numeric
);

CREATE TABLE products (
    product_no integer PRIMARY KEY,
    name text,
    price numeric
);

Primary keys can span more than one column; the syntax is similar to unique constraints:

CREATE TABLE example (
    a integer,
    b integer,
    c integer,
    PRIMARY KEY (a, c)
);

Adding a primary key will automatically create a unique B-tree index on the column or group of columns listed in the primary key, and will force the column(s) to be marked NOT NULL.

A table can have at most one primary key. (There can be any number of unique and not-null constraints, which are functionally almost the same thing, but only one can be identified as the primary key.) Relational database theory dictates that every table must have a primary key. This rule is not enforced by PostgreSQL, but it is usually best to follow it.

Primary keys are useful both for documentation purposes and for client applications. For example, a GUI application that allows modifying row values probably needs to know the primary key of a table to be able to identify rows uniquely. There are also various ways in which the database system makes use of a primary key if one has been declared; for example, the primary key defines the default target column(s) for foreign keys referencing its table.

5.4.5. Foreign Keys

A foreign key constraint specifies that the values in a column (or a group of columns) must match the values appearing in some row of another table. We say this maintains the referential integrity between two related tables.

Say you have the product table that we have used several times already:

CREATE TABLE products (
    product_no integer PRIMARY KEY,
    name text,
    price numeric
);

Let's also assume you have a table storing orders of those products. We want to ensure that the orders table only contains orders of products that actually exist. So we define a foreign key constraint in the orders table that references the products table:

CREATE TABLE orders (
    order_id integer PRIMARY KEY,
    product_no integer REFERENCES products (product_no),
    quantity integer
);

Now it is impossible to create orders with non-NULL product_no entries that do not appear in the products table.

We say that in this situation the orders table is the referencing table and the products table is the referenced table. Similarly, there are referencing and referenced columns.

You can also shorten the above command to:

CREATE TABLE orders (
    order_id integer PRIMARY KEY,
    product_no integer REFERENCES products,
    quantity integer
);

because in absence of a column list the primary key of the referenced table is used as the referenced column(s).

You can assign your own name for a foreign key constraint, in the usual way.

A foreign key can also constrain and reference a group of columns. As usual, it then needs to be written in table constraint form. Here is a contrived syntax example:

CREATE TABLE t1 (
  a integer PRIMARY KEY,
  b integer,
  c integer,
  FOREIGN KEY (b, c) REFERENCES other_table (c1, c2)
);

Of course, the number and type of the constrained columns need to match the number and type of the referenced columns.

Sometimes it is useful for the “other table” of a foreign key constraint to be the same table; this is called a self-referential foreign key. For example, if you want rows of a table to represent nodes of a tree structure, you could write

CREATE TABLE tree (
    node_id integer PRIMARY KEY,
    parent_id integer REFERENCES tree,
    name text,
    ...
);

A top-level node would have NULL parent_id, while non-NULL parent_id entries would be constrained to reference valid rows of the table.

A table can have more than one foreign key constraint. This is used to implement many-to-many relationships between tables. Say you have tables about products and orders, but now you want to allow one order to contain possibly many products (which the structure above did not allow). You could use this table structure:

CREATE TABLE products (
    product_no integer PRIMARY KEY,
    name text,
    price numeric
);

CREATE TABLE orders (
    order_id integer PRIMARY KEY,
    shipping_address text,
    ...
);

CREATE TABLE order_items (
    product_no integer REFERENCES products,
    order_id integer REFERENCES orders,
    quantity integer,
    PRIMARY KEY (product_no, order_id)
);

Notice that the primary key overlaps with the foreign keys in the last table.

We know that the foreign keys disallow creation of orders that do not relate to any products. But what if a product is removed after an order is created that references it? SQL allows you to handle that as well. Intuitively, we have a few options:

  • Disallow deleting a referenced product

  • Delete the orders as well

  • Something else?

To illustrate this, let's implement the following policy on the many-to-many relationship example above: when someone wants to remove a product that is still referenced by an order (via order_items), we disallow it. If someone removes an order, the order items are removed as well:

CREATE TABLE products (
    product_no integer PRIMARY KEY,
    name text,
    price numeric
);

CREATE TABLE orders (
    order_id integer PRIMARY KEY,
    shipping_address text,
    ...
);

CREATE TABLE order_items (
    product_no integer REFERENCES products ON DELETE RESTRICT,
    order_id integer REFERENCES orders ON DELETE CASCADE,
    quantity integer,
    PRIMARY KEY (product_no, order_id)
);

Restricting and cascading deletes are the two most common options. RESTRICT prevents deletion of a referenced row. NO ACTION means that if any referencing rows still exist when the constraint is checked, an error is raised; this is the default behavior if you do not specify anything. (The essential difference between these two choices is that NO ACTION allows the check to be deferred until later in the transaction, whereas RESTRICT does not.) CASCADE specifies that when a referenced row is deleted, row(s) referencing it should be automatically deleted as well. There are two other options: SET NULL and SET DEFAULT. These cause the referencing column(s) in the referencing row(s) to be set to nulls or their default values, respectively, when the referenced row is deleted. Note that these do not excuse you from observing any constraints. For example, if an action specifies SET DEFAULT but the default value would not satisfy the foreign key constraint, the operation will fail.

Analogous to ON DELETE there is also ON UPDATE which is invoked when a referenced column is changed (updated). The possible actions are the same. In this case, CASCADE means that the updated values of the referenced column(s) should be copied into the referencing row(s).

Normally, a referencing row need not satisfy the foreign key constraint if any of its referencing columns are null. If MATCH FULL is added to the foreign key declaration, a referencing row escapes satisfying the constraint only if all its referencing columns are null (so a mix of null and non-null values is guaranteed to fail a MATCH FULL constraint). If you don't want referencing rows to be able to avoid satisfying the foreign key constraint, declare the referencing column(s) as NOT NULL.

A foreign key must reference columns that either are a primary key or form a unique constraint. This means that the referenced columns always have an index (the one underlying the primary key or unique constraint); so checks on whether a referencing row has a match will be efficient. Since a DELETE of a row from the referenced table or an UPDATE of a referenced column will require a scan of the referencing table for rows matching the old value, it is often a good idea to index the referencing columns too. Because this is not always needed, and there are many choices available on how to index, declaration of a foreign key constraint does not automatically create an index on the referencing columns.

More information about updating and deleting data is in Chapter 6. Also see the description of foreign key constraint syntax in the reference documentation for CREATE TABLE.

5.4.6. Exclusion Constraints

Exclusion constraints ensure that if any two rows are compared on the specified columns or expressions using the specified operators, at least one of these operator comparisons will return false or null. The syntax is:

CREATE TABLE circles (
    c circle,
    EXCLUDE USING gist (c WITH &&)
);

See also CREATE TABLE ... CONSTRAINT ... EXCLUDE for details.

Adding an exclusion constraint will automatically create an index of the type specified in the constraint declaration.

Prev | Up | Next
5.3. Generated Columns | Home | 5.5. System Columns


5.5. System Columns
Prev Up Chapter 5. Data Definition Home Next

5.5. System Columns

Every table has several system columns that are implicitly defined by the system. Therefore, these names cannot be used as names of user-defined columns. (Note that these restrictions are separate from whether the name is a key word or not; quoting a name will not allow you to escape these restrictions.) You do not really need to be concerned about these columns; just know they exist.

tableoid

The OID of the table containing this row. This column is particularly handy for queries that select from partitioned tables (see Section 5.11) or inheritance hierarchies (see Section 5.10), since without it, it's difficult to tell which individual table a row came from. The tableoid can be joined against the oid column of pg_class to obtain the table name.

xmin

The identity (transaction ID) of the inserting transaction for this row version. (A row version is an individual state of a row; each update of a row creates a new row version for the same logical row.)

cmin

The command identifier (starting at zero) within the inserting transaction.

xmax

The identity (transaction ID) of the deleting transaction, or zero for an undeleted row version. It is possible for this column to be nonzero in a visible row version. That usually indicates that the deleting transaction hasn't committed yet, or that an attempted deletion was rolled back.

cmax

The command identifier within the deleting transaction, or zero.

ctid

The physical location of the row version within its table. Note that although the ctid can be used to locate the row version very quickly, a row's ctid will change if it is updated or moved by VACUUM FULL. Therefore ctid is useless as a long-term row identifier. A primary key should be used to identify logical rows.

Transaction identifiers are also 32-bit quantities. In a long-lived database it is possible for transaction IDs to wrap around. This is not a fatal problem given appropriate maintenance procedures; see Chapter 25 for details. It is unwise, however, to depend on the uniqueness of transaction IDs over the long term (more than one billion transactions).

Command identifiers are also 32-bit quantities. This creates a hard limit of 232 (4 billion) SQL commands within a single transaction. In practice this limit is not a problem — note that the limit is on the number of SQL commands, not the number of rows processed. Also, only commands that actually modify the database contents will consume a command identifier.

Prev | Up | Next
5.4. Constraints | Home | 5.6. Modifying Tables


5.6. Modifying Tables
Prev Up Chapter 5. Data Definition Home Next

5.6. Modifying Tables

5.6.1. Adding a Column
5.6.2. Removing a Column
5.6.3. Adding a Constraint
5.6.4. Removing a Constraint
5.6.5. Changing a Column's Default Value
5.6.6. Changing a Column's Data Type
5.6.7. Renaming a Column
5.6.8. Renaming a Table

When you create a table and you realize that you made a mistake, or the requirements of the application change, you can drop the table and create it again. But this is not a convenient option if the table is already filled with data, or if the table is referenced by other database objects (for instance a foreign key constraint). Therefore PostgreSQL provides a family of commands to make modifications to existing tables. Note that this is conceptually distinct from altering the data contained in the table: here we are interested in altering the definition, or structure, of the table.

You can:

  • Add columns

  • Remove columns

  • Add constraints

  • Remove constraints

  • Change default values

  • Change column data types

  • Rename columns

  • Rename tables

All these actions are performed using the ALTER TABLE command, whose reference page contains details beyond those given here.

5.6.1. Adding a Column

To add a column, use a command like:

ALTER TABLE products ADD COLUMN description text;

The new column is initially filled with whatever default value is given (null if you don't specify a DEFAULT clause).

Tip

From PostgreSQL 11, adding a column with a constant default value no longer means that each row of the table needs to be updated when the ALTER TABLE statement is executed. Instead, the default value will be returned the next time the row is accessed, and applied when the table is rewritten, making the ALTER TABLE very fast even on large tables.

However, if the default value is volatile (e.g., clock_timestamp()) each row will need to be updated with the value calculated at the time ALTER TABLE is executed. To avoid a potentially lengthy update operation, particularly if you intend to fill the column with mostly nondefault values anyway, it may be preferable to add the column with no default, insert the correct values using UPDATE, and then add any desired default as described below.

You can also define constraints on the column at the same time, using the usual syntax:

ALTER TABLE products ADD COLUMN description text CHECK (description <> '');

In fact all the options that can be applied to a column description in CREATE TABLE can be used here. Keep in mind however that the default value must satisfy the given constraints, or the ADD will fail. Alternatively, you can add constraints later (see below) after you've filled in the new column correctly.

5.6.2. Removing a Column

To remove a column, use a command like:

ALTER TABLE products DROP COLUMN description;

Whatever data was in the column disappears. Table constraints involving the column are dropped, too. However, if the column is referenced by a foreign key constraint of another table, PostgreSQL will not silently drop that constraint. You can authorize dropping everything that depends on the column by adding CASCADE:

ALTER TABLE products DROP COLUMN description CASCADE;

See Section 5.14 for a description of the general mechanism behind this.

5.6.3. Adding a Constraint

To add a constraint, the table constraint syntax is used. For example:

ALTER TABLE products ADD CHECK (name <> '');
ALTER TABLE products ADD CONSTRAINT some_name UNIQUE (product_no);
ALTER TABLE products ADD FOREIGN KEY (product_group_id) REFERENCES product_groups;

To add a not-null constraint, which cannot be written as a table constraint, use this syntax:

ALTER TABLE products ALTER COLUMN product_no SET NOT NULL;

The constraint will be checked immediately, so the table data must satisfy the constraint before it can be added.

5.6.4. Removing a Constraint

To remove a constraint you need to know its name. If you gave it a name then that's easy. Otherwise the system assigned a generated name, which you need to find out. The psql command \d tablename can be helpful here; other interfaces might also provide a way to inspect table details. Then the command is:

ALTER TABLE products DROP CONSTRAINT some_name;

(If you are dealing with a generated constraint name like $2, don't forget that you'll need to double-quote it to make it a valid identifier.)

As with dropping a column, you need to add CASCADE if you want to drop a constraint that something else depends on. An example is that a foreign key constraint depends on a unique or primary key constraint on the referenced column(s).

This works the same for all constraint types except not-null constraints. To drop a not null constraint use:

ALTER TABLE products ALTER COLUMN product_no DROP NOT NULL;

(Recall that not-null constraints do not have names.)

5.6.5. Changing a Column's Default Value

To set a new default for a column, use a command like:

ALTER TABLE products ALTER COLUMN price SET DEFAULT 7.77;

Note that this doesn't affect any existing rows in the table, it just changes the default for future INSERT commands.

To remove any default value, use:

ALTER TABLE products ALTER COLUMN price DROP DEFAULT;

This is effectively the same as setting the default to null. As a consequence, it is not an error to drop a default where one hadn't been defined, because the default is implicitly the null value.

5.6.6. Changing a Column's Data Type

To convert a column to a different data type, use a command like:

ALTER TABLE products ALTER COLUMN price TYPE numeric(10,2);

This will succeed only if each existing entry in the column can be converted to the new type by an implicit cast. If a more complex conversion is needed, you can add a USING clause that specifies how to compute the new values from the old.

PostgreSQL will attempt to convert the column's default value (if any) to the new type, as well as any constraints that involve the column. But these conversions might fail, or might produce surprising results. It's often best to drop any constraints on the column before altering its type, and then add back suitably modified constraints afterwards.

5.6.7. Renaming a Column

To rename a column:

ALTER TABLE products RENAME COLUMN product_no TO product_number;

5.6.8. Renaming a Table

To rename a table:

ALTER TABLE products RENAME TO items;

Prev | Up | Next
5.5. System Columns | Home | 5.7. Privileges


5.7. Privileges
Prev Up Chapter 5. Data Definition Home Next

5.7. Privileges

When an object is created, it is assigned an owner. The owner is normally the role that executed the creation statement. For most kinds of objects, the initial state is that only the owner (or a superuser) can do anything with the object. To allow other roles to use it, privileges must be granted.

There are different kinds of privileges: SELECT, INSERT, UPDATE, DELETE, TRUNCATE, REFERENCES, TRIGGER, CREATE, CONNECT, TEMPORARY, EXECUTE, and USAGE. The privileges applicable to a particular object vary depending on the object's type (table, function, etc). More detail about the meanings of these privileges appears below. The following sections and chapters will also show you how these privileges are used.

The right to modify or destroy an object is inherent in being the object's owner, and cannot be granted or revoked in itself. (However, like all privileges, that right can be inherited by members of the owning role; see Section 22.3.)

An object can be assigned to a new owner with an ALTER command of the appropriate kind for the object, for example

ALTER TABLE table_name OWNER TO new_owner;

Superusers can always do this; ordinary roles can only do it if they are both the current owner of the object (or a member of the owning role) and a member of the new owning role.

To assign privileges, the GRANT command is used. For example, if joe is an existing role, and accounts is an existing table, the privilege to update the table can be granted with:

GRANT UPDATE ON accounts TO joe;

Writing ALL in place of a specific privilege grants all privileges that are relevant for the object type.

The special “role” name PUBLIC can be used to grant a privilege to every role on the system. Also, “group” roles can be set up to help manage privileges when there are many users of a database — for details see Chapter 22.

To revoke a previously-granted privilege, use the fittingly named REVOKE command:

REVOKE ALL ON accounts FROM PUBLIC;

Ordinarily, only the object's owner (or a superuser) can grant or revoke privileges on an object. However, it is possible to grant a privilege “with grant option”, which gives the recipient the right to grant it in turn to others. If the grant option is subsequently revoked then all who received the privilege from that recipient (directly or through a chain of grants) will lose the privilege. For details see the GRANT and REVOKE reference pages.

An object's owner can choose to revoke their own ordinary privileges, for example to make a table read-only for themselves as well as others. But owners are always treated as holding all grant options, so they can always re-grant their own privileges.

The available privileges are:

SELECT

Allows SELECT from any column, or specific column(s), of a table, view, materialized view, or other table-like object. Also allows use of COPY TO. This privilege is also needed to reference existing column values in UPDATE or DELETE. For sequences, this privilege also allows use of the currval function. For large objects, this privilege allows the object to be read.

INSERT

Allows INSERT of a new row into a table, view, etc. Can be granted on specific column(s), in which case only those columns may be assigned to in the INSERT command (other columns will therefore receive default values). Also allows use of COPY FROM.

UPDATE

Allows UPDATE of any column, or specific column(s), of a table, view, etc. (In practice, any nontrivial UPDATE command will require SELECT privilege as well, since it must reference table columns to determine which rows to update, and/or to compute new values for columns.) SELECT ... FOR UPDATE and SELECT ... FOR SHARE also require this privilege on at least one column, in addition to the SELECT privilege. For sequences, this privilege allows use of the nextval and setval functions. For large objects, this privilege allows writing or truncating the object.

DELETE

Allows DELETE of a row from a table, view, etc. (In practice, any nontrivial DELETE command will require SELECT privilege as well, since it must reference table columns to determine which rows to delete.)

TRUNCATE

Allows TRUNCATE on a table.

REFERENCES

Allows creation of a foreign key constraint referencing a table, or specific column(s) of a table.

TRIGGER

Allows creation of a trigger on a table, view, etc.

CREATE

For databases, allows new schemas and publications to be created within the database, and allows trusted extensions to be installed within the database.

For schemas, allows new objects to be created within the schema. To rename an existing object, you must own the object and have this privilege for the containing schema.

For tablespaces, allows tables, indexes, and temporary files to be created within the tablespace, and allows databases to be created that have the tablespace as their default tablespace.

Note that revoking this privilege will not alter the existence or location of existing objects.

CONNECT

Allows the grantee to connect to the database. This privilege is checked at connection startup (in addition to checking any restrictions imposed by pg_hba.conf).

TEMPORARY

Allows temporary tables to be created while using the database.

EXECUTE

Allows calling a function or procedure, including use of any operators that are implemented on top of the function. This is the only type of privilege that is applicable to functions and procedures.

USAGE

For procedural languages, allows use of the language for the creation of functions in that language. This is the only type of privilege that is applicable to procedural languages.

For schemas, allows access to objects contained in the schema (assuming that the objects' own privilege requirements are also met). Essentially this allows the grantee to “look up” objects within the schema. Without this permission, it is still possible to see the object names, e.g., by querying system catalogs. Also, after revoking this permission, existing sessions might have statements that have previously performed this lookup, so this is not a completely secure way to prevent object access.

For sequences, allows use of the currval and nextval functions.

For types and domains, allows use of the type or domain in the creation of tables, functions, and other schema objects. (Note that this privilege does not control all “usage” of the type, such as values of the type appearing in queries. It only prevents objects from being created that depend on the type. The main purpose of this privilege is controlling which users can create dependencies on a type, which could prevent the owner from changing the type later.)

For foreign-data wrappers, allows creation of new servers using the foreign-data wrapper.

For foreign servers, allows creation of foreign tables using the server. Grantees may also create, alter, or drop their own user mappings associated with that server.

The privileges required by other commands are listed on the reference page of the respective command.

PostgreSQL grants privileges on some types of objects to PUBLIC by default when the objects are created. No privileges are granted to PUBLIC by default on tables, table columns, sequences, foreign data wrappers, foreign servers, large objects, schemas, or tablespaces. For other types of objects, the default privileges granted to PUBLIC are as follows: CONNECT and TEMPORARY (create temporary tables) privileges for databases; EXECUTE privilege for functions and procedures; and USAGE privilege for languages and data types (including domains). The object owner can, of course, REVOKE both default and expressly granted privileges. (For maximum security, issue the REVOKE in the same transaction that creates the object; then there is no window in which another user can use the object.) Also, these default privilege settings can be overridden using the ALTER DEFAULT PRIVILEGES command.

Table 5.1 shows the one-letter abbreviations that are used for these privilege types in ACL (Access Control List) values. You will see these letters in the output of the psql commands listed below, or when looking at ACL columns of system catalogs.

Table 5.1. ACL Privilege Abbreviations

Privilege | Abbreviation | Applicable Object Types
SELECT | r (“read”) | LARGE OBJECT, SEQUENCE, TABLE (and table-like objects), table column
INSERT | a (“append”) | TABLE, table column
UPDATE | w (“write”) | LARGE OBJECT, SEQUENCE, TABLE, table column
DELETE | d | TABLE
TRUNCATE | D | TABLE
REFERENCES | x | TABLE, table column
TRIGGER | t | TABLE
CREATE | C | DATABASE, SCHEMA, TABLESPACE
CONNECT | c | DATABASE
TEMPORARY | T | DATABASE
EXECUTE | X | FUNCTION, PROCEDURE
USAGE | U | DOMAIN, FOREIGN DATA WRAPPER, FOREIGN SERVER, LANGUAGE, SCHEMA, SEQUENCE, TYPE

Table 5.2 summarizes the privileges available for each type of SQL object, using the abbreviations shown above. It also shows the psql command that can be used to examine privilege settings for each object type.

Table 5.2. Summary of Access Privileges

Object Type | All Privileges | Default PUBLIC Privileges | psql Command
DATABASE | CTc | Tc | \l
DOMAIN | U | U | \dD+
FUNCTION or PROCEDURE | X | X | \df+
FOREIGN DATA WRAPPER | U | none | \dew+
FOREIGN SERVER | U | none | \des+
LANGUAGE | U | U | \dL+
LARGE OBJECT | rw | none |
SCHEMA | UC | none | \dn+
SEQUENCE | rwU | none | \dp
TABLE (and table-like objects) | arwdDxt | none | \dp
Table column | arwx | none | \dp
TABLESPACE | C | none | \db+
TYPE | U | U | \dT+

The privileges that have been granted for a particular object are displayed as a list of aclitem entries, where each aclitem describes the permissions of one grantee that have been granted by a particular grantor. For example, calvin=r*w/hobbes specifies that the role calvin has the privilege SELECT (r) with grant option (*) as well as the non-grantable privilege UPDATE (w), both granted by the role hobbes. If calvin also has some privileges on the same object granted by a different grantor, those would appear as a separate aclitem entry. An empty grantee field in an aclitem stands for PUBLIC.

As an example, suppose that user miriam creates table mytable and does:

GRANT SELECT ON mytable TO PUBLIC;
GRANT SELECT, UPDATE, INSERT ON mytable TO admin;
GRANT SELECT (col1), UPDATE (col1) ON mytable TO miriam_rw;

Then psql's \dp command would show:

=> \dp mytable
                                  Access privileges
 Schema |  Name   | Type  |   Access privileges   |   Column privileges   | Policies
--------+---------+-------+-----------------------+-----------------------+----------
 public | mytable | table | miriam=arwdDxt/miriam+| col1:                +|
        |         |       | =r/miriam            +|   miriam_rw=rw/miriam |
        |         |       | admin=arw/miriam      |                       |
(1 row)

If the “Access privileges” column is empty for a given object, it means the object has default privileges (that is, its privileges entry in the relevant system catalog is null). Default privileges always include all privileges for the owner, and can include some privileges for PUBLIC depending on the object type, as explained above. The first GRANT or REVOKE on an object will instantiate the default privileges (producing, for example, miriam=arwdDxt/miriam) and then modify them per the specified request. Similarly, entries are shown in “Column privileges” only for columns with nondefault privileges. (Note: for this purpose, “default privileges” always means the built-in default privileges for the object's type. An object whose privileges have been affected by an ALTER DEFAULT PRIVILEGES command will always be shown with an explicit privilege entry that includes the effects of the ALTER.)

Notice that the owner's implicit grant options are not marked in the access privileges display. A * will appear only when grant options have been explicitly granted to someone.

Prev | Up | Next
5.6. Modifying Tables | Home | 5.8. Row Security Policies


5.8. Row Security Policies
Prev Up Chapter 5. Data Definition Home Next

5.8. Row Security Policies

In addition to the SQL-standard privilege system available through GRANT, tables can have row security policies that restrict, on a per-user basis, which rows can be returned by normal queries or inserted, updated, or deleted by data modification commands. This feature is also known as Row-Level Security. By default, tables do not have any policies, so that if a user has access privileges to a table according to the SQL privilege system, all rows within it are equally available for querying or updating.

When row security is enabled on a table (with ALTER TABLE ... ENABLE ROW LEVEL SECURITY), all normal access to the table for selecting rows or modifying rows must be allowed by a row security policy. (However, the table's owner is typically not subject to row security policies.) If no policy exists for the table, a default-deny policy is used, meaning that no rows are visible or can be modified. Operations that apply to the whole table, such as TRUNCATE and REFERENCES, are not subject to row security.

Row security policies can be specific to commands, or to roles, or to both. A policy can be specified to apply to ALL commands, or to SELECT, INSERT, UPDATE, or DELETE. Multiple roles can be assigned to a given policy, and normal role membership and inheritance rules apply.

To specify which rows are visible or modifiable according to a policy, an expression is required that returns a Boolean result. This expression will be evaluated for each row prior to any conditions or functions coming from the user's query. (The only exceptions to this rule are leakproof functions, which are guaranteed to not leak information; the optimizer may choose to apply such functions ahead of the row-security check.) Rows for which the expression does not return true will not be processed. Separate expressions may be specified to provide independent control over the rows which are visible and the rows which are allowed to be modified. Policy expressions are run as part of the query and with the privileges of the user running the query, although security-definer functions can be used to access data not available to the calling user.

Superusers and roles with the BYPASSRLS attribute always bypass the row security system when accessing a table. Table owners normally bypass row security as well, though a table owner can choose to be subject to row security with ALTER TABLE ... FORCE ROW LEVEL SECURITY.

Enabling and disabling row security, as well as adding policies to a table, is always the privilege of the table owner only.

Policies are created using the CREATE POLICY command, altered using the ALTER POLICY command, and dropped using the DROP POLICY command. To enable and disable row security for a given table, use the ALTER TABLE command.

Each policy has a name and multiple policies can be defined for a table. As policies are table-specific, each policy for a table must have a unique name. Different tables may have policies with the same name.

When multiple policies apply to a given query, they are combined using either OR (for permissive policies, which are the default) or using AND (for restrictive policies). This is similar to the rule that a given role has the privileges of all roles that they are a member of. Permissive vs. restrictive policies are discussed further below.

As a simple example, here is how to create a policy on the account relation to allow only members of the managers role to access rows, and only rows of their accounts:

CREATE TABLE accounts (manager text, company text, contact_email text);

ALTER TABLE accounts ENABLE ROW LEVEL SECURITY;

CREATE POLICY account_managers ON accounts TO managers
    USING (manager = current_user);

The policy above implicitly provides a WITH CHECK clause identical to its USING clause, so that the constraint applies both to rows selected by a command (so a manager cannot SELECT, UPDATE, or DELETE existing rows belonging to a different manager) and to rows modified by a command (so rows belonging to a different manager cannot be created via INSERT or UPDATE).

If no role is specified, or the special user name PUBLIC is used, then the policy applies to all users on the system. To allow all users to access only their own row in a users table, a simple policy can be used:

CREATE POLICY user_policy ON users
    USING (user_name = current_user);

This works similarly to the previous example.

To use a different policy for rows that are being added to the table compared to those rows that are visible, multiple policies can be combined. This pair of policies would allow all users to view all rows in the users table, but only modify their own:

CREATE POLICY user_sel_policy ON users
    FOR SELECT
    USING (true);
CREATE POLICY user_mod_policy ON users
    USING (user_name = current_user);

In a SELECT command, these two policies are combined using OR, with the net effect being that all rows can be selected. In other command types, only the second policy applies, so that the effects are the same as before.

Row security can also be disabled with the ALTER TABLE command. Disabling row security does not remove any policies that are defined on the table; they are simply ignored. Then all rows in the table are visible and modifiable, subject to the standard SQL privileges system.

Below is a larger example of how this feature can be used in production environments. The table passwd emulates a Unix password file:

-- Simple passwd-file based example
CREATE TABLE passwd (
  user_name             text UNIQUE NOT NULL,
  pwhash                text,
  uid                   int  PRIMARY KEY,
  gid                   int  NOT NULL,
  real_name             text NOT NULL,
  home_phone            text,
  extra_info            text,
  home_dir              text NOT NULL,
  shell                 text NOT NULL
);

CREATE ROLE admin;  -- Administrator
CREATE ROLE bob;    -- Normal user
CREATE ROLE alice;  -- Normal user

-- Populate the table
INSERT INTO passwd VALUES
  ('admin','xxx',0,0,'Admin','111-222-3333',null,'/root','/bin/dash');
INSERT INTO passwd VALUES
  ('bob','xxx',1,1,'Bob','123-456-7890',null,'/home/bob','/bin/zsh');
INSERT INTO passwd VALUES
  ('alice','xxx',2,1,'Alice','098-765-4321',null,'/home/alice','/bin/zsh');

-- Be sure to enable row-level security on the table
ALTER TABLE passwd ENABLE ROW LEVEL SECURITY;

-- Create policies
-- Administrator can see all rows and add any rows
CREATE POLICY admin_all ON passwd TO admin USING (true) WITH CHECK (true);
-- Normal users can view all rows
CREATE POLICY all_view ON passwd FOR SELECT USING (true);
-- Normal users can update their own records, but
-- limit which shells a normal user is allowed to set
CREATE POLICY user_mod ON passwd FOR UPDATE
  USING (current_user = user_name)
  WITH CHECK (
    current_user = user_name AND
    shell IN ('/bin/bash','/bin/sh','/bin/dash','/bin/zsh','/bin/tcsh')
  );

-- Allow admin all normal rights
GRANT SELECT, INSERT, UPDATE, DELETE ON passwd TO admin;
-- Users only get select access on public columns
GRANT SELECT
  (user_name, uid, gid, real_name, home_phone, extra_info, home_dir, shell)
  ON passwd TO public;
-- Allow users to update certain columns
GRANT UPDATE
  (pwhash, real_name, home_phone, extra_info, shell)
  ON passwd TO public;

As with any security settings, it's important to test and ensure that the system is behaving as expected. Using the example above, this demonstrates that the permission system is working properly.

-- admin can view all rows and fields
postgres=> set role admin;
SET
postgres=> table passwd;
 user_name | pwhash | uid | gid | real_name |  home_phone  | extra_info | home_dir    |   shell
-----------+--------+-----+-----+-----------+--------------+------------+-------------+-----------
 admin     | xxx    |   0 |   0 | Admin     | 111-222-3333 |            | /root       | /bin/dash
 bob       | xxx    |   1 |   1 | Bob       | 123-456-7890 |            | /home/bob   | /bin/zsh
 alice     | xxx    |   2 |   1 | Alice     | 098-765-4321 |            | /home/alice | /bin/zsh
(3 rows)

-- Test what Alice is able to do
postgres=> set role alice;
SET
postgres=> table passwd;
ERROR:  permission denied for relation passwd
postgres=> select user_name,real_name,home_phone,extra_info,home_dir,shell from passwd;
 user_name | real_name |  home_phone  | extra_info | home_dir    |   shell
-----------+-----------+--------------+------------+-------------+-----------
 admin     | Admin     | 111-222-3333 |            | /root       | /bin/dash
 bob       | Bob       | 123-456-7890 |            | /home/bob   | /bin/zsh
 alice     | Alice     | 098-765-4321 |            | /home/alice | /bin/zsh
(3 rows)

postgres=> update passwd set user_name = 'joe';
ERROR:  permission denied for relation passwd
-- Alice is allowed to change her own real_name, but no others
postgres=> update passwd set real_name = 'Alice Doe';
UPDATE 1
postgres=> update passwd set real_name = 'John Doe' where user_name = 'admin';
UPDATE 0
postgres=> update passwd set shell = '/bin/xx';
ERROR:  new row violates WITH CHECK OPTION for "passwd"
postgres=> delete from passwd;
ERROR:  permission denied for relation passwd
postgres=> insert into passwd (user_name) values ('xxx');
ERROR:  permission denied for relation passwd
-- Alice can change her own password; RLS silently prevents updating other rows
postgres=> update passwd set pwhash = 'abc';
UPDATE 1

All of the policies constructed thus far have been permissive policies, meaning that when multiple policies are applied they are combined using the “OR” Boolean operator. While permissive policies can be constructed to only allow access to rows in the intended cases, it can be simpler to combine permissive policies with restrictive policies (which the records must pass and which are combined using the “AND” Boolean operator). Building on the example above, we add a restrictive policy to require the administrator to be connected over a local Unix socket to access the records of the passwd table:

CREATE POLICY admin_local_only ON passwd AS RESTRICTIVE TO admin
    USING (pg_catalog.inet_client_addr() IS NULL);

We can then see that an administrator connecting over a network will not see any records, due to the restrictive policy:

=> SELECT current_user;
 current_user
--------------
 admin
(1 row)

=> select inet_client_addr();
 inet_client_addr
------------------
 127.0.0.1
(1 row)

=> TABLE passwd;
 user_name | pwhash | uid | gid | real_name | home_phone | extra_info | home_dir | shell
-----------+--------+-----+-----+-----------+------------+------------+----------+-------
(0 rows)

=> UPDATE passwd set pwhash = NULL;
UPDATE 0

Referential integrity checks, such as unique or primary key constraints and foreign key references, always bypass row security to ensure that data integrity is maintained. Care must be taken when developing schemas and row level policies to avoid “covert channel” leaks of information through such referential integrity checks.

In some contexts it is important to be sure that row security is not being applied. For example, when taking a backup, it could be disastrous if row security silently caused some rows to be omitted from the backup. In such a situation, you can set the row_security configuration parameter to off. This does not in itself bypass row security; what it does is throw an error if any query's results would get filtered by a policy. The reason for the error can then be investigated and fixed.

In the examples above, the policy expressions consider only the current values in the row to be accessed or updated. This is the simplest and best-performing case; when possible, it's best to design row security applications to work this way. If it is necessary to consult other rows or other tables to make a policy decision, that can be accomplished using sub-SELECTs, or functions that contain SELECTs, in the policy expressions. Be aware however that such accesses can create race conditions that could allow information leakage if care is not taken. As an example, consider the following table design:

-- definition of privilege groups
CREATE TABLE groups (group_id int PRIMARY KEY,
                     group_name text NOT NULL);

INSERT INTO groups VALUES
  (1, 'low'),
  (2, 'medium'),
  (5, 'high');

GRANT ALL ON groups TO alice;  -- alice is the administrator
GRANT SELECT ON groups TO public;

-- definition of users' privilege levels
CREATE TABLE users (user_name text PRIMARY KEY,
                    group_id int NOT NULL REFERENCES groups);

INSERT INTO users VALUES
  ('alice', 5),
  ('bob', 2),
  ('mallory', 2);

GRANT ALL ON users TO alice;
GRANT SELECT ON users TO public;

-- table holding the information to be protected
CREATE TABLE information (info text,
                          group_id int NOT NULL REFERENCES groups);

INSERT INTO information VALUES
  ('barely secret', 1),
  ('slightly secret', 2),
  ('very secret', 5);

ALTER TABLE information ENABLE ROW LEVEL SECURITY;

-- a row should be visible to/updatable by users whose security group_id is
-- greater than or equal to the row's group_id
CREATE POLICY fp_s ON information FOR SELECT
  USING (group_id <= (SELECT group_id FROM users WHERE user_name = current_user));
CREATE POLICY fp_u ON information FOR UPDATE
  USING (group_id <= (SELECT group_id FROM users WHERE user_name = current_user));

-- we rely only on RLS to protect the information table
GRANT ALL ON information TO public;

Now suppose that alice wishes to change the “slightly secret” information, but decides that mallory should not be trusted with the new content of that row, so she does:

BEGIN;
UPDATE users SET group_id = 1 WHERE user_name = 'mallory';
UPDATE information SET info = 'secret from mallory' WHERE group_id = 2;
COMMIT;

That looks safe; there is no window wherein mallory should be able to see the “secret from mallory” string. However, there is a race condition here. If mallory is concurrently doing, say,

SELECT * FROM information WHERE group_id = 2 FOR UPDATE;

and her transaction is in READ COMMITTED mode, it is possible for her to see “secret from mallory”. That happens if her transaction reaches the information row just after alice's does. It blocks waiting for alice's transaction to commit, then fetches the updated row contents thanks to the FOR UPDATE clause. However, it does not fetch an updated row for the implicit SELECT from users, because that sub-SELECT did not have FOR UPDATE; instead the users row is read with the snapshot taken at the start of the query. Therefore, the policy expression tests the old value of mallory's privilege level and allows her to see the updated row.

There are several ways around this problem. One simple answer is to use SELECT ... FOR SHARE in sub-SELECTs in row security policies. However, that requires granting UPDATE privilege on the referenced table (here users) to the affected users, which might be undesirable. (But another row security policy could be applied to prevent them from actually exercising that privilege; or the sub-SELECT could be embedded into a security definer function.) Also, heavy concurrent use of row share locks on the referenced table could pose a performance problem, especially if updates of it are frequent. Another solution, practical if updates of the referenced table are infrequent, is to take an ACCESS EXCLUSIVE lock on the referenced table when updating it, so that no concurrent transactions could be examining old row values. Or one could just wait for all concurrent transactions to end after committing an update of the referenced table and before making changes that rely on the new security situation.

For additional details see CREATE POLICY and ALTER TABLE.

Prev | Up | Next
5.7. Privileges | Home | 5.9. Schemas


5.9. Schemas
Prev Up Chapter 5. Data Definition Home Next

5.9. Schemas

5.9.1. Creating a Schema
5.9.2. The Public Schema
5.9.3. The Schema Search Path
5.9.4. Schemas and Privileges
5.9.5. The System Catalog Schema
5.9.6. Usage Patterns
5.9.7. Portability

A PostgreSQL database cluster contains one or more named databases. Roles and a few other object types are shared across the entire cluster. A client connection to the server can only access data in a single database, the one specified in the connection request.

Note

Users of a cluster do not necessarily have the privilege to access every database in the cluster. Sharing of role names means that there cannot be different roles named, say, joe in two databases in the same cluster; but the system can be configured to allow joe access to only some of the databases.

A database contains one or more named schemas, which in turn contain tables. Schemas also contain other kinds of named objects, including data types, functions, and operators. The same object name can be used in different schemas without conflict; for example, both schema1 and myschema can contain tables named mytable. Unlike databases, schemas are not rigidly separated: a user can access objects in any of the schemas in the database they are connected to, if they have privileges to do so.

There are several reasons why one might want to use schemas:

  • To allow many users to use one database without interfering with each other.

  • To organize database objects into logical groups to make them more manageable.

  • Third-party applications can be put into separate schemas so they do not collide with the names of other objects.

Schemas are analogous to directories at the operating system level, except that schemas cannot be nested.

5.9.1. Creating a Schema

To create a schema, use the CREATE SCHEMA command. Give the schema a name of your choice. For example:

CREATE SCHEMA myschema;

To create or access objects in a schema, write a qualified name consisting of the schema name and table name separated by a dot:

schema.table

This works anywhere a table name is expected, including the table modification commands and the data access commands discussed in the following chapters. (For brevity we will speak of tables only, but the same ideas apply to other kinds of named objects, such as types and functions.)

Actually, the even more general syntax

database.schema.table

can be used too, but at present this is just for pro forma compliance with the SQL standard. If you write a database name, it must be the same as the database you are connected to.

So to create a table in the new schema, use:

CREATE TABLE myschema.mytable (
 ...
);

To drop a schema if it's empty (all objects in it have been dropped), use:

DROP SCHEMA myschema;

To drop a schema including all contained objects, use:

DROP SCHEMA myschema CASCADE;

See Section 5.14 for a description of the general mechanism behind this.

Often you will want to create a schema owned by someone else (since this is one of the ways to restrict the activities of your users to well-defined namespaces). The syntax for that is:

CREATE SCHEMA schema_name AUTHORIZATION user_name;

You can even omit the schema name, in which case the schema name will be the same as the user name. See Section 5.9.6 for how this can be useful.

Schema names beginning with pg_ are reserved for system purposes and cannot be created by users.

5.9.2. The Public Schema

In the previous sections we created tables without specifying any schema names. By default such tables (and other objects) are automatically put into a schema named “public”. Every new database contains such a schema. Thus, the following are equivalent:

CREATE TABLE products ( ... );

and:

CREATE TABLE public.products ( ... );

5.9.3. The Schema Search Path

Qualified names are tedious to write, and it's often best not to wire a particular schema name into applications anyway. Therefore tables are often referred to by unqualified names, which consist of just the table name. The system determines which table is meant by following a search path, which is a list of schemas to look in. The first matching table in the search path is taken to be the one wanted. If there is no match in the search path, an error is reported, even if matching table names exist in other schemas in the database.

The ability to create like-named objects in different schemas complicates writing a query that references precisely the same objects every time. It also opens up the potential for users to change the behavior of other users' queries, maliciously or accidentally. Due to the prevalence of unqualified names in queries and their use in PostgreSQL internals, adding a schema to search_path effectively trusts all users having CREATE privilege on that schema. When you run an ordinary query, a malicious user able to create objects in a schema of your search path can take control and execute arbitrary SQL functions as though you executed them.

The first schema named in the search path is called the current schema. Aside from being the first schema searched, it is also the schema in which new tables will be created if the CREATE TABLE command does not specify a schema name.

To show the current search path, use the following command:

SHOW search_path;

In the default setup this returns:

 search_path
--------------
 "$user", public

The first element specifies that a schema with the same name as the current user is to be searched. If no such schema exists, the entry is ignored. The second element refers to the public schema that we have seen already.

The first schema in the search path that exists is the default location for creating new objects. That is the reason that by default objects are created in the public schema. When objects are referenced in any other context without schema qualification (table modification, data modification, or query commands) the search path is traversed until a matching object is found. Therefore, in the default configuration, any unqualified access again can only refer to the public schema.

To put our new schema in the path, we use:

SET search_path TO myschema,public;

(We omit the $user here because we have no immediate need for it.) And then we can access the table without schema qualification:

DROP TABLE mytable;

Also, since myschema is the first element in the path, new objects would by default be created in it.

We could also have written:

SET search_path TO myschema;

Then we no longer have access to the public schema without explicit qualification. There is nothing special about the public schema except that it exists by default. It can be dropped, too.

See also Section 9.26 for other ways to manipulate the schema search path.

The search path works in the same way for data type names, function names, and operator names as it does for table names. Data type and function names can be qualified in exactly the same way as table names. If you need to write a qualified operator name in an expression, there is a special provision: you must write

OPERATOR(schema.operator)

This is needed to avoid syntactic ambiguity. An example is:

SELECT 3 OPERATOR(pg_catalog.+) 4;

In practice one usually relies on the search path for operators, so as not to have to write anything so ugly as that.

5.9.4. Schemas and Privileges

By default, users cannot access any objects in schemas they do not own. To allow that, the owner of the schema must grant the USAGE privilege on the schema. To allow users to make use of the objects in the schema, additional privileges might need to be granted, as appropriate for the object.

A user can also be allowed to create objects in someone else's schema. To allow that, the CREATE privilege on the schema needs to be granted. Note that by default, everyone has CREATE and USAGE privileges on the schema public. This allows all users that are able to connect to a given database to create objects in its public schema. Some usage patterns call for revoking that privilege:

REVOKE CREATE ON SCHEMA public FROM PUBLIC;

(The first “public” is the schema, the second “public” means “every user”. In the first sense it is an identifier, in the second sense it is a key word, hence the different capitalization; recall the guidelines from Section 4.1.1.)

5.9.5. The System Catalog Schema

In addition to public and user-created schemas, each database contains a pg_catalog schema, which contains the system tables and all the built-in data types, functions, and operators. pg_catalog is always effectively part of the search path. If it is not named explicitly in the path then it is implicitly searched before searching the path's schemas. This ensures that built-in names will always be findable. However, you can explicitly place pg_catalog at the end of your search path if you prefer to have user-defined names override built-in names.

Since system table names begin with pg_, it is best to avoid such names to ensure that you won't suffer a conflict if some future version defines a system table named the same as your table. (With the default search path, an unqualified reference to your table name would then be resolved as the system table instead.) System tables will continue to follow the convention of having names beginning with pg_, so that they will not conflict with unqualified user-table names so long as users avoid the pg_ prefix.

5.9.6. Usage Patterns

Schemas can be used to organize your data in many ways. A secure schema usage pattern prevents untrusted users from changing the behavior of other users' queries. When a database does not use a secure schema usage pattern, users wishing to securely query that database would take protective action at the beginning of each session. Specifically, they would begin each session by setting search_path to the empty string or otherwise removing non-superuser-writable schemas from search_path. There are a few usage patterns easily supported by the default configuration:

  • Constrain ordinary users to user-private schemas. To implement this, issue REVOKE CREATE ON SCHEMA public FROM PUBLIC, and create a schema for each user with the same name as that user. Recall that the default search path starts with $user, which resolves to the user name. Therefore, if each user has a separate schema, they access their own schemas by default. After adopting this pattern in a database where untrusted users had already logged in, consider auditing the public schema for objects named like objects in schema pg_catalog. This pattern is a secure schema usage pattern unless an untrusted user is the database owner or holds the CREATEROLE privilege, in which case no secure schema usage pattern exists.

  • Remove the public schema from the default search path, by modifying postgresql.conf or by issuing ALTER ROLE ALL SET search_path = "$user". Everyone retains the ability to create objects in the public schema, but only qualified names will choose those objects. While qualified table references are fine, calls to functions in the public schema will be unsafe or unreliable. If you create functions or extensions in the public schema, use the first pattern instead. Otherwise, like the first pattern, this is secure unless an untrusted user is the database owner or holds the CREATEROLE privilege.

  • Keep the default. All users access the public schema implicitly. This simulates the situation where schemas are not available at all, giving a smooth transition from the non-schema-aware world. However, this is never a secure pattern. It is acceptable only when the database has a single user or a few mutually-trusting users.

For any pattern, to install shared applications (tables to be used by everyone, additional functions provided by third parties, etc.), put them into separate schemas. Remember to grant appropriate privileges to allow the other users to access them. Users can then refer to these additional objects by qualifying the names with a schema name, or they can put the additional schemas into their search path, as they choose.

5.9.7. Portability

In the SQL standard, the notion of objects in the same schema being owned by different users does not exist. Moreover, some implementations do not allow you to create schemas that have a different name than their owner. In fact, the concepts of schema and user are nearly equivalent in a database system that implements only the basic schema support specified in the standard. Therefore, many users consider qualified names to really consist of user_name.table_name. This is how PostgreSQL will effectively behave if you create a per-user schema for every user.

Also, there is no concept of a public schema in the SQL standard. For maximum conformance to the standard, you should not use the public schema.

Of course, some SQL database systems might not implement schemas at all, or provide namespace support by allowing (possibly limited) cross-database access. If you need to work with those systems, then maximum portability would be achieved by not using schemas at all.

Prev | Up | Next
5.8. Row Security Policies | Home | 5.10. Inheritance


5.10. Inheritance
Prev Up Chapter 5. Data Definition Home Next

5.10. Inheritance

5.10.1. Caveats

PostgreSQL implements table inheritance, which can be a useful tool for database designers. (SQL:1999 and later define a type inheritance feature, which differs in many respects from the features described here.)

Let's start with an example: suppose we are trying to build a data model for cities. Each state has many cities, but only one capital. We want to be able to quickly retrieve the capital city for any particular state. This can be done by creating two tables, one for state capitals and one for cities that are not capitals. However, what happens when we want to ask for data about a city, regardless of whether it is a capital or not? The inheritance feature can help to resolve this problem. We define the capitals table so that it inherits from cities:

CREATE TABLE cities (
    name            text,
    population      float,
    elevation       int     -- in feet
);

CREATE TABLE capitals (
    state           char(2)
) INHERITS (cities);

In this case, the capitals table inherits all the columns of its parent table, cities. State capitals also have an extra column, state, that shows their state.

In PostgreSQL, a table can inherit from zero or more other tables, and a query can reference either all rows of a table or all rows of a table plus all of its descendant tables. The latter behavior is the default. For example, the following query finds the names of all cities, including state capitals, that are located at an elevation over 500 feet:

SELECT name, elevation
    FROM cities
    WHERE elevation > 500;

Given the sample data from the PostgreSQL tutorial (see Section 2.1), this returns:

   name    | elevation
-----------+-----------
 Las Vegas |      2174
 Mariposa  |      1953
 Madison   |       845

On the other hand, the following query finds all the cities that are not state capitals and are situated at an elevation over 500 feet:

SELECT name, elevation
    FROM ONLY cities
    WHERE elevation > 500;

   name    | elevation
-----------+-----------
 Las Vegas |      2174
 Mariposa  |      1953

Here the ONLY keyword indicates that the query should apply only to cities, and not any tables below cities in the inheritance hierarchy. Many of the commands that we have already discussed — SELECT, UPDATE and DELETE — support the ONLY keyword.

You can also write the table name with a trailing * to explicitly specify that descendant tables are included:

SELECT name, elevation
    FROM cities*
    WHERE elevation > 500;

Writing * is not necessary, since this behavior is always the default. However, this syntax is still supported for compatibility with older releases where the default could be changed.

In some cases you might wish to know which table a particular row originated from. There is a system column called tableoid in each table which can tell you the originating table:

SELECT c.tableoid, c.name, c.elevation
FROM cities c
WHERE c.elevation > 500;

which returns:

 tableoid |   name    | elevation
----------+-----------+-----------
   139793 | Las Vegas |      2174
   139793 | Mariposa  |      1953
   139798 | Madison   |       845

(If you try to reproduce this example, you will probably get different numeric OIDs.) By doing a join with pg_class you can see the actual table names:

SELECT p.relname, c.name, c.elevation
FROM cities c, pg_class p
WHERE c.elevation > 500 AND c.tableoid = p.oid;

which returns:

 relname  |   name    | elevation
----------+-----------+-----------
 cities   | Las Vegas |      2174
 cities   | Mariposa  |      1953
 capitals | Madison   |       845

Another way to get the same effect is to use the regclass alias type, which will print the table OID symbolically:

SELECT c.tableoid::regclass, c.name, c.elevation
FROM cities c
WHERE c.elevation > 500;

Inheritance does not automatically propagate data from INSERT or COPY commands to other tables in the inheritance hierarchy. In our example, the following INSERT statement will fail:

INSERT INTO cities (name, population, elevation, state)
VALUES ('Albany', NULL, NULL, 'NY');

We might hope that the data would somehow be routed to the capitals table, but this does not happen: INSERT always inserts into exactly the table specified. In some cases it is possible to redirect the insertion using a rule (see Chapter 41). However that does not help for the above case because the cities table does not contain the column state, and so the command will be rejected before the rule can be applied.

All check constraints and not-null constraints on a parent table are automatically inherited by its children, unless explicitly specified otherwise with NO INHERIT clauses. Other types of constraints (unique, primary key, and foreign key constraints) are not inherited.

A table can inherit from more than one parent table, in which case it has the union of the columns defined by the parent tables. Any columns declared in the child table's definition are added to these. If the same column name appears in multiple parent tables, or in both a parent table and the child's definition, then these columns are “merged” so that there is only one such column in the child table. To be merged, columns must have the same data types, else an error is raised. Inheritable check constraints and not-null constraints are merged in a similar fashion. Thus, for example, a merged column will be marked not-null if any one of the column definitions it came from is marked not-null. Check constraints are merged if they have the same name, and the merge will fail if their conditions are different.

Table inheritance is typically established when the child table is created, using the INHERITS clause of the CREATE TABLE statement. Alternatively, a table which is already defined in a compatible way can have a new parent relationship added, using the INHERIT variant of ALTER TABLE. To do this the new child table must already include columns with the same names and types as the columns of the parent. It must also include check constraints with the same names and check expressions as those of the parent. Similarly an inheritance link can be removed from a child using the NO INHERIT variant of ALTER TABLE. Dynamically adding and removing inheritance links like this can be useful when the inheritance relationship is being used for table partitioning (see Section 5.11).

One convenient way to create a compatible table that will later be made a new child is to use the LIKE clause in CREATE TABLE. This creates a new table with the same columns as the source table. If there are any CHECK constraints defined on the source table, the INCLUDING CONSTRAINTS option to LIKE should be specified, as the new child must have constraints matching the parent to be considered compatible.

A parent table cannot be dropped while any of its children remain. Neither can columns or check constraints of child tables be dropped or altered if they are inherited from any parent tables. If you wish to remove a table and all of its descendants, one easy way is to drop the parent table with the CASCADE option (see Section 5.14).

ALTER TABLE will propagate any changes in column data definitions and check constraints down the inheritance hierarchy. Again, dropping columns that are depended on by other tables is only possible when using the CASCADE option. ALTER TABLE follows the same rules for duplicate column merging and rejection that apply during CREATE TABLE.

Inherited queries perform access permission checks on the parent table only. Thus, for example, granting UPDATE permission on the cities table implies permission to update rows in the capitals table as well, when they are accessed through cities. This preserves the appearance that the data is (also) in the parent table. But the capitals table could not be updated directly without an additional grant. In a similar way, the parent table's row security policies (see Section 5.8) are applied to rows coming from child tables during an inherited query. A child table's policies, if any, are applied only when it is the table explicitly named in the query; and in that case, any policies attached to its parent(s) are ignored.

Foreign tables (see Section 5.12) can also be part of inheritance hierarchies, either as parent or child tables, just as regular tables can be. If a foreign table is part of an inheritance hierarchy then any operations not supported by the foreign table are not supported on the whole hierarchy either.

5.10.1. Caveats

Note that not all SQL commands are able to work on inheritance hierarchies. Commands that are used for data querying, data modification, or schema modification (e.g., SELECT, UPDATE, DELETE, most variants of ALTER TABLE, but not INSERT or ALTER TABLE ... RENAME) typically default to including child tables and support the ONLY notation to exclude them. Commands that do database maintenance and tuning (e.g., REINDEX, VACUUM) typically only work on individual, physical tables and do not support recursing over inheritance hierarchies. The respective behavior of each individual command is documented in its reference page (SQL Commands).

A serious limitation of the inheritance feature is that indexes (including unique constraints) and foreign key constraints only apply to single tables, not to their inheritance children. This is true on both the referencing and referenced sides of a foreign key constraint. Thus, in the terms of the above example:

  • If we declared cities.name to be UNIQUE or a PRIMARY KEY, this would not stop the capitals table from having rows with names duplicating rows in cities. And those duplicate rows would by default show up in queries from cities. In fact, by default capitals would have no unique constraint at all, and so could contain multiple rows with the same name. You could add a unique constraint to capitals, but this would not prevent duplication compared to cities.

  • Similarly, if we were to specify that cities.name REFERENCES some other table, this constraint would not automatically propagate to capitals. In this case you could work around it by manually adding the same REFERENCES constraint to capitals.

  • Specifying that another table's column REFERENCES cities(name) would allow the other table to contain city names, but not capital names. There is no good workaround for this case.

Some functionality not implemented for inheritance hierarchies is implemented for declarative partitioning. Considerable care is needed in deciding whether partitioning with legacy inheritance is useful for your application.

Prev | Up | Next
5.9. Schemas | Home | 5.11. Table Partitioning


5.11. Table Partitioning
Prev Up Chapter 5. Data Definition Home Next

5.11. Table Partitioning

5.11.1. Overview
5.11.2. Declarative Partitioning
5.11.3. Partitioning Using Inheritance
5.11.4. Partition Pruning
5.11.5. Partitioning and Constraint Exclusion
5.11.6. Best Practices for Declarative Partitioning

PostgreSQL supports basic table partitioning. This section describes why and how to implement partitioning as part of your database design.

5.11.1. Overview

Partitioning refers to splitting what is logically one large table into smaller physical pieces. Partitioning can provide several benefits:

  • Query performance can be improved dramatically in certain situations, particularly when most of the heavily accessed rows of the table are in a single partition or a small number of partitions. Partitioning effectively substitutes for the upper tree levels of indexes, making it more likely that the heavily-used parts of the indexes fit in memory.

  • When queries or updates access a large percentage of a single partition, performance can be improved by using a sequential scan of that partition instead of using an index, which would require random-access reads scattered across the whole table.

  • Bulk loads and deletes can be accomplished by adding or removing partitions, if the usage pattern is accounted for in the partitioning design. Dropping an individual partition using DROP TABLE, or doing ALTER TABLE DETACH PARTITION, is far faster than a bulk operation. These commands also entirely avoid the VACUUM overhead caused by a bulk DELETE.

  • Seldom-used data can be migrated to cheaper and slower storage media.

These benefits will normally be worthwhile only when a table would otherwise be very large. The exact point at which a table will benefit from partitioning depends on the application, although a rule of thumb is that the size of the table should exceed the physical memory of the database server.

PostgreSQL offers built-in support for the following forms of partitioning:

Range Partitioning

The table is partitioned into “ranges” defined by a key column or set of columns, with no overlap between the ranges of values assigned to different partitions. For example, one might partition by date ranges, or by ranges of identifiers for particular business objects. Each range's bounds are understood as being inclusive at the lower end and exclusive at the upper end. For example, if one partition's range is from 1 to 10, and the next one's range is from 10 to 20, then value 10 belongs to the second partition not the first.

List Partitioning

The table is partitioned by explicitly listing which key value(s) appear in each partition.

Hash Partitioning

The table is partitioned by specifying a modulus and a remainder for each partition. Each partition will hold the rows for which the hash value of the partition key divided by the specified modulus will produce the specified remainder.

If your application needs to use other forms of partitioning not listed above, alternative methods such as inheritance and UNION ALL views can be used instead. Such methods offer flexibility but do not have some of the performance benefits of built-in declarative partitioning.

5.11.2. Declarative Partitioning

PostgreSQL allows you to declare that a table is divided into partitions. The table that is divided is referred to as a partitioned table. The declaration includes the partitioning method as described above, plus a list of columns or expressions to be used as the partition key.

The partitioned table itself is a “virtual” table having no storage of its own. Instead, the storage belongs to partitions, which are otherwise-ordinary tables associated with the partitioned table. Each partition stores a subset of the data as defined by its partition bounds. All rows inserted into a partitioned table will be routed to the appropriate one of the partitions based on the values of the partition key column(s). Updating the partition key of a row will cause it to be moved into a different partition if it no longer satisfies the partition bounds of its original partition.

Partitions may themselves be defined as partitioned tables, resulting in sub-partitioning. Although all partitions must have the same columns as their partitioned parent, partitions may have their own indexes, constraints and default values, distinct from those of other partitions. See CREATE TABLE for more details on creating partitioned tables and partitions.

It is not possible to turn a regular table into a partitioned table or vice versa. However, it is possible to add an existing regular or partitioned table as a partition of a partitioned table, or remove a partition from a partitioned table turning it into a standalone table; this can simplify and speed up many maintenance processes. See ALTER TABLE to learn more about the ATTACH PARTITION and DETACH PARTITION sub-commands.

Partitions can also be foreign tables, although considerable care is needed because it is then the user's responsibility that the contents of the foreign table satisfy the partitioning rule. There are some other restrictions as well. See CREATE FOREIGN TABLE for more information.

5.11.2.1. Example

Suppose we are constructing a database for a large ice cream company. The company measures peak temperatures every day as well as ice cream sales in each region. Conceptually, we want a table like:

CREATE TABLE measurement (
    city_id         int not null,
    logdate         date not null,
    peaktemp        int,
    unitsales       int
);

We know that most queries will access just the last week's, month's or quarter's data, since the main use of this table will be to prepare online reports for management. To reduce the amount of old data that needs to be stored, we decide to keep only the most recent 3 years worth of data. At the beginning of each month we will remove the oldest month's data. In this situation we can use partitioning to help us meet all of our different requirements for the measurements table.

To use declarative partitioning in this case, use the following steps:

  1. Create the measurement table as a partitioned table by specifying the PARTITION BY clause, which includes the partitioning method (RANGE in this case) and the list of column(s) to use as the partition key.

    CREATE TABLE measurement (
    city_id         int not null,
    logdate         date not null,
    peaktemp        int,
    unitsales       int
) PARTITION BY RANGE (logdate);

  2. Create partitions. Each partition's definition must specify bounds that correspond to the partitioning method and partition key of the parent. Note that specifying bounds such that the new partition's values would overlap with those in one or more existing partitions will cause an error.

    Partitions thus created are in every way normal PostgreSQL tables (or, possibly, foreign tables). It is possible to specify a tablespace and storage parameters for each partition separately.

    For our example, each partition should hold one month's worth of data, to match the requirement of deleting one month's data at a time. So the commands might look like:

    CREATE TABLE measurement_y2006m02 PARTITION OF measurement
    FOR VALUES FROM ('2006-02-01') TO ('2006-03-01');

CREATE TABLE measurement_y2006m03 PARTITION OF measurement
    FOR VALUES FROM ('2006-03-01') TO ('2006-04-01');

...
CREATE TABLE measurement_y2007m11 PARTITION OF measurement
    FOR VALUES FROM ('2007-11-01') TO ('2007-12-01');

CREATE TABLE measurement_y2007m12 PARTITION OF measurement
    FOR VALUES FROM ('2007-12-01') TO ('2008-01-01')
    TABLESPACE fasttablespace;

CREATE TABLE measurement_y2008m01 PARTITION OF measurement
    FOR VALUES FROM ('2008-01-01') TO ('2008-02-01')
    WITH (parallel_workers = 4)
    TABLESPACE fasttablespace;

    (Recall that adjacent partitions can share a bound value, since range upper bounds are treated as exclusive bounds.)

    If you wish to implement sub-partitioning, again specify the PARTITION BY clause in the commands used to create individual partitions, for example:

    CREATE TABLE measurement_y2006m02 PARTITION OF measurement
    FOR VALUES FROM ('2006-02-01') TO ('2006-03-01')
    PARTITION BY RANGE (peaktemp);

    After creating partitions of measurement_y2006m02, any data inserted into measurement that is mapped to measurement_y2006m02 (or data that is directly inserted into measurement_y2006m02, which is allowed provided its partition constraint is satisfied) will be further redirected to one of its partitions based on the peaktemp column. The partition key specified may overlap with the parent's partition key, although care should be taken when specifying the bounds of a sub-partition such that the set of data it accepts constitutes a subset of what the partition's own bounds allow; the system does not try to check whether that's really the case.

    Inserting data into the parent table that does not map to one of the existing partitions will cause an error; an appropriate partition must be added manually.

    It is not necessary to manually create table constraints describing the partition boundary conditions for partitions. Such constraints will be created automatically.

  3. Create an index on the key column(s), as well as any other indexes you might want, on the partitioned table. (The key index is not strictly necessary, but in most scenarios it is helpful.) This automatically creates a matching index on each partition, and any partitions you create or attach later will also have such an index. An index or unique constraint declared on a partitioned table is “virtual” in the same way that the partitioned table is: the actual data is in child indexes on the individual partition tables.

    CREATE INDEX ON measurement (logdate);

  4. Ensure that the enable_partition_pruning configuration parameter is not disabled in postgresql.conf. If it is, queries will not be optimized as desired.

In the above example we would be creating a new partition each month, so it might be wise to write a script that generates the required DDL automatically.

5.11.2.2. Partition Maintenance

Normally the set of partitions established when initially defining the table is not intended to remain static. It is common to want to remove partitions holding old data and periodically add new partitions for new data. One of the most important advantages of partitioning is precisely that it allows this otherwise painful task to be executed nearly instantaneously by manipulating the partition structure, rather than physically moving large amounts of data around.

The simplest option for removing old data is to drop the partition that is no longer necessary:

DROP TABLE measurement_y2006m02;

This can very quickly delete millions of records because it doesn't have to individually delete every record. Note however that the above command requires taking an ACCESS EXCLUSIVE lock on the parent table.

Another option that is often preferable is to remove the partition from the partitioned table but retain access to it as a table in its own right. This has two forms:

ALTER TABLE measurement DETACH PARTITION measurement_y2006m02;
ALTER TABLE measurement DETACH PARTITION measurement_y2006m02 CONCURRENTLY;

These allow further operations to be performed on the data before it is dropped. For example, this is often a useful time to back up the data using COPY, pg_dump, or similar tools. It might also be a useful time to aggregate data into smaller formats, perform other data manipulations, or run reports. The first form of the command requires an ACCESS EXCLUSIVE lock on the parent table. Adding the CONCURRENTLY qualifier as in the second form allows the detach operation to require only SHARE UPDATE EXCLUSIVE lock on the parent table, but see ALTER TABLE ... DETACH PARTITION for details on the restrictions.

Similarly we can add a new partition to handle new data. We can create an empty partition in the partitioned table just as the original partitions were created above:

CREATE TABLE measurement_y2008m02 PARTITION OF measurement
    FOR VALUES FROM ('2008-02-01') TO ('2008-03-01')
    TABLESPACE fasttablespace;

As an alternative, it is sometimes more convenient to create the new table outside the partition structure, and make it a proper partition later. This allows new data to be loaded, checked, and transformed prior to it appearing in the partitioned table. The CREATE TABLE ... LIKE option is helpful to avoid tediously repeating the parent table's definition:

CREATE TABLE measurement_y2008m02
  (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS)
  TABLESPACE fasttablespace;

ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02
   CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' );

\copy measurement_y2008m02 from 'measurement_y2008m02'
-- possibly some other data preparation work

ALTER TABLE measurement ATTACH PARTITION measurement_y2008m02
    FOR VALUES FROM ('2008-02-01') TO ('2008-03-01' );

The ATTACH PARTITION command requires taking a SHARE UPDATE EXCLUSIVE lock on the partitioned table.

Before running the ATTACH PARTITION command, it is recommended to create a CHECK constraint on the table to be attached that matches the expected partition constraint, as illustrated above. That way, the system will be able to skip the scan which is otherwise needed to validate the implicit partition constraint. Without the CHECK constraint, the table will be scanned to validate the partition constraint while holding an ACCESS EXCLUSIVE lock on that partition. It is recommended to drop the now-redundant CHECK constraint after the ATTACH PARTITION is complete. If the table being attached is itself a partitioned table, then each of its sub-partitions will be recursively locked and scanned until either a suitable CHECK constraint is encountered or the leaf partitions are reached.

Similarly, if the partitioned table has a DEFAULT partition, it is recommended to create a CHECK constraint which excludes the to-be-attached partition's constraint. If this is not done then the DEFAULT partition will be scanned to verify that it contains no records which should be located in the partition being attached. This operation will be performed whilst holding an ACCESS EXCLUSIVE lock on the DEFAULT partition. If the DEFAULT partition is itself a partitioned table, then each of its partitions will be recursively checked in the same way as the table being attached, as mentioned above.

As explained above, it is possible to create indexes on partitioned tables so that they are applied automatically to the entire hierarchy. This is very convenient, as not only will the existing partitions become indexed, but also any partitions that are created in the future will. One limitation is that it's not possible to use the CONCURRENTLY qualifier when creating such a partitioned index. To avoid long lock times, it is possible to use CREATE INDEX ON ONLY the partitioned table; such an index is marked invalid, and the partitions do not get the index applied automatically. The indexes on partitions can be created individually using CONCURRENTLY, and then attached to the index on the parent using ALTER INDEX .. ATTACH PARTITION. Once indexes for all partitions are attached to the parent index, the parent index is marked valid automatically. Example:

CREATE INDEX measurement_usls_idx ON ONLY measurement (unitsales);

CREATE INDEX measurement_usls_200602_idx
    ON measurement_y2006m02 (unitsales);
ALTER INDEX measurement_usls_idx
    ATTACH PARTITION measurement_usls_200602_idx;
...

This technique can be used with UNIQUE and PRIMARY KEY constraints too; the indexes are created implicitly when the constraint is created. Example:

ALTER TABLE ONLY measurement ADD UNIQUE (city_id, logdate);

ALTER TABLE measurement_y2006m02 ADD UNIQUE (city_id, logdate);
ALTER INDEX measurement_city_id_logdate_key
    ATTACH PARTITION measurement_y2006m02_city_id_logdate_key;
...

5.11.2.3. Limitations

The following limitations apply to partitioned tables:

  • Unique constraints (and hence primary keys) on partitioned tables must include all the partition key columns. This limitation exists because the individual indexes making up the constraint can only directly enforce uniqueness within their own partitions; therefore, the partition structure itself must guarantee that there are not duplicates in different partitions.

  • There is no way to create an exclusion constraint spanning the whole partitioned table. It is only possible to put such a constraint on each leaf partition individually. Again, this limitation stems from not being able to enforce cross-partition restrictions.

  • BEFORE ROW triggers on INSERT cannot change which partition is the final destination for a new row.

  • Mixing temporary and permanent relations in the same partition tree is not allowed. Hence, if the partitioned table is permanent, so must be its partitions and likewise if the partitioned table is temporary. When using temporary relations, all members of the partition tree have to be from the same session.

Individual partitions are linked to their partitioned table using inheritance behind-the-scenes. However, it is not possible to use all of the generic features of inheritance with declaratively partitioned tables or their partitions, as discussed below. Notably, a partition cannot have any parents other than the partitioned table it is a partition of, nor can a table inherit from both a partitioned table and a regular table. That means partitioned tables and their partitions never share an inheritance hierarchy with regular tables.

Since a partition hierarchy consisting of the partitioned table and its partitions is still an inheritance hierarchy, tableoid and all the normal rules of inheritance apply as described in Section 5.10, with a few exceptions:

  • Partitions cannot have columns that are not present in the parent. It is not possible to specify columns when creating partitions with CREATE TABLE, nor is it possible to add columns to partitions after-the-fact using ALTER TABLE. Tables may be added as a partition with ALTER TABLE ... ATTACH PARTITION only if their columns exactly match the parent.

  • Both CHECK and NOT NULL constraints of a partitioned table are always inherited by all its partitions. CHECK constraints that are marked NO INHERIT are not allowed to be created on partitioned tables. You cannot drop a NOT NULL constraint on a partition's column if the same constraint is present in the parent table.

  • Using ONLY to add or drop a constraint on only the partitioned table is supported as long as there are no partitions. Once partitions exist, using ONLY will result in an error. Instead, constraints on the partitions themselves can be added and (if they are not present in the parent table) dropped.

  • As a partitioned table does not have any data itself, attempts to use TRUNCATE ONLY on a partitioned table will always return an error.

5.11.3. Partitioning Using Inheritance

While the built-in declarative partitioning is suitable for most common use cases, there are some circumstances where a more flexible approach may be useful. Partitioning can be implemented using table inheritance, which allows for several features not supported by declarative partitioning, such as:

  • For declarative partitioning, partitions must have exactly the same set of columns as the partitioned table, whereas with table inheritance, child tables may have extra columns not present in the parent.

  • Table inheritance allows for multiple inheritance.

  • Declarative partitioning only supports range, list and hash partitioning, whereas table inheritance allows data to be divided in a manner of the user's choosing. (Note, however, that if constraint exclusion is unable to prune child tables effectively, query performance might be poor.)

5.11.3.1. Example

This example builds a partitioning structure equivalent to the declarative partitioning example above. Use the following steps:

  1. Create the “root” table, from which all of the “child” tables will inherit. This table will contain no data. Do not define any check constraints on this table, unless you intend them to be applied equally to all child tables. There is no point in defining any indexes or unique constraints on it, either. For our example, the root table is the measurement table as originally defined:

    CREATE TABLE measurement (
    city_id         int not null,
    logdate         date not null,
    peaktemp        int,
    unitsales       int
);

  2. Create several “child” tables that each inherit from the root table. Normally, these tables will not add any columns to the set inherited from the root. Just as with declarative partitioning, these tables are in every way normal PostgreSQL tables (or foreign tables).

    CREATE TABLE measurement_y2006m02 () INHERITS (measurement);
CREATE TABLE measurement_y2006m03 () INHERITS (measurement);
...
CREATE TABLE measurement_y2007m11 () INHERITS (measurement);
CREATE TABLE measurement_y2007m12 () INHERITS (measurement);
CREATE TABLE measurement_y2008m01 () INHERITS (measurement);

  3. Add non-overlapping table constraints to the child tables to define the allowed key values in each.

    Typical examples would be:

    CHECK ( x = 1 )
CHECK ( county IN ( 'Oxfordshire', 'Buckinghamshire', 'Warwickshire' ))
CHECK ( outletID >= 100 AND outletID < 200 )

    Ensure that the constraints guarantee that there is no overlap between the key values permitted in different child tables. A common mistake is to set up range constraints like:

    CHECK ( outletID BETWEEN 100 AND 200 )
CHECK ( outletID BETWEEN 200 AND 300 )

    This is wrong since it is not clear which child table the key value 200 belongs in. Instead, ranges should be defined in this style:

    CREATE TABLE measurement_y2006m02 (
    CHECK ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' )
) INHERITS (measurement);

CREATE TABLE measurement_y2006m03 (
    CHECK ( logdate >= DATE '2006-03-01' AND logdate < DATE '2006-04-01' )
) INHERITS (measurement);

...
CREATE TABLE measurement_y2007m11 (
    CHECK ( logdate >= DATE '2007-11-01' AND logdate < DATE '2007-12-01' )
) INHERITS (measurement);

CREATE TABLE measurement_y2007m12 (
    CHECK ( logdate >= DATE '2007-12-01' AND logdate < DATE '2008-01-01' )
) INHERITS (measurement);

CREATE TABLE measurement_y2008m01 (
    CHECK ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' )
) INHERITS (measurement);

  4. For each child table, create an index on the key column(s), as well as any other indexes you might want.

    CREATE INDEX measurement_y2006m02_logdate ON measurement_y2006m02 (logdate);
CREATE INDEX measurement_y2006m03_logdate ON measurement_y2006m03 (logdate);
CREATE INDEX measurement_y2007m11_logdate ON measurement_y2007m11 (logdate);
CREATE INDEX measurement_y2007m12_logdate ON measurement_y2007m12 (logdate);
CREATE INDEX measurement_y2008m01_logdate ON measurement_y2008m01 (logdate);

  5. We want our application to be able to say INSERT INTO measurement ... and have the data be redirected into the appropriate child table. We can arrange that by attaching a suitable trigger function to the root table. If data will be added only to the latest child, we can use a very simple trigger function:

    CREATE OR REPLACE FUNCTION measurement_insert_trigger()
RETURNS TRIGGER AS $$
BEGIN
    INSERT INTO measurement_y2008m01 VALUES (NEW.*);
    RETURN NULL;
END;
$$
LANGUAGE plpgsql;

    After creating the function, we create a trigger which calls the trigger function:

    CREATE TRIGGER insert_measurement_trigger
    BEFORE INSERT ON measurement
    FOR EACH ROW EXECUTE FUNCTION measurement_insert_trigger();

    We must redefine the trigger function each month so that it always inserts into the current child table. The trigger definition does not need to be updated, however.

    We might want to insert data and have the server automatically locate the child table into which the row should be added. We could do this with a more complex trigger function, for example:

    CREATE OR REPLACE FUNCTION measurement_insert_trigger()
RETURNS TRIGGER AS $$
BEGIN
    IF ( NEW.logdate >= DATE '2006-02-01' AND
         NEW.logdate < DATE '2006-03-01' ) THEN
        INSERT INTO measurement_y2006m02 VALUES (NEW.*);
    ELSIF ( NEW.logdate >= DATE '2006-03-01' AND
            NEW.logdate < DATE '2006-04-01' ) THEN
        INSERT INTO measurement_y2006m03 VALUES (NEW.*);
    ...
    ELSIF ( NEW.logdate >= DATE '2008-01-01' AND
            NEW.logdate < DATE '2008-02-01' ) THEN
        INSERT INTO measurement_y2008m01 VALUES (NEW.*);
    ELSE
        RAISE EXCEPTION 'Date out of range.  Fix the measurement_insert_trigger() function!';
    END IF;
    RETURN NULL;
END;
$$
LANGUAGE plpgsql;

    The trigger definition is the same as before. Note that each IF test must exactly match the CHECK constraint for its child table.

    While this function is more complex than the single-month case, it doesn't need to be updated as often, since branches can be added in advance of being needed.

    Note

    In practice, it might be best to check the newest child first, if most inserts go into that child. For simplicity, we have shown the trigger's tests in the same order as in other parts of this example.

    A different approach to redirecting inserts into the appropriate child table is to set up rules, instead of a trigger, on the root table. For example:

    CREATE RULE measurement_insert_y2006m02 AS
ON INSERT TO measurement WHERE
    ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' )
DO INSTEAD
    INSERT INTO measurement_y2006m02 VALUES (NEW.*);
...
CREATE RULE measurement_insert_y2008m01 AS
ON INSERT TO measurement WHERE
    ( logdate >= DATE '2008-01-01' AND logdate < DATE '2008-02-01' )
DO INSTEAD
    INSERT INTO measurement_y2008m01 VALUES (NEW.*);

    A rule has significantly more overhead than a trigger, but the overhead is paid once per query rather than once per row, so this method might be advantageous for bulk-insert situations. In most cases, however, the trigger method will offer better performance.

    Be aware that COPY ignores rules. If you want to use COPY to insert data, you'll need to copy into the correct child table rather than directly into the root. COPY does fire triggers, so you can use it normally if you use the trigger approach.

    Another disadvantage of the rule approach is that there is no simple way to force an error if the set of rules doesn't cover the insertion date; the data will silently go into the root table instead.

  6. Ensure that the constraint_exclusion configuration parameter is not disabled in postgresql.conf; otherwise child tables may be accessed unnecessarily.

As we can see, a complex table hierarchy could require a substantial amount of DDL. In the above example we would be creating a new child table each month, so it might be wise to write a script that generates the required DDL automatically.

5.11.3.2. Maintenance for Inheritance Partitioning

To remove old data quickly, simply drop the child table that is no longer necessary:

DROP TABLE measurement_y2006m02;

To remove the child table from the inheritance hierarchy table but retain access to it as a table in its own right:

ALTER TABLE measurement_y2006m02 NO INHERIT measurement;

To add a new child table to handle new data, create an empty child table just as the original children were created above:

CREATE TABLE measurement_y2008m02 (
    CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' )
) INHERITS (measurement);

Alternatively, one may want to create and populate the new child table before adding it to the table hierarchy. This could allow data to be loaded, checked, and transformed before being made visible to queries on the parent table.

CREATE TABLE measurement_y2008m02
  (LIKE measurement INCLUDING DEFAULTS INCLUDING CONSTRAINTS);
ALTER TABLE measurement_y2008m02 ADD CONSTRAINT y2008m02
   CHECK ( logdate >= DATE '2008-02-01' AND logdate < DATE '2008-03-01' );
\copy measurement_y2008m02 from 'measurement_y2008m02'
-- possibly some other data preparation work
ALTER TABLE measurement_y2008m02 INHERIT measurement;

5.11.3.3. Caveats

The following caveats apply to partitioning implemented using inheritance:

  • There is no automatic way to verify that all of the CHECK constraints are mutually exclusive. It is safer to create code that generates child tables and creates and/or modifies associated objects than to write each by hand.

  • Indexes and foreign key constraints apply to single tables and not to their inheritance children, hence they have some caveats to be aware of.

  • The schemes shown here assume that the values of a row's key column(s) never change, or at least do not change enough to require it to move to another partition. An UPDATE that attempts to do that will fail because of the CHECK constraints. If you need to handle such cases, you can put suitable update triggers on the child tables, but it makes management of the structure much more complicated.

  • If you are using manual VACUUM or ANALYZE commands, don't forget that you need to run them on each child table individually. A command like:

    ANALYZE measurement;

    will only process the root table.

  • INSERT statements with ON CONFLICT clauses are unlikely to work as expected, as the ON CONFLICT action is only taken in case of unique violations on the specified target relation, not its child relations.

  • Triggers or rules will be needed to route rows to the desired child table, unless the application is explicitly aware of the partitioning scheme. Triggers may be complicated to write, and will be much slower than the tuple routing performed internally by declarative partitioning.

5.11.4. Partition Pruning

Partition pruning is a query optimization technique that improves performance for declaratively partitioned tables. As an example:

SET enable_partition_pruning = on;                 -- the default
SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';

Without partition pruning, the above query would scan each of the partitions of the measurement table. With partition pruning enabled, the planner will examine the definition of each partition and prove that the partition need not be scanned because it could not contain any rows meeting the query's WHERE clause. When the planner can prove this, it excludes (prunes) the partition from the query plan.

By using the EXPLAIN command and the enable_partition_pruning configuration parameter, it's possible to show the difference between a plan for which partitions have been pruned and one for which they have not. A typical unoptimized plan for this type of table setup is:

SET enable_partition_pruning = off;
EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
                                    QUERY PLAN
-------------------------------------------------------------------​----------------
 Aggregate  (cost=188.76..188.77 rows=1 width=8)
   ->  Append  (cost=0.00..181.05 rows=3085 width=0)
         ->  Seq Scan on measurement_y2006m02  (cost=0.00..33.12 rows=617 width=0)
               Filter: (logdate >= '2008-01-01'::date)
         ->  Seq Scan on measurement_y2006m03  (cost=0.00..33.12 rows=617 width=0)
               Filter: (logdate >= '2008-01-01'::date)
...
         ->  Seq Scan on measurement_y2007m11  (cost=0.00..33.12 rows=617 width=0)
               Filter: (logdate >= '2008-01-01'::date)
         ->  Seq Scan on measurement_y2007m12  (cost=0.00..33.12 rows=617 width=0)
               Filter: (logdate >= '2008-01-01'::date)
         ->  Seq Scan on measurement_y2008m01  (cost=0.00..33.12 rows=617 width=0)
               Filter: (logdate >= '2008-01-01'::date)

Some or all of the partitions might use index scans instead of full-table sequential scans, but the point here is that there is no need to scan the older partitions at all to answer this query. When we enable partition pruning, we get a significantly cheaper plan that will deliver the same answer:

SET enable_partition_pruning = on;
EXPLAIN SELECT count(*) FROM measurement WHERE logdate >= DATE '2008-01-01';
                                    QUERY PLAN
-------------------------------------------------------------------​----------------
 Aggregate  (cost=37.75..37.76 rows=1 width=8)
   ->  Seq Scan on measurement_y2008m01  (cost=0.00..33.12 rows=617 width=0)
         Filter: (logdate >= '2008-01-01'::date)

Note that partition pruning is driven only by the constraints defined implicitly by the partition keys, not by the presence of indexes. Therefore it isn't necessary to define indexes on the key columns. Whether an index needs to be created for a given partition depends on whether you expect that queries that scan the partition will generally scan a large part of the partition or just a small part. An index will be helpful in the latter case but not the former.

Partition pruning can be performed not only during the planning of a given query, but also during its execution. This is useful as it can allow more partitions to be pruned when clauses contain expressions whose values are not known at query planning time, for example, parameters defined in a PREPARE statement, using a value obtained from a subquery, or using a parameterized value on the inner side of a nested loop join. Partition pruning during execution can be performed at any of the following times:

  • During initialization of the query plan. Partition pruning can be performed here for parameter values which are known during the initialization phase of execution. Partitions which are pruned during this stage will not show up in the query's EXPLAIN or EXPLAIN ANALYZE. It is possible to determine the number of partitions which were removed during this phase by observing the “Subplans Removed” property in the EXPLAIN output.

  • During actual execution of the query plan. Partition pruning may also be performed here to remove partitions using values which are only known during actual query execution. This includes values from subqueries and values from execution-time parameters such as those from parameterized nested loop joins. Since the value of these parameters may change many times during the execution of the query, partition pruning is performed whenever one of the execution parameters being used by partition pruning changes. Determining if partitions were pruned during this phase requires careful inspection of the loops property in the EXPLAIN ANALYZE output. Subplans corresponding to different partitions may have different values for it depending on how many times each of them was pruned during execution. Some may be shown as (never executed) if they were pruned every time.

Partition pruning can be disabled using the enable_partition_pruning setting.

5.11.5. Partitioning and Constraint Exclusion

Constraint exclusion is a query optimization technique similar to partition pruning. While it is primarily used for partitioning implemented using the legacy inheritance method, it can be used for other purposes, including with declarative partitioning.

Constraint exclusion works in a very similar way to partition pruning, except that it uses each table's CHECK constraints — which gives it its name — whereas partition pruning uses the table's partition bounds, which exist only in the case of declarative partitioning. Another difference is that constraint exclusion is only applied at plan time; there is no attempt to remove partitions at execution time.

The fact that constraint exclusion uses CHECK constraints, which makes it slow compared to partition pruning, can sometimes be used as an advantage: because constraints can be defined even on declaratively-partitioned tables, in addition to their internal partition bounds, constraint exclusion may be able to elide additional partitions from the query plan.

The default (and recommended) setting of constraint_exclusion is neither on nor off, but an intermediate setting called partition, which causes the technique to be applied only to queries that are likely to be working on inheritance partitioned tables. The on setting causes the planner to examine CHECK constraints in all queries, even simple ones that are unlikely to benefit.

The following caveats apply to constraint exclusion:

  • Constraint exclusion is only applied during query planning, unlike partition pruning, which can also be applied during query execution.

  • Constraint exclusion only works when the query's WHERE clause contains constants (or externally supplied parameters). For example, a comparison against a non-immutable function such as CURRENT_TIMESTAMP cannot be optimized, since the planner cannot know which child table the function's value might fall into at run time.

  • Keep the partitioning constraints simple, else the planner may not be able to prove that child tables might not need to be visited. Use simple equality conditions for list partitioning, or simple range tests for range partitioning, as illustrated in the preceding examples. A good rule of thumb is that partitioning constraints should contain only comparisons of the partitioning column(s) to constants using B-tree-indexable operators, because only B-tree-indexable column(s) are allowed in the partition key.

  • All constraints on all children of the parent table are examined during constraint exclusion, so large numbers of children are likely to increase query planning time considerably. So the legacy inheritance based partitioning will work well with up to perhaps a hundred child tables; don't try to use many thousands of children.

5.11.6. Best Practices for Declarative Partitioning

The choice of how to partition a table should be made carefully, as the performance of query planning and execution can be negatively affected by poor design.

One of the most critical design decisions will be the column or columns by which you partition your data. Often the best choice will be to partition by the column or set of columns which most commonly appear in WHERE clauses of queries being executed on the partitioned table. WHERE clauses that are compatible with the partition bound constraints can be used to prune unneeded partitions. However, you may be forced into making other decisions by requirements for the PRIMARY KEY or a UNIQUE constraint. Removal of unwanted data is also a factor to consider when planning your partitioning strategy. An entire partition can be detached fairly quickly, so it may be beneficial to design the partition strategy in such a way that all data to be removed at once is located in a single partition.

Choosing the target number of partitions that the table should be divided into is also a critical decision to make. Not having enough partitions may mean that indexes remain too large and that data locality remains poor which could result in low cache hit ratios. However, dividing the table into too many partitions can also cause issues. Too many partitions can mean longer query planning times and higher memory consumption during both query planning and execution, as further described below. When choosing how to partition your table, it's also important to consider what changes may occur in the future. For example, if you choose to have one partition per customer and you currently have a small number of large customers, consider the implications if in several years you instead find yourself with a large number of small customers. In this case, it may be better to choose to partition by HASH and choose a reasonable number of partitions rather than trying to partition by LIST and hoping that the number of customers does not increase beyond what it is practical to partition the data by.

Sub-partitioning can be useful to further divide partitions that are expected to become larger than other partitions. Another option is to use range partitioning with multiple columns in the partition key. Either of these can easily lead to excessive numbers of partitions, so restraint is advisable.

It is important to consider the overhead of partitioning during query planning and execution. The query planner is generally able to handle partition hierarchies with up to a few thousand partitions fairly well, provided that typical queries allow the query planner to prune all but a small number of partitions. Planning times become longer and memory consumption becomes higher when more partitions remain after the planner performs partition pruning. Another reason to be concerned about having a large number of partitions is that the server's memory consumption may grow significantly over time, especially if many sessions touch large numbers of partitions. That's because each partition requires its metadata to be loaded into the local memory of each session that touches it.

With data warehouse type workloads, it can make sense to use a larger number of partitions than with an OLTP type workload. Generally, in data warehouses, query planning time is less of a concern as the majority of processing time is spent during query execution. With either of these two types of workload, it is important to make the right decisions early, as re-partitioning large quantities of data can be painfully slow. Simulations of the intended workload are often beneficial for optimizing the partitioning strategy. Never just assume that more partitions are better than fewer partitions, nor vice-versa.

Prev | Up | Next
5.10. Inheritance | Home | 5.12. Foreign Data


5.12. Foreign Data
Prev Up Chapter 5. Data Definition Home Next

5.12. Foreign Data

PostgreSQL implements portions of the SQL/MED specification, allowing you to access data that resides outside PostgreSQL using regular SQL queries. Such data is referred to as foreign data. (Note that this usage is not to be confused with foreign keys, which are a type of constraint within the database.)

Foreign data is accessed with help from a foreign data wrapper. A foreign data wrapper is a library that can communicate with an external data source, hiding the details of connecting to the data source and obtaining data from it. There are some foreign data wrappers available as contrib modules; see Appendix F. Other kinds of foreign data wrappers might be found as third party products. If none of the existing foreign data wrappers suit your needs, you can write your own; see Chapter 57.

To access foreign data, you need to create a foreign server object, which defines how to connect to a particular external data source according to the set of options used by its supporting foreign data wrapper. Then you need to create one or more foreign tables, which define the structure of the remote data. A foreign table can be used in queries just like a normal table, but a foreign table has no storage in the PostgreSQL server. Whenever it is used, PostgreSQL asks the foreign data wrapper to fetch data from the external source, or transmit data to the external source in the case of update commands.

Accessing remote data may require authenticating to the external data source. This information can be provided by a user mapping, which can provide additional data such as user names and passwords based on the current PostgreSQL role.

For additional information, see CREATE FOREIGN DATA WRAPPER, CREATE SERVER, CREATE USER MAPPING, CREATE FOREIGN TABLE, and IMPORT FOREIGN SCHEMA.

Prev | Up | Next
5.11. Table Partitioning | Home | 5.13. Other Database Objects


5.13. Other Database Objects
Prev Up Chapter 5. Data Definition Home Next

5.13. Other Database Objects

Tables are the central objects in a relational database structure, because they hold your data. But they are not the only objects that exist in a database. Many other kinds of objects can be created to make the use and management of the data more efficient or convenient. They are not discussed in this chapter, but we give you a list here so that you are aware of what is possible:

  • Views

  • Functions, procedures, and operators

  • Data types and domains

  • Triggers and rewrite rules

Detailed information on these topics appears in Part V.

Prev | Up | Next
5.12. Foreign Data | Home | 5.14. Dependency Tracking


5.14. Dependency Tracking
Prev Up Chapter 5. Data Definition Home Next

5.14. Dependency Tracking

When you create complex database structures involving many tables with foreign key constraints, views, triggers, functions, etc. you implicitly create a net of dependencies between the objects. For instance, a table with a foreign key constraint depends on the table it references.

To ensure the integrity of the entire database structure, PostgreSQL makes sure that you cannot drop objects that other objects still depend on. For example, attempting to drop the products table we considered in Section 5.4.5, with the orders table depending on it, would result in an error message like this:

DROP TABLE products;

ERROR:  cannot drop table products because other objects depend on it
DETAIL:  constraint orders_product_no_fkey on table orders depends on table products
HINT:  Use DROP ... CASCADE to drop the dependent objects too.

The error message contains a useful hint: if you do not want to bother deleting all the dependent objects individually, you can run:

DROP TABLE products CASCADE;

and all the dependent objects will be removed, as will any objects that depend on them, recursively. In this case, it doesn't remove the orders table, it only removes the foreign key constraint. It stops there because nothing depends on the foreign key constraint. (If you want to check what DROP ... CASCADE will do, run DROP without CASCADE and read the DETAIL output.)

Almost all DROP commands in PostgreSQL support specifying CASCADE. Of course, the nature of the possible dependencies varies with the type of the object. You can also write RESTRICT instead of CASCADE to get the default behavior, which is to prevent dropping objects that any other objects depend on.

Note

According to the SQL standard, specifying either RESTRICT or CASCADE is required in a DROP command. No database system actually enforces that rule, but whether the default behavior is RESTRICT or CASCADE varies across systems.

If a DROP command lists multiple objects, CASCADE is only required when there are dependencies outside the specified group. For example, when saying DROP TABLE tab1, tab2 the existence of a foreign key referencing tab1 from tab2 would not mean that CASCADE is needed to succeed.

For user-defined functions, PostgreSQL tracks dependencies associated with a function's externally-visible properties, such as its argument and result types, but not dependencies that could only be known by examining the function body. As an example, consider this situation:

CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow',
                             'green', 'blue', 'purple');

CREATE TABLE my_colors (color rainbow, note text);

CREATE FUNCTION get_color_note (rainbow) RETURNS text AS
  'SELECT note FROM my_colors WHERE color = $1'
  LANGUAGE SQL;

(See Section 38.5 for an explanation of SQL-language functions.) PostgreSQL will be aware that the get_color_note function depends on the rainbow type: dropping the type would force dropping the function, because its argument type would no longer be defined. But PostgreSQL will not consider get_color_note to depend on the my_colors table, and so will not drop the function if the table is dropped. While there are disadvantages to this approach, there are also benefits. The function is still valid in some sense if the table is missing, though executing it would cause an error; creating a new table of the same name would allow the function to work again.

Prev | Up | Next
5.13. Other Database Objects | Home | Chapter 6. Data Manipulation


Chapter 6. Data Manipulation
Prev Up Part II. The SQL Language Home Next

Chapter 6. Data Manipulation

Table of Contents

6.1. Inserting Data
6.2. Updating Data
6.3. Deleting Data
6.4. Returning Data from Modified Rows

The previous chapter discussed how to create tables and other structures to hold your data. Now it is time to fill the tables with data. This chapter covers how to insert, update, and delete table data. The chapter after this will finally explain how to extract your long-lost data from the database.

Prev | Up | Next
5.14. Dependency Tracking | Home | 6.1. Inserting Data


6.1. Inserting Data
Prev Up Chapter 6. Data Manipulation Home Next

6.1. Inserting Data

When a table is created, it contains no data. The first thing to do before a database can be of much use is to insert data. Data is inserted one row at a time. You can also insert more than one row in a single command, but it is not possible to insert something that is not a complete row. Even if you know only some column values, a complete row must be created.

To create a new row, use the INSERT command. The command requires the table name and column values. For example, consider the products table from Chapter 5:

CREATE TABLE products (
    product_no integer,
    name text,
    price numeric
);

An example command to insert a row would be:

INSERT INTO products VALUES (1, 'Cheese', 9.99);

The data values are listed in the order in which the columns appear in the table, separated by commas. Usually, the data values will be literals (constants), but scalar expressions are also allowed.

The above syntax has the drawback that you need to know the order of the columns in the table. To avoid this you can also list the columns explicitly. For example, both of the following commands have the same effect as the one above:

INSERT INTO products (product_no, name, price) VALUES (1, 'Cheese', 9.99);
INSERT INTO products (name, price, product_no) VALUES ('Cheese', 9.99, 1);

Many users consider it good practice to always list the column names.

If you don't have values for all the columns, you can omit some of them. In that case, the columns will be filled with their default values. For example:

INSERT INTO products (product_no, name) VALUES (1, 'Cheese');
INSERT INTO products VALUES (1, 'Cheese');

The second form is a PostgreSQL extension. It fills the columns from the left with as many values as are given, and the rest will be defaulted.

For clarity, you can also request default values explicitly, for individual columns or for the entire row:

INSERT INTO products (product_no, name, price) VALUES (1, 'Cheese', DEFAULT);
INSERT INTO products DEFAULT VALUES;

You can insert multiple rows in a single command:

INSERT INTO products (product_no, name, price) VALUES
    (1, 'Cheese', 9.99),
    (2, 'Bread', 1.99),
    (3, 'Milk', 2.99);

It is also possible to insert the result of a query (which might be no rows, one row, or many rows):

INSERT INTO products (product_no, name, price)
  SELECT product_no, name, price FROM new_products
    WHERE release_date = 'today';

This provides the full power of the SQL query mechanism (Chapter 7) for computing the rows to be inserted.

Tip

When inserting a lot of data at the same time, consider using the COPY command. It is not as flexible as the INSERT command, but is more efficient. Refer to Section 14.4 for more information on improving bulk loading performance.

Prev | Up | Next
Chapter 6. Data Manipulation | Home | 6.2. Updating Data


6.2. Updating Data
Prev Up Chapter 6. Data Manipulation Home Next

6.2. Updating Data

The modification of data that is already in the database is referred to as updating. You can update individual rows, all the rows in a table, or a subset of all rows. Each column can be updated separately; the other columns are not affected.

To update existing rows, use the UPDATE command. This requires three pieces of information:

  1. The name of the table and column to update

  2. The new value of the column

  3. Which row(s) to update

Recall from Chapter 5 that SQL does not, in general, provide a unique identifier for rows. Therefore it is not always possible to directly specify which row to update. Instead, you specify which conditions a row must meet in order to be updated. Only if you have a primary key in the table (independent of whether you declared it or not) can you reliably address individual rows by choosing a condition that matches the primary key. Graphical database access tools rely on this fact to allow you to update rows individually.

For example, this command updates all products that have a price of 5 to have a price of 10:

UPDATE products SET price = 10 WHERE price = 5;

This might cause zero, one, or many rows to be updated. It is not an error to attempt an update that does not match any rows.

Let's look at that command in detail. First is the key word UPDATE followed by the table name. As usual, the table name can be schema-qualified, otherwise it is looked up in the path. Next is the key word SET followed by the column name, an equal sign, and the new column value. The new column value can be any scalar expression, not just a constant. For example, if you want to raise the price of all products by 10% you could use:

UPDATE products SET price = price * 1.10;

As you see, the expression for the new value can refer to the existing value(s) in the row. We also left out the WHERE clause. If it is omitted, it means that all rows in the table are updated. If it is present, only those rows that match the WHERE condition are updated. Note that the equals sign in the SET clause is an assignment while the one in the WHERE clause is a comparison, but this does not create any ambiguity. Of course, the WHERE condition does not have to be an equality test. Many other operators are available (see Chapter 9). But the expression needs to evaluate to a Boolean result.

You can update more than one column in an UPDATE command by listing more than one assignment in the SET clause. For example:

UPDATE mytable SET a = 5, b = 3, c = 1 WHERE a > 0;

Prev | Up | Next
6.1. Inserting Data | Home | 6.3. Deleting Data


6.3. Deleting Data
Prev Up Chapter 6. Data Manipulation Home Next

6.3. Deleting Data

So far we have explained how to add data to tables and how to change data. What remains is to discuss how to remove data that is no longer needed. Just as adding data is only possible in whole rows, you can only remove entire rows from a table. In the previous section we explained that SQL does not provide a way to directly address individual rows. Therefore, removing rows can only be done by specifying conditions that the rows to be removed have to match. If you have a primary key in the table then you can specify the exact row. But you can also remove groups of rows matching a condition, or you can remove all rows in the table at once.

You use the DELETE command to remove rows; the syntax is very similar to the UPDATE command. For instance, to remove all rows from the products table that have a price of 10, use:

DELETE FROM products WHERE price = 10;

If you simply write:

DELETE FROM products;

then all rows in the table will be deleted! Caveat programmer.

Prev | Up | Next
6.2. Updating Data | Home | 6.4. Returning Data from Modified Rows


6.4. Returning Data from Modified Rows
Prev Up Chapter 6. Data Manipulation Home Next

6.4. Returning Data from Modified Rows

Sometimes it is useful to obtain data from modified rows while they are being manipulated. The INSERT, UPDATE, and DELETE commands all have an optional RETURNING clause that supports this. Use of RETURNING avoids performing an extra database query to collect the data, and is especially valuable when it would otherwise be difficult to identify the modified rows reliably.

The allowed contents of a RETURNING clause are the same as a SELECT command's output list (see Section 7.3). It can contain column names of the command's target table, or value expressions using those columns. A common shorthand is RETURNING *, which selects all columns of the target table in order.

In an INSERT, the data available to RETURNING is the row as it was inserted. This is not so useful in trivial inserts, since it would just repeat the data provided by the client. But it can be very handy when relying on computed default values. For example, when using a serial column to provide unique identifiers, RETURNING can return the ID assigned to a new row:

CREATE TABLE users (firstname text, lastname text, id serial primary key);

INSERT INTO users (firstname, lastname) VALUES ('Joe', 'Cool') RETURNING id;

The RETURNING clause is also very useful with INSERT ... SELECT.

In an UPDATE, the data available to RETURNING is the new content of the modified row. For example:

UPDATE products SET price = price * 1.10
  WHERE price <= 99.99
  RETURNING name, price AS new_price;

In a DELETE, the data available to RETURNING is the content of the deleted row. For example:

DELETE FROM products
  WHERE obsoletion_date = 'today'
  RETURNING *;

If there are triggers (Chapter 39) on the target table, the data available to RETURNING is the row as modified by the triggers. Thus, inspecting columns computed by triggers is another common use-case for RETURNING.

Prev | Up | Next
6.3. Deleting Data | Home | Chapter 7. Queries


Chapter 7. Queries
Prev Up Part II. The SQL Language Home Next

Chapter 7. Queries

Table of Contents

7.1. Overview
7.2. Table Expressions
7.2.1. The FROM Clause
7.2.2. The WHERE Clause
7.2.3. The GROUP BY and HAVING Clauses
7.2.4. GROUPING SETS, CUBE, and ROLLUP
7.2.5. Window Function Processing
7.3. Select Lists
7.3.1. Select-List Items
7.3.2. Column Labels
7.3.3. DISTINCT
7.4. Combining Queries (UNION, INTERSECT, EXCEPT)
7.5. Sorting Rows (ORDER BY)
7.6. LIMIT and OFFSET
7.7. VALUES Lists
7.8. WITH Queries (Common Table Expressions)
7.8.1. SELECT in WITH
7.8.2. Recursive Queries
7.8.3. Common Table Expression Materialization
7.8.4. Data-Modifying Statements in WITH

The previous chapters explained how to create tables, how to fill them with data, and how to manipulate that data. Now we finally discuss how to retrieve the data from the database.

Prev | Up | Next
6.4. Returning Data from Modified Rows | Home | 7.1. Overview


7.1. Overview
Prev Up Chapter 7. Queries Home Next

7.1. Overview

The process of retrieving or the command to retrieve data from a database is called a query. In SQL the SELECT command is used to specify queries. The general syntax of the SELECT command is

[WITH with_queries] SELECT select_list FROM table_expression [sort_specification]

The following sections describe the details of the select list, the table expression, and the sort specification. WITH queries are treated last since they are an advanced feature.

A simple kind of query has the form:

SELECT * FROM table1;

Assuming that there is a table called table1, this command would retrieve all rows and all user-defined columns from table1. (The method of retrieval depends on the client application. For example, the psql program will display an ASCII-art table on the screen, while client libraries will offer functions to extract individual values from the query result.) The select list specification * means all columns that the table expression happens to provide. A select list can also select a subset of the available columns or make calculations using the columns. For example, if table1 has columns named a, b, and c (and perhaps others) you can make the following query:

SELECT a, b + c FROM table1;

(assuming that b and c are of a numerical data type). See Section 7.3 for more details.

FROM table1 is a simple kind of table expression: it reads just one table. In general, table expressions can be complex constructs of base tables, joins, and subqueries. But you can also omit the table expression entirely and use the SELECT command as a calculator:

SELECT 3 * 4;

This is more useful if the expressions in the select list return varying results. For example, you could call a function this way:

SELECT random();

Prev | Up | Next
Chapter 7. Queries | Home | 7.2. Table Expressions


7.2. Table Expressions
Prev Up Chapter 7. Queries Home Next

7.2. Table Expressions

7.2.1. The FROM Clause
7.2.2. The WHERE Clause
7.2.3. The GROUP BY and HAVING Clauses
7.2.4. GROUPING SETS, CUBE, and ROLLUP
7.2.5. Window Function Processing

A table expression computes a table. The table expression contains a FROM clause that is optionally followed by WHERE, GROUP BY, and HAVING clauses. Trivial table expressions simply refer to a table on disk, a so-called base table, but more complex expressions can be used to modify or combine base tables in various ways.

The optional WHERE, GROUP BY, and HAVING clauses in the table expression specify a pipeline of successive transformations performed on the table derived in the FROM clause. All these transformations produce a virtual table that provides the rows that are passed to the select list to compute the output rows of the query.

7.2.1. The FROM Clause

The FROM clause derives a table from one or more other tables given in a comma-separated table reference list.

FROM table_reference [, table_reference [, ...]]

A table reference can be a table name (possibly schema-qualified), or a derived table such as a subquery, a JOIN construct, or complex combinations of these. If more than one table reference is listed in the FROM clause, the tables are cross-joined (that is, the Cartesian product of their rows is formed; see below). The result of the FROM list is an intermediate virtual table that can then be subject to transformations by the WHERE, GROUP BY, and HAVING clauses and is finally the result of the overall table expression.

When a table reference names a table that is the parent of a table inheritance hierarchy, the table reference produces rows of not only that table but all of its descendant tables, unless the key word ONLY precedes the table name. However, the reference produces only the columns that appear in the named table — any columns added in subtables are ignored.

Instead of writing ONLY before the table name, you can write * after the table name to explicitly specify that descendant tables are included. There is no real reason to use this syntax any more, because searching descendant tables is now always the default behavior. However, it is supported for compatibility with older releases.

7.2.1.1. Joined Tables

A joined table is a table derived from two other (real or derived) tables according to the rules of the particular join type. Inner, outer, and cross-joins are available. The general syntax of a joined table is

T1 join_type T2 [ join_condition ]

Joins of all types can be chained together, or nested: either or both T1 and T2 can be joined tables. Parentheses can be used around JOIN clauses to control the join order. In the absence of parentheses, JOIN clauses nest left-to-right.

Join Types

Cross join
T1 CROSS JOIN T2

For every possible combination of rows from T1 and T2 (i.e., a Cartesian product), the joined table will contain a row consisting of all columns in T1 followed by all columns in T2. If the tables have N and M rows respectively, the joined table will have N * M rows.

FROM T1 CROSS JOIN T2 is equivalent to FROM T1 INNER JOIN T2 ON TRUE (see below). It is also equivalent to FROM T1, T2.

Note

This latter equivalence does not hold exactly when more than two tables appear, because JOIN binds more tightly than comma. For example FROM T1 CROSS JOIN T2 INNER JOIN T3 ON condition is not the same as FROM T1, T2 INNER JOIN T3 ON condition because the condition can reference T1 in the first case but not the second.

Qualified joins
T1 { [INNER] | { LEFT | RIGHT | FULL } [OUTER] } JOIN T2 ON boolean_expression
T1 { [INNER] | { LEFT | RIGHT | FULL } [OUTER] } JOIN T2 USING ( join column list )
T1 NATURAL { [INNER] | { LEFT | RIGHT | FULL } [OUTER] } JOIN T2

The words INNER and OUTER are optional in all forms. INNER is the default; LEFT, RIGHT, and FULL imply an outer join.

The join condition is specified in the ON or USING clause, or implicitly by the word NATURAL. The join condition determines which rows from the two source tables are considered to “match”, as explained in detail below.

The possible types of qualified join are:

INNER JOIN

For each row R1 of T1, the joined table has a row for each row in T2 that satisfies the join condition with R1.

LEFT OUTER JOIN

First, an inner join is performed. Then, for each row in T1 that does not satisfy the join condition with any row in T2, a joined row is added with null values in columns of T2. Thus, the joined table always has at least one row for each row in T1.

RIGHT OUTER JOIN

First, an inner join is performed. Then, for each row in T2 that does not satisfy the join condition with any row in T1, a joined row is added with null values in columns of T1. This is the converse of a left join: the result table will always have a row for each row in T2.

FULL OUTER JOIN

First, an inner join is performed. Then, for each row in T1 that does not satisfy the join condition with any row in T2, a joined row is added with null values in columns of T2. Also, for each row of T2 that does not satisfy the join condition with any row in T1, a joined row with null values in the columns of T1 is added.

The ON clause is the most general kind of join condition: it takes a Boolean value expression of the same kind as is used in a WHERE clause. A pair of rows from T1 and T2 match if the ON expression evaluates to true.

The USING clause is a shorthand that allows you to take advantage of the specific situation where both sides of the join use the same name for the joining column(s). It takes a comma-separated list of the shared column names and forms a join condition that includes an equality comparison for each one. For example, joining T1 and T2 with USING (a, b) produces the join condition ON T1.a = T2.a AND T1.b = T2.b.

Furthermore, the output of JOIN USING suppresses redundant columns: there is no need to print both of the matched columns, since they must have equal values. While JOIN ON produces all columns from T1 followed by all columns from T2, JOIN USING produces one output column for each of the listed column pairs (in the listed order), followed by any remaining columns from T1, followed by any remaining columns from T2.

Finally, NATURAL is a shorthand form of USING: it forms a USING list consisting of all column names that appear in both input tables. As with USING, these columns appear only once in the output table. If there are no common column names, NATURAL JOIN behaves like JOIN ... ON TRUE, producing a cross-product join.

Note

USING is reasonably safe from column changes in the joined relations since only the listed columns are combined. NATURAL is considerably more risky since any schema changes to either relation that cause a new matching column name to be present will cause the join to combine that new column as well.

To put this together, assume we have tables t1:

 num | name
-----+------
   1 | a
   2 | b
   3 | c

and t2:

 num | value
-----+-------
   1 | xxx
   3 | yyy
   5 | zzz

then we get the following results for the various joins:

=> SELECT * FROM t1 CROSS JOIN t2;
 num | name | num | value
-----+------+-----+-------
   1 | a    |   1 | xxx
   1 | a    |   3 | yyy
   1 | a    |   5 | zzz
   2 | b    |   1 | xxx
   2 | b    |   3 | yyy
   2 | b    |   5 | zzz
   3 | c    |   1 | xxx
   3 | c    |   3 | yyy
   3 | c    |   5 | zzz
(9 rows)

=> SELECT * FROM t1 INNER JOIN t2 ON t1.num = t2.num;
 num | name | num | value
-----+------+-----+-------
   1 | a    |   1 | xxx
   3 | c    |   3 | yyy
(2 rows)

=> SELECT * FROM t1 INNER JOIN t2 USING (num);
 num | name | value
-----+------+-------
   1 | a    | xxx
   3 | c    | yyy
(2 rows)

=> SELECT * FROM t1 NATURAL INNER JOIN t2;
 num | name | value
-----+------+-------
   1 | a    | xxx
   3 | c    | yyy
(2 rows)

=> SELECT * FROM t1 LEFT JOIN t2 ON t1.num = t2.num;
 num | name | num | value
-----+------+-----+-------
   1 | a    |   1 | xxx
   2 | b    |     |
   3 | c    |   3 | yyy
(3 rows)

=> SELECT * FROM t1 LEFT JOIN t2 USING (num);
 num | name | value
-----+------+-------
   1 | a    | xxx
   2 | b    |
   3 | c    | yyy
(3 rows)

=> SELECT * FROM t1 RIGHT JOIN t2 ON t1.num = t2.num;
 num | name | num | value
-----+------+-----+-------
   1 | a    |   1 | xxx
   3 | c    |   3 | yyy
     |      |   5 | zzz
(3 rows)

=> SELECT * FROM t1 FULL JOIN t2 ON t1.num = t2.num;
 num | name | num | value
-----+------+-----+-------
   1 | a    |   1 | xxx
   2 | b    |     |
   3 | c    |   3 | yyy
     |      |   5 | zzz
(4 rows)

The join condition specified with ON can also contain conditions that do not relate directly to the join. This can prove useful for some queries but needs to be thought out carefully. For example:

=> SELECT * FROM t1 LEFT JOIN t2 ON t1.num = t2.num AND t2.value = 'xxx';
 num | name | num | value
-----+------+-----+-------
   1 | a    |   1 | xxx
   2 | b    |     |
   3 | c    |     |
(3 rows)

Notice that placing the restriction in the WHERE clause produces a different result:

=> SELECT * FROM t1 LEFT JOIN t2 ON t1.num = t2.num WHERE t2.value = 'xxx';
 num | name | num | value
-----+------+-----+-------
   1 | a    |   1 | xxx
(1 row)

This is because a restriction placed in the ON clause is processed before the join, while a restriction placed in the WHERE clause is processed after the join. That does not matter with inner joins, but it matters a lot with outer joins.

7.2.1.2. Table and Column Aliases

A temporary name can be given to tables and complex table references to be used for references to the derived table in the rest of the query. This is called a table alias.

To create a table alias, write

FROM table_reference AS alias

or

FROM table_reference alias

The AS key word is optional noise. alias can be any identifier.

A typical application of table aliases is to assign short identifiers to long table names to keep the join clauses readable. For example:

SELECT * FROM some_very_long_table_name s JOIN another_fairly_long_name a ON s.id = a.num;

The alias becomes the new name of the table reference so far as the current query is concerned — it is not allowed to refer to the table by the original name elsewhere in the query. Thus, this is not valid:

SELECT * FROM my_table AS m WHERE my_table.a > 5;    -- wrong

Table aliases are mainly for notational convenience, but it is necessary to use them when joining a table to itself, e.g.:

SELECT * FROM people AS mother JOIN people AS child ON mother.id = child.mother_id;

Additionally, an alias is required if the table reference is a subquery (see Section 7.2.1.3).

Parentheses are used to resolve ambiguities. In the following example, the first statement assigns the alias b to the second instance of my_table, but the second statement assigns the alias to the result of the join:

SELECT * FROM my_table AS a CROSS JOIN my_table AS b ...
SELECT * FROM (my_table AS a CROSS JOIN my_table) AS b ...

Another form of table aliasing gives temporary names to the columns of the table, as well as the table itself:

FROM table_reference [AS] alias ( column1 [, column2 [, ...]] )

If fewer column aliases are specified than the actual table has columns, the remaining columns are not renamed. This syntax is especially useful for self-joins or subqueries.

When an alias is applied to the output of a JOIN clause, the alias hides the original name(s) within the JOIN. For example:

SELECT a.* FROM my_table AS a JOIN your_table AS b ON ...

is valid SQL, but:

SELECT a.* FROM (my_table AS a JOIN your_table AS b ON ...) AS c

is not valid; the table alias a is not visible outside the alias c.

7.2.1.3. Subqueries

Subqueries specifying a derived table must be enclosed in parentheses and must be assigned a table alias name (as in Section 7.2.1.2). For example:

FROM (SELECT * FROM table1) AS alias_name

This example is equivalent to FROM table1 AS alias_name. More interesting cases, which cannot be reduced to a plain join, arise when the subquery involves grouping or aggregation.

A subquery can also be a VALUES list:

FROM (VALUES ('anne', 'smith'), ('bob', 'jones'), ('joe', 'blow'))
     AS names(first, last)

Again, a table alias is required. Assigning alias names to the columns of the VALUES list is optional, but is good practice. For more information see Section 7.7.

7.2.1.4. Table Functions

Table functions are functions that produce a set of rows, made up of either base data types (scalar types) or composite data types (table rows). They are used like a table, view, or subquery in the FROM clause of a query. Columns returned by table functions can be included in SELECT, JOIN, or WHERE clauses in the same manner as columns of a table, view, or subquery.

Table functions may also be combined using the ROWS FROM syntax, with the results returned in parallel columns; the number of result rows in this case is that of the largest function result, with smaller results padded with null values to match.

function_call [WITH ORDINALITY] [[AS] table_alias [(column_alias [, ... ])]]
ROWS FROM( function_call [, ... ] ) [WITH ORDINALITY] [[AS] table_alias [(column_alias [, ... ])]]

If the WITH ORDINALITY clause is specified, an additional column of type bigint will be added to the function result columns. This column numbers the rows of the function result set, starting from 1. (This is a generalization of the SQL-standard syntax for UNNEST ... WITH ORDINALITY.) By default, the ordinal column is called ordinality, but a different column name can be assigned to it using an AS clause.

The special table function UNNEST may be called with any number of array parameters, and it returns a corresponding number of columns, as if UNNEST (Section 9.19) had been called on each parameter separately and combined using the ROWS FROM construct.

UNNEST( array_expression [, ... ] ) [WITH ORDINALITY] [[AS] table_alias [(column_alias [, ... ])]]

If no table_alias is specified, the function name is used as the table name; in the case of a ROWS FROM() construct, the first function's name is used.

If column aliases are not supplied, then for a function returning a base data type, the column name is also the same as the function name. For a function returning a composite type, the result columns get the names of the individual attributes of the type.

Some examples:

CREATE TABLE foo (fooid int, foosubid int, fooname text);

CREATE FUNCTION getfoo(int) RETURNS SETOF foo AS $$
    SELECT * FROM foo WHERE fooid = $1;
$$ LANGUAGE SQL;

SELECT * FROM getfoo(1) AS t1;

SELECT * FROM foo
    WHERE foosubid IN (
                        SELECT foosubid
                        FROM getfoo(foo.fooid) z
                        WHERE z.fooid = foo.fooid
                      );

CREATE VIEW vw_getfoo AS SELECT * FROM getfoo(1);

SELECT * FROM vw_getfoo;

In some cases it is useful to define table functions that can return different column sets depending on how they are invoked. To support this, the table function can be declared as returning the pseudo-type record with no OUT parameters. When such a function is used in a query, the expected row structure must be specified in the query itself, so that the system can know how to parse and plan the query. This syntax looks like:

function_call [AS] alias (column_definition [, ... ])
function_call AS [alias] (column_definition [, ... ])
ROWS FROM( ... function_call AS (column_definition [, ... ]) [, ... ] )

When not using the ROWS FROM() syntax, the column_definition list replaces the column alias list that could otherwise be attached to the FROM item; the names in the column definitions serve as column aliases. When using the ROWS FROM() syntax, a column_definition list can be attached to each member function separately; or if there is only one member function and no WITH ORDINALITY clause, a column_definition list can be written in place of a column alias list following ROWS FROM().

Consider this example:

SELECT *
    FROM dblink('dbname=mydb', 'SELECT proname, prosrc FROM pg_proc')
      AS t1(proname name, prosrc text)
    WHERE proname LIKE 'bytea%';

The dblink function (part of the dblink module) executes a remote query. It is declared to return record since it might be used for any kind of query. The actual column set must be specified in the calling query so that the parser knows, for example, what * should expand to.

This example uses ROWS FROM:

SELECT *
FROM ROWS FROM
    (
        json_to_recordset('[{"a":40,"b":"foo"},{"a":"100","b":"bar"}]')
            AS (a INTEGER, b TEXT),
        generate_series(1, 3)
    ) AS x (p, q, s)
ORDER BY p;

  p  |  q  | s
-----+-----+---
  40 | foo | 1
 100 | bar | 2
     |     | 3

It joins two functions into a single FROM target. json_to_recordset() is instructed to return two columns, the first integer and the second text. The result of generate_series() is used directly. The ORDER BY clause sorts the column values as integers.

7.2.1.5. LATERAL Subqueries

Subqueries appearing in FROM can be preceded by the key word LATERAL. This allows them to reference columns provided by preceding FROM items. (Without LATERAL, each subquery is evaluated independently and so cannot cross-reference any other FROM item.)

Table functions appearing in FROM can also be preceded by the key word LATERAL, but for functions the key word is optional; the function's arguments can contain references to columns provided by preceding FROM items in any case.

A LATERAL item can appear at top level in the FROM list, or within a JOIN tree. In the latter case it can also refer to any items that are on the left-hand side of a JOIN that it is on the right-hand side of.

When a FROM item contains LATERAL cross-references, evaluation proceeds as follows: for each row of the FROM item providing the cross-referenced column(s), or set of rows of multiple FROM items providing the columns, the LATERAL item is evaluated using that row or row set's values of the columns. The resulting row(s) are joined as usual with the rows they were computed from. This is repeated for each row or set of rows from the column source table(s).

A trivial example of LATERAL is

SELECT * FROM foo, LATERAL (SELECT * FROM bar WHERE bar.id = foo.bar_id) ss;

This is not especially useful since it has exactly the same result as the more conventional

SELECT * FROM foo, bar WHERE bar.id = foo.bar_id;

LATERAL is primarily useful when the cross-referenced column is necessary for computing the row(s) to be joined. A common application is providing an argument value for a set-returning function. For example, supposing that vertices(polygon) returns the set of vertices of a polygon, we could identify close-together vertices of polygons stored in a table with:

SELECT p1.id, p2.id, v1, v2
FROM polygons p1, polygons p2,
     LATERAL vertices(p1.poly) v1,
     LATERAL vertices(p2.poly) v2
WHERE (v1 <-> v2) < 10 AND p1.id != p2.id;

This query could also be written

SELECT p1.id, p2.id, v1, v2
FROM polygons p1 CROSS JOIN LATERAL vertices(p1.poly) v1,
     polygons p2 CROSS JOIN LATERAL vertices(p2.poly) v2
WHERE (v1 <-> v2) < 10 AND p1.id != p2.id;

or in several other equivalent formulations. (As already mentioned, the LATERAL key word is unnecessary in this example, but we use it for clarity.)

It is often particularly handy to LEFT JOIN to a LATERAL subquery, so that source rows will appear in the result even if the LATERAL subquery produces no rows for them. For example, if get_product_names() returns the names of products made by a manufacturer, but some manufacturers in our table currently produce no products, we could find out which ones those are like this:

SELECT m.name
FROM manufacturers m LEFT JOIN LATERAL get_product_names(m.id) pname ON true
WHERE pname IS NULL;

7.2.2. The WHERE Clause

The syntax of the WHERE clause is

WHERE search_condition

where search_condition is any value expression (see Section 4.2) that returns a value of type boolean.

After the processing of the FROM clause is done, each row of the derived virtual table is checked against the search condition. If the result of the condition is true, the row is kept in the output table, otherwise (i.e., if the result is false or null) it is discarded. The search condition typically references at least one column of the table generated in the FROM clause; this is not required, but otherwise the WHERE clause will be fairly useless.

Note

The join condition of an inner join can be written either in the WHERE clause or in the JOIN clause. For example, these table expressions are equivalent:

FROM a, b WHERE a.id = b.id AND b.val > 5

and:

FROM a INNER JOIN b ON (a.id = b.id) WHERE b.val > 5

or perhaps even:

FROM a NATURAL JOIN b WHERE b.val > 5

Which one of these you use is mainly a matter of style. The JOIN syntax in the FROM clause is probably not as portable to other SQL database management systems, even though it is in the SQL standard. For outer joins there is no choice: they must be done in the FROM clause. The ON or USING clause of an outer join is not equivalent to a WHERE condition, because it results in the addition of rows (for unmatched input rows) as well as the removal of rows in the final result.

Here are some examples of WHERE clauses:

SELECT ... FROM fdt WHERE c1 > 5

SELECT ... FROM fdt WHERE c1 IN (1, 2, 3)

SELECT ... FROM fdt WHERE c1 IN (SELECT c1 FROM t2)

SELECT ... FROM fdt WHERE c1 IN (SELECT c3 FROM t2 WHERE c2 = fdt.c1 + 10)

SELECT ... FROM fdt WHERE c1 BETWEEN (SELECT c3 FROM t2 WHERE c2 = fdt.c1 + 10) AND 100

SELECT ... FROM fdt WHERE EXISTS (SELECT c1 FROM t2 WHERE c2 > fdt.c1)

fdt is the table derived in the FROM clause. Rows that do not meet the search condition of the WHERE clause are eliminated from fdt. Notice the use of scalar subqueries as value expressions. Just like any other query, the subqueries can employ complex table expressions. Notice also how fdt is referenced in the subqueries. Qualifying c1 as fdt.c1 is only necessary if c1 is also the name of a column in the derived input table of the subquery. But qualifying the column name adds clarity even when it is not needed. This example shows how the column naming scope of an outer query extends into its inner queries.

7.2.3. The GROUP BY and HAVING Clauses

After passing the WHERE filter, the derived input table might be subject to grouping, using the GROUP BY clause, and elimination of group rows using the HAVING clause.

SELECT select_list
    FROM ...
    [WHERE ...]
    GROUP BY grouping_column_reference [, grouping_column_reference]...

The GROUP BY clause is used to group together those rows in a table that have the same values in all the columns listed. The order in which the columns are listed does not matter. The effect is to combine each set of rows having common values into one group row that represents all rows in the group. This is done to eliminate redundancy in the output and/or compute aggregates that apply to these groups. For instance:

=> SELECT * FROM test1;
 x | y
---+---
 a | 3
 c | 2
 b | 5
 a | 1
(4 rows)

=> SELECT x FROM test1 GROUP BY x;
 x
---
 a
 b
 c
(3 rows)

In the second query, we could not have written SELECT * FROM test1 GROUP BY x, because there is no single value for the column y that could be associated with each group. The grouped-by columns can be referenced in the select list since they have a single value in each group.

In general, if a table is grouped, columns that are not listed in GROUP BY cannot be referenced except in aggregate expressions. An example with aggregate expressions is:

=> SELECT x, sum(y) FROM test1 GROUP BY x;
 x | sum
---+-----
 a |   4
 b |   5
 c |   2
(3 rows)

Here sum is an aggregate function that computes a single value over the entire group. More information about the available aggregate functions can be found in Section 9.21.

Tip

Grouping without aggregate expressions effectively calculates the set of distinct values in a column. This can also be achieved using the DISTINCT clause (see Section 7.3.3).

Here is another example: it calculates the total sales for each product (rather than the total sales of all products):

SELECT product_id, p.name, (sum(s.units) * p.price) AS sales
    FROM products p LEFT JOIN sales s USING (product_id)
    GROUP BY product_id, p.name, p.price;

In this example, the columns product_id, p.name, and p.price must be in the GROUP BY clause since they are referenced in the query select list (but see below). The column s.units does not have to be in the GROUP BY list since it is only used in an aggregate expression (sum(...)), which represents the sales of a product. For each product, the query returns a summary row about all sales of the product.

If the products table is set up so that, say, product_id is the primary key, then it would be enough to group by product_id in the above example, since name and price would be functionally dependent on the product ID, and so there would be no ambiguity about which name and price value to return for each product ID group.

In strict SQL, GROUP BY can only group by columns of the source table but PostgreSQL extends this to also allow GROUP BY to group by columns in the select list. Grouping by value expressions instead of simple column names is also allowed.

If a table has been grouped using GROUP BY, but only certain groups are of interest, the HAVING clause can be used, much like a WHERE clause, to eliminate groups from the result. The syntax is:

SELECT select_list FROM ... [WHERE ...] GROUP BY ... HAVING boolean_expression

Expressions in the HAVING clause can refer both to grouped expressions and to ungrouped expressions (which necessarily involve an aggregate function).

Example:

=> SELECT x, sum(y) FROM test1 GROUP BY x HAVING sum(y) > 3;
 x | sum
---+-----
 a |   4
 b |   5
(2 rows)

=> SELECT x, sum(y) FROM test1 GROUP BY x HAVING x < 'c';
 x | sum
---+-----
 a |   4
 b |   5
(2 rows)

Again, a more realistic example:

SELECT product_id, p.name, (sum(s.units) * (p.price - p.cost)) AS profit
    FROM products p LEFT JOIN sales s USING (product_id)
    WHERE s.date > CURRENT_DATE - INTERVAL '4 weeks'
    GROUP BY product_id, p.name, p.price, p.cost
    HAVING sum(p.price * s.units) > 5000;

In the example above, the WHERE clause is selecting rows by a column that is not grouped (the expression is only true for sales during the last four weeks), while the HAVING clause restricts the output to groups with total gross sales over 5000. Note that the aggregate expressions do not necessarily need to be the same in all parts of the query.

If a query contains aggregate function calls, but no GROUP BY clause, grouping still occurs: the result is a single group row (or perhaps no rows at all, if the single row is then eliminated by HAVING). The same is true if it contains a HAVING clause, even without any aggregate function calls or GROUP BY clause.

7.2.4. GROUPING SETS, CUBE, and ROLLUP

More complex grouping operations than those described above are possible using the concept of grouping sets. The data selected by the FROM and WHERE clauses is grouped separately by each specified grouping set, aggregates computed for each group just as for simple GROUP BY clauses, and then the results returned. For example:

=> SELECT * FROM items_sold;
 brand | size | sales
-------+------+-------
 Foo   | L    |  10
 Foo   | M    |  20
 Bar   | M    |  15
 Bar   | L    |  5
(4 rows)

=> SELECT brand, size, sum(sales) FROM items_sold GROUP BY GROUPING SETS ((brand), (size), ());
 brand | size | sum
-------+------+-----
 Foo   |      |  30
 Bar   |      |  20
       | L    |  15
       | M    |  35
       |      |  50
(5 rows)

Each sublist of GROUPING SETS may specify zero or more columns or expressions and is interpreted the same way as though it were directly in the GROUP BY clause. An empty grouping set means that all rows are aggregated down to a single group (which is output even if no input rows were present), as described above for the case of aggregate functions with no GROUP BY clause.

References to the grouping columns or expressions are replaced by null values in result rows for grouping sets in which those columns do not appear. To distinguish which grouping a particular output row resulted from, see Table 9.61.

A shorthand notation is provided for specifying two common types of grouping set. A clause of the form

ROLLUP ( e1, e2, e3, ... )

represents the given list of expressions and all prefixes of the list including the empty list; thus it is equivalent to

GROUPING SETS (
    ( e1, e2, e3, ... ),
    ...
    ( e1, e2 ),
    ( e1 ),
    ( )
)

This is commonly used for analysis over hierarchical data; e.g., total salary by department, division, and company-wide total.

A clause of the form

CUBE ( e1, e2, ... )

represents the given list and all of its possible subsets (i.e., the power set). Thus

CUBE ( a, b, c )

is equivalent to

GROUPING SETS (
    ( a, b, c ),
    ( a, b    ),
    ( a,    c ),
    ( a       ),
    (    b, c ),
    (    b    ),
    (       c ),
    (         )
)

The individual elements of a CUBE or ROLLUP clause may be either individual expressions, or sublists of elements in parentheses. In the latter case, the sublists are treated as single units for the purposes of generating the individual grouping sets. For example:

CUBE ( (a, b), (c, d) )

is equivalent to

GROUPING SETS (
    ( a, b, c, d ),
    ( a, b       ),
    (       c, d ),
    (            )
)

and

ROLLUP ( a, (b, c), d )

is equivalent to

GROUPING SETS (
    ( a, b, c, d ),
    ( a, b, c    ),
    ( a          ),
    (            )
)

The CUBE and ROLLUP constructs can be used either directly in the GROUP BY clause, or nested inside a GROUPING SETS clause. If one GROUPING SETS clause is nested inside another, the effect is the same as if all the elements of the inner clause had been written directly in the outer clause.

If multiple grouping items are specified in a single GROUP BY clause, then the final list of grouping sets is the cross product of the individual items. For example:

GROUP BY a, CUBE (b, c), GROUPING SETS ((d), (e))

is equivalent to

GROUP BY GROUPING SETS (
    (a, b, c, d), (a, b, c, e),
    (a, b, d),    (a, b, e),
    (a, c, d),    (a, c, e),
    (a, d),       (a, e)
)

When specifying multiple grouping items together, the final set of grouping sets might contain duplicates. For example:

GROUP BY ROLLUP (a, b), ROLLUP (a, c)

is equivalent to

GROUP BY GROUPING SETS (
    (a, b, c),
    (a, b),
    (a, b),
    (a, c),
    (a),
    (a),
    (a, c),
    (a),
    ()
)

If these duplicates are undesirable, they can be removed using the DISTINCT clause directly on the GROUP BY. Therefore:

GROUP BY DISTINCT ROLLUP (a, b), ROLLUP (a, c)

is equivalent to

GROUP BY GROUPING SETS (
    (a, b, c),
    (a, b),
    (a, c),
    (a),
    ()
)

This is not the same as using SELECT DISTINCT because the output rows may still contain duplicates. If any of the ungrouped columns contains NULL, it will be indistinguishable from the NULL used when that same column is grouped.

Note

The construct (a, b) is normally recognized in expressions as a row constructor. Within the GROUP BY clause, this does not apply at the top levels of expressions, and (a, b) is parsed as a list of expressions as described above. If for some reason you need a row constructor in a grouping expression, use ROW(a, b).

7.2.5. Window Function Processing

If the query contains any window functions (see Section 3.5, Section 9.22 and Section 4.2.8), these functions are evaluated after any grouping, aggregation, and HAVING filtering is performed. That is, if the query uses any aggregates, GROUP BY, or HAVING, then the rows seen by the window functions are the group rows instead of the original table rows from FROM/WHERE.

When multiple window functions are used, all the window functions having syntactically equivalent PARTITION BY and ORDER BY clauses in their window definitions are guaranteed to be evaluated in a single pass over the data. Therefore they will see the same sort ordering, even if the ORDER BY does not uniquely determine an ordering. However, no guarantees are made about the evaluation of functions having different PARTITION BY or ORDER BY specifications. (In such cases a sort step is typically required between the passes of window function evaluations, and the sort is not guaranteed to preserve ordering of rows that its ORDER BY sees as equivalent.)

Currently, window functions always require presorted data, and so the query output will be ordered according to one or another of the window functions' PARTITION BY/ORDER BY clauses. It is not recommended to rely on this, however. Use an explicit top-level ORDER BY clause if you want to be sure the results are sorted in a particular way.

Prev | Up | Next
7.1. Overview | Home | 7.3. Select Lists


7.3. Select Lists
Prev Up Chapter 7. Queries Home Next

7.3. Select Lists

7.3.1. Select-List Items
7.3.2. Column Labels
7.3.3. DISTINCT

As shown in the previous section, the table expression in the SELECT command constructs an intermediate virtual table by possibly combining tables, views, eliminating rows, grouping, etc. This table is finally passed on to processing by the select list. The select list determines which columns of the intermediate table are actually output.

7.3.1. Select-List Items

The simplest kind of select list is * which emits all columns that the table expression produces. Otherwise, a select list is a comma-separated list of value expressions (as defined in Section 4.2). For instance, it could be a list of column names:

SELECT a, b, c FROM ...

The columns names a, b, and c are either the actual names of the columns of tables referenced in the FROM clause, or the aliases given to them as explained in Section 7.2.1.2. The name space available in the select list is the same as in the WHERE clause, unless grouping is used, in which case it is the same as in the HAVING clause.

If more than one table has a column of the same name, the table name must also be given, as in:

SELECT tbl1.a, tbl2.a, tbl1.b FROM ...

When working with multiple tables, it can also be useful to ask for all the columns of a particular table:

SELECT tbl1.*, tbl2.a FROM ...

See Section 8.16.5 for more about the table_name.* notation.

If an arbitrary value expression is used in the select list, it conceptually adds a new virtual column to the returned table. The value expression is evaluated once for each result row, with the row's values substituted for any column references. But the expressions in the select list do not have to reference any columns in the table expression of the FROM clause; they can be constant arithmetic expressions, for instance.

7.3.2. Column Labels

The entries in the select list can be assigned names for subsequent processing, such as for use in an ORDER BY clause or for display by the client application. For example:

SELECT a AS value, b + c AS sum FROM ...

If no output column name is specified using AS, the system assigns a default column name. For simple column references, this is the name of the referenced column. For function calls, this is the name of the function. For complex expressions, the system will generate a generic name.

The AS key word is usually optional, but in some cases where the desired column name matches a PostgreSQL key word, you must write AS or double-quote the column name in order to avoid ambiguity. (Appendix C shows which key words require AS to be used as a column label.) For example, FROM is one such key word, so this does not work:

SELECT a from, b + c AS sum FROM ...

but either of these do:

SELECT a AS from, b + c AS sum FROM ...
SELECT a "from", b + c AS sum FROM ...

For greatest safety against possible future key word additions, it is recommended that you always either write AS or double-quote the output column name.

Note

The naming of output columns here is different from that done in the FROM clause (see Section 7.2.1.2). It is possible to rename the same column twice, but the name assigned in the select list is the one that will be passed on.

7.3.3. DISTINCT

After the select list has been processed, the result table can optionally be subject to the elimination of duplicate rows. The DISTINCT key word is written directly after SELECT to specify this:

SELECT DISTINCT select_list ...

(Instead of DISTINCT the key word ALL can be used to specify the default behavior of retaining all rows.)

Obviously, two rows are considered distinct if they differ in at least one column value. Null values are considered equal in this comparison.

Alternatively, an arbitrary expression can determine what rows are to be considered distinct:

SELECT DISTINCT ON (expression [, expression ...]) select_list ...

Here expression is an arbitrary value expression that is evaluated for all rows. A set of rows for which all the expressions are equal are considered duplicates, and only the first row of the set is kept in the output. Note that the “first row” of a set is unpredictable unless the query is sorted on enough columns to guarantee a unique ordering of the rows arriving at the DISTINCT filter. (DISTINCT ON processing occurs after ORDER BY sorting.)

The DISTINCT ON clause is not part of the SQL standard and is sometimes considered bad style because of the potentially indeterminate nature of its results. With judicious use of GROUP BY and subqueries in FROM, this construct can be avoided, but it is often the most convenient alternative.

Prev | Up | Next
7.2. Table Expressions | Home | 7.4. Combining Queries (UNION, INTERSECT, EXCEPT)


7.4. Combining Queries (UNION, INTERSECT, EXCEPT)
Prev Up Chapter 7. Queries Home Next

7.4. Combining Queries (UNION, INTERSECT, EXCEPT)

The results of two queries can be combined using the set operations union, intersection, and difference. The syntax is

query1 UNION [ALL] query2
query1 INTERSECT [ALL] query2
query1 EXCEPT [ALL] query2

where query1 and query2 are queries that can use any of the features discussed up to this point.

UNION effectively appends the result of query2 to the result of query1 (although there is no guarantee that this is the order in which the rows are actually returned). Furthermore, it eliminates duplicate rows from its result, in the same way as DISTINCT, unless UNION ALL is used.

INTERSECT returns all rows that are both in the result of query1 and in the result of query2. Duplicate rows are eliminated unless INTERSECT ALL is used.

EXCEPT returns all rows that are in the result of query1 but not in the result of query2. (This is sometimes called the difference between two queries.) Again, duplicates are eliminated unless EXCEPT ALL is used.

In order to calculate the union, intersection, or difference of two queries, the two queries must be “union compatible”, which means that they return the same number of columns and the corresponding columns have compatible data types, as described in Section 10.5.

Set operations can be combined, for example

query1 UNION query2 EXCEPT query3

which is equivalent to

(query1 UNION query2) EXCEPT query3

As shown here, you can use parentheses to control the order of evaluation. Without parentheses, UNION and EXCEPT associate left-to-right, but INTERSECT binds more tightly than those two operators. Thus

query1 UNION query2 INTERSECT query3

means

query1 UNION (query2 INTERSECT query3)

You can also surround an individual query with parentheses. This is important if the query needs to use any of the clauses discussed in following sections, such as LIMIT. Without parentheses, you'll get a syntax error, or else the clause will be understood as applying to the output of the set operation rather than one of its inputs. For example,

SELECT a FROM b UNION SELECT x FROM y LIMIT 10

is accepted, but it means

(SELECT a FROM b UNION SELECT x FROM y) LIMIT 10

not

SELECT a FROM b UNION (SELECT x FROM y LIMIT 10)

Prev | Up | Next
7.3. Select Lists | Home | 7.5. Sorting Rows (ORDER BY)


7.5. Sorting Rows (ORDER BY)
Prev Up Chapter 7. Queries Home Next

7.5. Sorting Rows (ORDER BY)

After a query has produced an output table (after the select list has been processed) it can optionally be sorted. If sorting is not chosen, the rows will be returned in an unspecified order. The actual order in that case will depend on the scan and join plan types and the order on disk, but it must not be relied on. A particular output ordering can only be guaranteed if the sort step is explicitly chosen.

The ORDER BY clause specifies the sort order:

SELECT select_list
    FROM table_expression
    ORDER BY sort_expression1 [ASC | DESC] [NULLS { FIRST | LAST }]
             [, sort_expression2 [ASC | DESC] [NULLS { FIRST | LAST }] ...]

The sort expression(s) can be any expression that would be valid in the query's select list. An example is:

SELECT a, b FROM table1 ORDER BY a + b, c;

When more than one expression is specified, the later values are used to sort rows that are equal according to the earlier values. Each expression can be followed by an optional ASC or DESC keyword to set the sort direction to ascending or descending. ASC order is the default. Ascending order puts smaller values first, where “smaller” is defined in terms of the < operator. Similarly, descending order is determined with the > operator. [6]

The NULLS FIRST and NULLS LAST options can be used to determine whether nulls appear before or after non-null values in the sort ordering. By default, null values sort as if larger than any non-null value; that is, NULLS FIRST is the default for DESC order, and NULLS LAST otherwise.

Note that the ordering options are considered independently for each sort column. For example ORDER BY x, y DESC means ORDER BY x ASC, y DESC, which is not the same as ORDER BY x DESC, y DESC.

A sort_expression can also be the column label or number of an output column, as in:

SELECT a + b AS sum, c FROM table1 ORDER BY sum;
SELECT a, max(b) FROM table1 GROUP BY a ORDER BY 1;

both of which sort by the first output column. Note that an output column name has to stand alone, that is, it cannot be used in an expression — for example, this is not correct:

SELECT a + b AS sum, c FROM table1 ORDER BY sum + c;          -- wrong

This restriction is made to reduce ambiguity. There is still ambiguity if an ORDER BY item is a simple name that could match either an output column name or a column from the table expression. The output column is used in such cases. This would only cause confusion if you use AS to rename an output column to match some other table column's name.

ORDER BY can be applied to the result of a UNION, INTERSECT, or EXCEPT combination, but in this case it is only permitted to sort by output column names or numbers, not by expressions.

[6] Actually, PostgreSQL uses the default B-tree operator class for the expression's data type to determine the sort ordering for ASC and DESC. Conventionally, data types will be set up so that the < and > operators correspond to this sort ordering, but a user-defined data type's designer could choose to do something different.

Prev | Up | Next
7.4. Combining Queries (UNION, INTERSECT, EXCEPT) | Home | 7.6. LIMIT and OFFSET


7.6. LIMIT and OFFSET
Prev Up Chapter 7. Queries Home Next

7.6. LIMIT and OFFSET

LIMIT and OFFSET allow you to retrieve just a portion of the rows that are generated by the rest of the query:

SELECT select_list
    FROM table_expression
    [ ORDER BY ... ]
    [ LIMIT { number | ALL } ] [ OFFSET number ]

If a limit count is given, no more than that many rows will be returned (but possibly fewer, if the query itself yields fewer rows). LIMIT ALL is the same as omitting the LIMIT clause, as is LIMIT with a NULL argument.

OFFSET says to skip that many rows before beginning to return rows. OFFSET 0 is the same as omitting the OFFSET clause, as is OFFSET with a NULL argument.

If both OFFSET and LIMIT appear, then OFFSET rows are skipped before starting to count the LIMIT rows that are returned.

When using LIMIT, it is important to use an ORDER BY clause that constrains the result rows into a unique order. Otherwise you will get an unpredictable subset of the query's rows. You might be asking for the tenth through twentieth rows, but tenth through twentieth in what ordering? The ordering is unknown, unless you specified ORDER BY.

The query optimizer takes LIMIT into account when generating query plans, so you are very likely to get different plans (yielding different row orders) depending on what you give for LIMIT and OFFSET. Thus, using different LIMIT/OFFSET values to select different subsets of a query result will give inconsistent results unless you enforce a predictable result ordering with ORDER BY. This is not a bug; it is an inherent consequence of the fact that SQL does not promise to deliver the results of a query in any particular order unless ORDER BY is used to constrain the order.

The rows skipped by an OFFSET clause still have to be computed inside the server; therefore a large OFFSET might be inefficient.

Prev | Up | Next
7.5. Sorting Rows (ORDER BY) | Home | 7.7. VALUES Lists


7.7. VALUES Lists
Prev Up Chapter 7. Queries Home Next

7.7. VALUES Lists

VALUES provides a way to generate a “constant table” that can be used in a query without having to actually create and populate a table on-disk. The syntax is

VALUES ( expression [, ...] ) [, ...]

Each parenthesized list of expressions generates a row in the table. The lists must all have the same number of elements (i.e., the number of columns in the table), and corresponding entries in each list must have compatible data types. The actual data type assigned to each column of the result is determined using the same rules as for UNION (see Section 10.5).

As an example:

VALUES (1, 'one'), (2, 'two'), (3, 'three');

will return a table of two columns and three rows. It's effectively equivalent to:

SELECT 1 AS column1, 'one' AS column2
UNION ALL
SELECT 2, 'two'
UNION ALL
SELECT 3, 'three';

By default, PostgreSQL assigns the names column1, column2, etc. to the columns of a VALUES table. The column names are not specified by the SQL standard and different database systems do it differently, so it's usually better to override the default names with a table alias list, like this:

=> SELECT * FROM (VALUES (1, 'one'), (2, 'two'), (3, 'three')) AS t (num,letter);
 num | letter
-----+--------
   1 | one
   2 | two
   3 | three
(3 rows)

Syntactically, VALUES followed by expression lists is treated as equivalent to:

SELECT select_list FROM table_expression

and can appear anywhere a SELECT can. For example, you can use it as part of a UNION, or attach a sort_specification (ORDER BY, LIMIT, and/or OFFSET) to it. VALUES is most commonly used as the data source in an INSERT command, and next most commonly as a subquery.

For more information see VALUES.

Prev | Up | Next
7.6. LIMIT and OFFSET | Home | 7.8. WITH Queries (Common Table Expressions)


7.8. WITH Queries (Common Table Expressions)
Prev Up Chapter 7. Queries Home Next

7.8. WITH Queries (Common Table Expressions)

7.8.1. SELECT in WITH
7.8.2. Recursive Queries
7.8.3. Common Table Expression Materialization
7.8.4. Data-Modifying Statements in WITH

WITH provides a way to write auxiliary statements for use in a larger query. These statements, which are often referred to as Common Table Expressions or CTEs, can be thought of as defining temporary tables that exist just for one query. Each auxiliary statement in a WITH clause can be a SELECT, INSERT, UPDATE, or DELETE; and the WITH clause itself is attached to a primary statement that can also be a SELECT, INSERT, UPDATE, or DELETE.

7.8.1. SELECT in WITH

The basic value of SELECT in WITH is to break down complicated queries into simpler parts. An example is:

WITH regional_sales AS (
    SELECT region, SUM(amount) AS total_sales
    FROM orders
    GROUP BY region
), top_regions AS (
    SELECT region
    FROM regional_sales
    WHERE total_sales > (SELECT SUM(total_sales)/10 FROM regional_sales)
)
SELECT region,
       product,
       SUM(quantity) AS product_units,
       SUM(amount) AS product_sales
FROM orders
WHERE region IN (SELECT region FROM top_regions)
GROUP BY region, product;

which displays per-product sales totals in only the top sales regions. The WITH clause defines two auxiliary statements named regional_sales and top_regions, where the output of regional_sales is used in top_regions and the output of top_regions is used in the primary SELECT query. This example could have been written without WITH, but we'd have needed two levels of nested sub-SELECTs. It's a bit easier to follow this way.

7.8.2. Recursive Queries

The optional RECURSIVE modifier changes WITH from a mere syntactic convenience into a feature that accomplishes things not otherwise possible in standard SQL. Using RECURSIVE, a WITH query can refer to its own output. A very simple example is this query to sum the integers from 1 through 100:

WITH RECURSIVE t(n) AS (
    VALUES (1)
  UNION ALL
    SELECT n+1 FROM t WHERE n < 100
)
SELECT sum(n) FROM t;

The general form of a recursive WITH query is always a non-recursive term, then UNION (or UNION ALL), then a recursive term, where only the recursive term can contain a reference to the query's own output. Such a query is executed as follows:

Recursive Query Evaluation

  1. Evaluate the non-recursive term. For UNION (but not UNION ALL), discard duplicate rows. Include all remaining rows in the result of the recursive query, and also place them in a temporary working table.

  2. So long as the working table is not empty, repeat these steps:

    1. Evaluate the recursive term, substituting the current contents of the working table for the recursive self-reference. For UNION (but not UNION ALL), discard duplicate rows and rows that duplicate any previous result row. Include all remaining rows in the result of the recursive query, and also place them in a temporary intermediate table.

    2. Replace the contents of the working table with the contents of the intermediate table, then empty the intermediate table.

Note

Strictly speaking, this process is iteration not recursion, but RECURSIVE is the terminology chosen by the SQL standards committee.

In the example above, the working table has just a single row in each step, and it takes on the values from 1 through 100 in successive steps. In the 100th step, there is no output because of the WHERE clause, and so the query terminates.

Recursive queries are typically used to deal with hierarchical or tree-structured data. A useful example is this query to find all the direct and indirect sub-parts of a product, given only a table that shows immediate inclusions:

WITH RECURSIVE included_parts(sub_part, part, quantity) AS (
    SELECT sub_part, part, quantity FROM parts WHERE part = 'our_product'
  UNION ALL
    SELECT p.sub_part, p.part, p.quantity
    FROM included_parts pr, parts p
    WHERE p.part = pr.sub_part
)
SELECT sub_part, SUM(quantity) as total_quantity
FROM included_parts
GROUP BY sub_part

7.8.2.1. Search Order

When computing a tree traversal using a recursive query, you might want to order the results in either depth-first or breadth-first order. This can be done by computing an ordering column alongside the other data columns and using that to sort the results at the end. Note that this does not actually control in which order the query evaluation visits the rows; that is as always in SQL implementation-dependent. This approach merely provides a convenient way to order the results afterwards.

To create a depth-first order, we compute for each result row an array of rows that we have visited so far. For example, consider the following query that searches a table tree using a link field:

WITH RECURSIVE search_tree(id, link, data) AS (
    SELECT t.id, t.link, t.data
    FROM tree t
  UNION ALL
    SELECT t.id, t.link, t.data
    FROM tree t, search_tree st
    WHERE t.id = st.link
)
SELECT * FROM search_tree;

To add depth-first ordering information, you can write this:

WITH RECURSIVE search_tree(id, link, data, path) AS (
    SELECT t.id, t.link, t.data, ARRAY[t.id]
    FROM tree t
  UNION ALL
    SELECT t.id, t.link, t.data, path || t.id
    FROM tree t, search_tree st
    WHERE t.id = st.link
)
SELECT * FROM search_tree ORDER BY path;

In the general case where more than one field needs to be used to identify a row, use an array of rows. For example, if we needed to track fields f1 and f2:

WITH RECURSIVE search_tree(id, link, data, path) AS (
    SELECT t.id, t.link, t.data, ARRAY[ROW(t.f1, t.f2)]
    FROM tree t
  UNION ALL
    SELECT t.id, t.link, t.data, path || ROW(t.f1, t.f2)
    FROM tree t, search_tree st
    WHERE t.id = st.link
)
SELECT * FROM search_tree ORDER BY path;

Tip

Omit the ROW() syntax in the common case where only one field needs to be tracked. This allows a simple array rather than a composite-type array to be used, gaining efficiency.

To create a breadth-first order, you can add a column that tracks the depth of the search, for example:

WITH RECURSIVE search_tree(id, link, data, depth) AS (
    SELECT t.id, t.link, t.data, 0
    FROM tree t
  UNION ALL
    SELECT t.id, t.link, t.data, depth + 1
    FROM tree t, search_tree st
    WHERE t.id = st.link
)
SELECT * FROM search_tree ORDER BY depth;

To get a stable sort, add data columns as secondary sorting columns.

Tip

The recursive query evaluation algorithm produces its output in breadth-first search order. However, this is an implementation detail and it is perhaps unsound to rely on it. The order of the rows within each level is certainly undefined, so some explicit ordering might be desired in any case.

There is built-in syntax to compute a depth- or breadth-first sort column. For example:

WITH RECURSIVE search_tree(id, link, data) AS (
    SELECT t.id, t.link, t.data
    FROM tree t
  UNION ALL
    SELECT t.id, t.link, t.data
    FROM tree t, search_tree st
    WHERE t.id = st.link
) SEARCH DEPTH FIRST BY id SET ordercol
SELECT * FROM search_tree ORDER BY ordercol;

WITH RECURSIVE search_tree(id, link, data) AS (
    SELECT t.id, t.link, t.data
    FROM tree t
  UNION ALL
    SELECT t.id, t.link, t.data
    FROM tree t, search_tree st
    WHERE t.id = st.link
) SEARCH BREADTH FIRST BY id SET ordercol
SELECT * FROM search_tree ORDER BY ordercol;

This syntax is internally expanded to something similar to the above hand-written forms. The SEARCH clause specifies whether depth- or breadth first search is wanted, the list of columns to track for sorting, and a column name that will contain the result data that can be used for sorting. That column will implicitly be added to the output rows of the CTE.

7.8.2.2. Cycle Detection

When working with recursive queries it is important to be sure that the recursive part of the query will eventually return no tuples, or else the query will loop indefinitely. Sometimes, using UNION instead of UNION ALL can accomplish this by discarding rows that duplicate previous output rows. However, often a cycle does not involve output rows that are completely duplicate: it may be necessary to check just one or a few fields to see if the same point has been reached before. The standard method for handling such situations is to compute an array of the already-visited values. For example, consider again the following query that searches a table graph using a link field:

WITH RECURSIVE search_graph(id, link, data, depth) AS (
    SELECT g.id, g.link, g.data, 0
    FROM graph g
  UNION ALL
    SELECT g.id, g.link, g.data, sg.depth + 1
    FROM graph g, search_graph sg
    WHERE g.id = sg.link
)
SELECT * FROM search_graph;

This query will loop if the link relationships contain cycles. Because we require a “depth” output, just changing UNION ALL to UNION would not eliminate the looping. Instead we need to recognize whether we have reached the same row again while following a particular path of links. We add two columns is_cycle and path to the loop-prone query:

WITH RECURSIVE search_graph(id, link, data, depth, is_cycle, path) AS (
    SELECT g.id, g.link, g.data, 0,
      false,
      ARRAY[g.id]
    FROM graph g
  UNION ALL
    SELECT g.id, g.link, g.data, sg.depth + 1,
      g.id = ANY(path),
      path || g.id
    FROM graph g, search_graph sg
    WHERE g.id = sg.link AND NOT is_cycle
)
SELECT * FROM search_graph;

Aside from preventing cycles, the array value is often useful in its own right as representing the “path” taken to reach any particular row.

In the general case where more than one field needs to be checked to recognize a cycle, use an array of rows. For example, if we needed to compare fields f1 and f2:

WITH RECURSIVE search_graph(id, link, data, depth, is_cycle, path) AS (
    SELECT g.id, g.link, g.data, 0,
      false,
      ARRAY[ROW(g.f1, g.f2)]
    FROM graph g
  UNION ALL
    SELECT g.id, g.link, g.data, sg.depth + 1,
      ROW(g.f1, g.f2) = ANY(path),
      path || ROW(g.f1, g.f2)
    FROM graph g, search_graph sg
    WHERE g.id = sg.link AND NOT is_cycle
)
SELECT * FROM search_graph;

Tip

Omit the ROW() syntax in the common case where only one field needs to be checked to recognize a cycle. This allows a simple array rather than a composite-type array to be used, gaining efficiency.

There is built-in syntax to simplify cycle detection. The above query can also be written like this:

WITH RECURSIVE search_graph(id, link, data, depth) AS (
    SELECT g.id, g.link, g.data, 1
    FROM graph g
  UNION ALL
    SELECT g.id, g.link, g.data, sg.depth + 1
    FROM graph g, search_graph sg
    WHERE g.id = sg.link
) CYCLE id SET is_cycle USING path
SELECT * FROM search_graph;

and it will be internally rewritten to the above form. The CYCLE clause specifies first the list of columns to track for cycle detection, then a column name that will show whether a cycle has been detected, and finally the name of another column that will track the path. The cycle and path columns will implicitly be added to the output rows of the CTE.

Tip

The cycle path column is computed in the same way as the depth-first ordering column show in the previous section. A query can have both a SEARCH and a CYCLE clause, but a depth-first search specification and a cycle detection specification would create redundant computations, so it's more efficient to just use the CYCLE clause and order by the path column. If breadth-first ordering is wanted, then specifying both SEARCH and CYCLE can be useful.

A helpful trick for testing queries when you are not certain if they might loop is to place a LIMIT in the parent query. For example, this query would loop forever without the LIMIT:

WITH RECURSIVE t(n) AS (
    SELECT 1
  UNION ALL
    SELECT n+1 FROM t
)
SELECT n FROM t LIMIT 100;

This works because PostgreSQL's implementation evaluates only as many rows of a WITH query as are actually fetched by the parent query. Using this trick in production is not recommended, because other systems might work differently. Also, it usually won't work if you make the outer query sort the recursive query's results or join them to some other table, because in such cases the outer query will usually try to fetch all of the WITH query's output anyway.

7.8.3. Common Table Expression Materialization

A useful property of WITH queries is that they are normally evaluated only once per execution of the parent query, even if they are referred to more than once by the parent query or sibling WITH queries. Thus, expensive calculations that are needed in multiple places can be placed within a WITH query to avoid redundant work. Another possible application is to prevent unwanted multiple evaluations of functions with side-effects. However, the other side of this coin is that the optimizer is not able to push restrictions from the parent query down into a multiply-referenced WITH query, since that might affect all uses of the WITH query's output when it should affect only one. The multiply-referenced WITH query will be evaluated as written, without suppression of rows that the parent query might discard afterwards. (But, as mentioned above, evaluation might stop early if the reference(s) to the query demand only a limited number of rows.)

However, if a WITH query is non-recursive and side-effect-free (that is, it is a SELECT containing no volatile functions) then it can be folded into the parent query, allowing joint optimization of the two query levels. By default, this happens if the parent query references the WITH query just once, but not if it references the WITH query more than once. You can override that decision by specifying MATERIALIZED to force separate calculation of the WITH query, or by specifying NOT MATERIALIZED to force it to be merged into the parent query. The latter choice risks duplicate computation of the WITH query, but it can still give a net savings if each usage of the WITH query needs only a small part of the WITH query's full output.

A simple example of these rules is

WITH w AS (
    SELECT * FROM big_table
)
SELECT * FROM w WHERE key = 123;

This WITH query will be folded, producing the same execution plan as

SELECT * FROM big_table WHERE key = 123;

In particular, if there's an index on key, it will probably be used to fetch just the rows having key = 123. On the other hand, in

WITH w AS (
    SELECT * FROM big_table
)
SELECT * FROM w AS w1 JOIN w AS w2 ON w1.key = w2.ref
WHERE w2.key = 123;

the WITH query will be materialized, producing a temporary copy of big_table that is then joined with itself — without benefit of any index. This query will be executed much more efficiently if written as

WITH w AS NOT MATERIALIZED (
    SELECT * FROM big_table
)
SELECT * FROM w AS w1 JOIN w AS w2 ON w1.key = w2.ref
WHERE w2.key = 123;

so that the parent query's restrictions can be applied directly to scans of big_table.

An example where NOT MATERIALIZED could be undesirable is

WITH w AS (
    SELECT key, very_expensive_function(val) as f FROM some_table
)
SELECT * FROM w AS w1 JOIN w AS w2 ON w1.f = w2.f;

Here, materialization of the WITH query ensures that very_expensive_function is evaluated only once per table row, not twice.

The examples above only show WITH being used with SELECT, but it can be attached in the same way to INSERT, UPDATE, or DELETE. In each case it effectively provides temporary table(s) that can be referred to in the main command.

7.8.4. Data-Modifying Statements in WITH

You can use data-modifying statements (INSERT, UPDATE, or DELETE) in WITH. This allows you to perform several different operations in the same query. An example is:

WITH moved_rows AS (
    DELETE FROM products
    WHERE
        "date" >= '2010-10-01' AND
        "date" < '2010-11-01'
    RETURNING *
)
INSERT INTO products_log
SELECT * FROM moved_rows;

This query effectively moves rows from products to products_log. The DELETE in WITH deletes the specified rows from products, returning their contents by means of its RETURNING clause; and then the primary query reads that output and inserts it into products_log.

A fine point of the above example is that the WITH clause is attached to the INSERT, not the sub-SELECT within the INSERT. This is necessary because data-modifying statements are only allowed in WITH clauses that are attached to the top-level statement. However, normal WITH visibility rules apply, so it is possible to refer to the WITH statement's output from the sub-SELECT.

Data-modifying statements in WITH usually have RETURNING clauses (see Section 6.4), as shown in the example above. It is the output of the RETURNING clause, not the target table of the data-modifying statement, that forms the temporary table that can be referred to by the rest of the query. If a data-modifying statement in WITH lacks a RETURNING clause, then it forms no temporary table and cannot be referred to in the rest of the query. Such a statement will be executed nonetheless. A not-particularly-useful example is:

WITH t AS (
    DELETE FROM foo
)
DELETE FROM bar;

This example would remove all rows from tables foo and bar. The number of affected rows reported to the client would only include rows removed from bar.

Recursive self-references in data-modifying statements are not allowed. In some cases it is possible to work around this limitation by referring to the output of a recursive WITH, for example:

WITH RECURSIVE included_parts(sub_part, part) AS (
    SELECT sub_part, part FROM parts WHERE part = 'our_product'
  UNION ALL
    SELECT p.sub_part, p.part
    FROM included_parts pr, parts p
    WHERE p.part = pr.sub_part
)
DELETE FROM parts
  WHERE part IN (SELECT part FROM included_parts);

This query would remove all direct and indirect subparts of a product.

Data-modifying statements in WITH are executed exactly once, and always to completion, independently of whether the primary query reads all (or indeed any) of their output. Notice that this is different from the rule for SELECT in WITH: as stated in the previous section, execution of a SELECT is carried only as far as the primary query demands its output.

The sub-statements in WITH are executed concurrently with each other and with the main query. Therefore, when using data-modifying statements in WITH, the order in which the specified updates actually happen is unpredictable. All the statements are executed with the same snapshot (see Chapter 13), so they cannot “see” one another's effects on the target tables. This alleviates the effects of the unpredictability of the actual order of row updates, and means that RETURNING data is the only way to communicate changes between different WITH sub-statements and the main query. An example of this is that in

WITH t AS (
    UPDATE products SET price = price * 1.05
    RETURNING *
)
SELECT * FROM products;

the outer SELECT would return the original prices before the action of the UPDATE, while in

WITH t AS (
    UPDATE products SET price = price * 1.05
    RETURNING *
)
SELECT * FROM t;

the outer SELECT would return the updated data.

Trying to update the same row twice in a single statement is not supported. Only one of the modifications takes place, but it is not easy (and sometimes not possible) to reliably predict which one. This also applies to deleting a row that was already updated in the same statement: only the update is performed. Therefore you should generally avoid trying to modify a single row twice in a single statement. In particular avoid writing WITH sub-statements that could affect the same rows changed by the main statement or a sibling sub-statement. The effects of such a statement will not be predictable.

At present, any table used as the target of a data-modifying statement in WITH must not have a conditional rule, nor an ALSO rule, nor an INSTEAD rule that expands to multiple statements.

Prev | Up | Next
7.7. VALUES Lists | Home | Chapter 8. Data Types


Chapter 8. Data Types
Prev Up Part II. The SQL Language Home Next

Chapter 8. Data Types

Table of Contents

8.1. Numeric Types
8.1.1. Integer Types
8.1.2. Arbitrary Precision Numbers
8.1.3. Floating-Point Types
8.1.4. Serial Types
8.2. Monetary Types
8.3. Character Types
8.4. Binary Data Types
8.4.1. bytea Hex Format
8.4.2. bytea Escape Format
8.5. Date/Time Types
8.5.1. Date/Time Input
8.5.2. Date/Time Output
8.5.3. Time Zones
8.5.4. Interval Input
8.5.5. Interval Output
8.6. Boolean Type
8.7. Enumerated Types
8.7.1. Declaration of Enumerated Types
8.7.2. Ordering
8.7.3. Type Safety
8.7.4. Implementation Details
8.8. Geometric Types
8.8.1. Points
8.8.2. Lines
8.8.3. Line Segments
8.8.4. Boxes
8.8.5. Paths
8.8.6. Polygons
8.8.7. Circles
8.9. Network Address Types
8.9.1. inet
8.9.2. cidr
8.9.3. inet vs. cidr
8.9.4. macaddr
8.9.5. macaddr8
8.10. Bit String Types
8.11. Text Search Types
8.11.1. tsvector
8.11.2. tsquery
8.12. UUID Type
8.13. XML Type
8.13.1. Creating XML Values
8.13.2. Encoding Handling
8.13.3. Accessing XML Values
8.14. JSON Types
8.14.1. JSON Input and Output Syntax
8.14.2. Designing JSON Documents
8.14.3. jsonb Containment and Existence
8.14.4. jsonb Indexing
8.14.5. jsonb Subscripting
8.14.6. Transforms
8.14.7. jsonpath Type
8.15. Arrays
8.15.1. Declaration of Array Types
8.15.2. Array Value Input
8.15.3. Accessing Arrays
8.15.4. Modifying Arrays
8.15.5. Searching in Arrays
8.15.6. Array Input and Output Syntax
8.16. Composite Types
8.16.1. Declaration of Composite Types
8.16.2. Constructing Composite Values
8.16.3. Accessing Composite Types
8.16.4. Modifying Composite Types
8.16.5. Using Composite Types in Queries
8.16.6. Composite Type Input and Output Syntax
8.17. Range Types
8.17.1. Built-in Range and Multirange Types
8.17.2. Examples
8.17.3. Inclusive and Exclusive Bounds
8.17.4. Infinite (Unbounded) Ranges
8.17.5. Range Input/Output
8.17.6. Constructing Ranges and Multiranges
8.17.7. Discrete Range Types
8.17.8. Defining New Range Types
8.17.9. Indexing
8.17.10. Constraints on Ranges
8.18. Domain Types
8.19. Object Identifier Types
8.20. pg_lsn Type
8.21. Pseudo-Types

PostgreSQL has a rich set of native data types available to users. Users can add new types to PostgreSQL using the CREATE TYPE command.

Table 8.1 shows all the built-in general-purpose data types. Most of the alternative names listed in the “Aliases” column are the names used internally by PostgreSQL for historical reasons. In addition, some internally used or deprecated types are available, but are not listed here.

Table 8.1. Data Types

Name | Aliases | Description
bigint | int8 | signed eight-byte integer
bigserial | serial8 | autoincrementing eight-byte integer
bit [ (n) ] |  | fixed-length bit string
bit varying [ (n) ] | varbit [ (n) ] | variable-length bit string
boolean | bool | logical Boolean (true/false)
box |  | rectangular box on a plane
bytea |  | binary data (“byte array”)
character [ (n) ] | char [ (n) ] | fixed-length character string
character varying [ (n) ] | varchar [ (n) ] | variable-length character string
cidr |  | IPv4 or IPv6 network address
circle |  | circle on a plane
date |  | calendar date (year, month, day)
double precision | float8 | double precision floating-point number (8 bytes)
inet |  | IPv4 or IPv6 host address
integer | int, int4 | signed four-byte integer
interval [ fields ] [ (p) ] |  | time span
json |  | textual JSON data
jsonb |  | binary JSON data, decomposed
line |  | infinite line on a plane
lseg |  | line segment on a plane
macaddr |  | MAC (Media Access Control) address
macaddr8 |  | MAC (Media Access Control) address (EUI-64 format)
money |  | currency amount
numeric [ (p, s) ] | decimal [ (p, s) ] | exact numeric of selectable precision
path |  | geometric path on a plane
pg_lsn |  | PostgreSQL Log Sequence Number
pg_snapshot |  | user-level transaction ID snapshot
point |  | geometric point on a plane
polygon |  | closed geometric path on a plane
real | float4 | single precision floating-point number (4 bytes)
smallint | int2 | signed two-byte integer
smallserial | serial2 | autoincrementing two-byte integer
serial | serial4 | autoincrementing four-byte integer
text |  | variable-length character string
time [ (p) ] [ without time zone ] |  | time of day (no time zone)
time [ (p) ] with time zone | timetz | time of day, including time zone
timestamp [ (p) ] [ without time zone ] |  | date and time (no time zone)
timestamp [ (p) ] with time zone | timestamptz | date and time, including time zone
tsquery |  | text search query
tsvector |  | text search document
txid_snapshot |  | user-level transaction ID snapshot (deprecated; see pg_snapshot)
uuid |  | universally unique identifier
xml |  | XML data

Compatibility

The following types (or spellings thereof) are specified by SQL: bigint, bit, bit varying, boolean, char, character varying, character, varchar, date, double precision, integer, interval, numeric, decimal, real, smallint, time (with or without time zone), timestamp (with or without time zone), xml.

Each data type has an external representation determined by its input and output functions. Many of the built-in types have obvious external formats. However, several types are either unique to PostgreSQL, such as geometric paths, or have several possible formats, such as the date and time types. Some of the input and output functions are not invertible, i.e., the result of an output function might lose accuracy when compared to the original input.

Prev | Up | Next
7.8. WITH Queries (Common Table Expressions) | Home | 8.1. Numeric Types


8.1. Numeric Types
Prev Up Chapter 8. Data Types Home Next

8.1. Numeric Types

8.1.1. Integer Types
8.1.2. Arbitrary Precision Numbers
8.1.3. Floating-Point Types
8.1.4. Serial Types

Numeric types consist of two-, four-, and eight-byte integers, four- and eight-byte floating-point numbers, and selectable-precision decimals. Table 8.2 lists the available types.

Table 8.2. Numeric Types

Name | Storage Size | Description | Range
smallint | 2 bytes | small-range integer | -32768 to +32767
integer | 4 bytes | typical choice for integer | -2147483648 to +2147483647
bigint | 8 bytes | large-range integer | -9223372036854775808 to +9223372036854775807
decimal | variable | user-specified precision, exact | up to 131072 digits before the decimal point; up to 16383 digits after the decimal point
numeric | variable | user-specified precision, exact | up to 131072 digits before the decimal point; up to 16383 digits after the decimal point
real | 4 bytes | variable-precision, inexact | 6 decimal digits precision
double precision | 8 bytes | variable-precision, inexact | 15 decimal digits precision
smallserial | 2 bytes | small autoincrementing integer | 1 to 32767
serial | 4 bytes | autoincrementing integer | 1 to 2147483647
bigserial | 8 bytes | large autoincrementing integer | 1 to 9223372036854775807

The syntax of constants for the numeric types is described in Section 4.1.2. The numeric types have a full set of corresponding arithmetic operators and functions. Refer to Chapter 9 for more information. The following sections describe the types in detail.

8.1.1. Integer Types

The types smallint, integer, and bigint store whole numbers, that is, numbers without fractional components, of various ranges. Attempts to store values outside of the allowed range will result in an error.

The type integer is the common choice, as it offers the best balance between range, storage size, and performance. The smallint type is generally only used if disk space is at a premium. The bigint type is designed to be used when the range of the integer type is insufficient.

SQL only specifies the integer types integer (or int), smallint, and bigint. The type names int2, int4, and int8 are extensions, which are also used by some other SQL database systems.

8.1.2. Arbitrary Precision Numbers

The type numeric can store numbers with a very large number of digits. It is especially recommended for storing monetary amounts and other quantities where exactness is required. Calculations with numeric values yield exact results where possible, e.g., addition, subtraction, multiplication. However, calculations on numeric values are very slow compared to the integer types, or to the floating-point types described in the next section.

We use the following terms below: The precision of a numeric is the total count of significant digits in the whole number, that is, the number of digits to both sides of the decimal point. The scale of a numeric is the count of decimal digits in the fractional part, to the right of the decimal point. So the number 23.5141 has a precision of 6 and a scale of 4. Integers can be considered to have a scale of zero.

Both the maximum precision and the maximum scale of a numeric column can be configured. To declare a column of type numeric use the syntax:

NUMERIC(precision, scale)

The precision must be positive, the scale zero or positive. Alternatively:

NUMERIC(precision)

selects a scale of 0. Specifying:

NUMERIC

without any precision or scale creates an “unconstrained numeric” column in which numeric values of any length can be stored, up to the implementation limits. A column of this kind will not coerce input values to any particular scale, whereas numeric columns with a declared scale will coerce input values to that scale. (The SQL standard requires a default scale of 0, i.e., coercion to integer precision. We find this a bit useless. If you're concerned about portability, always specify the precision and scale explicitly.)

Note

The maximum precision that can be explicitly specified in a NUMERIC type declaration is 1000. An unconstrained NUMERIC column is subject to the limits described in Table 8.2.

If the scale of a value to be stored is greater than the declared scale of the column, the system will round the value to the specified number of fractional digits. Then, if the number of digits to the left of the decimal point exceeds the declared precision minus the declared scale, an error is raised.

Numeric values are physically stored without any extra leading or trailing zeroes. Thus, the declared precision and scale of a column are maximums, not fixed allocations. (In this sense the numeric type is more akin to varchar(n) than to char(n).) The actual storage requirement is two bytes for each group of four decimal digits, plus three to eight bytes overhead.

In addition to ordinary numeric values, the numeric type has several special values:

Infinity
-Infinity
NaN

These are adapted from the IEEE 754 standard, and represent “infinity”, “negative infinity”, and “not-a-number”, respectively. When writing these values as constants in an SQL command, you must put quotes around them, for example UPDATE table SET x = '-Infinity'. On input, these strings are recognized in a case-insensitive manner. The infinity values can alternatively be spelled inf and -inf.

The infinity values behave as per mathematical expectations. For example, Infinity plus any finite value equals Infinity, as does Infinity plus Infinity; but Infinity minus Infinity yields NaN (not a number), because it has no well-defined interpretation. Note that an infinity can only be stored in an unconstrained numeric column, because it notionally exceeds any finite precision limit.

The NaN (not a number) value is used to represent undefined calculational results. In general, any operation with a NaN input yields another NaN. The only exception is when the operation's other inputs are such that the same output would be obtained if the NaN were to be replaced by any finite or infinite numeric value; then, that output value is used for NaN too. (An example of this principle is that NaN raised to the zero power yields one.)

Note

In most implementations of the “not-a-number” concept, NaN is not considered equal to any other numeric value (including NaN). In order to allow numeric values to be sorted and used in tree-based indexes, PostgreSQL treats NaN values as equal, and greater than all non-NaN values.

The types decimal and numeric are equivalent. Both types are part of the SQL standard.

When rounding values, the numeric type rounds ties away from zero, while (on most machines) the real and double precision types round ties to the nearest even number. For example:

SELECT x,
  round(x::numeric) AS num_round,
  round(x::double precision) AS dbl_round
FROM generate_series(-3.5, 3.5, 1) as x;
  x   | num_round | dbl_round
------+-----------+-----------
 -3.5 |        -4 |        -4
 -2.5 |        -3 |        -2
 -1.5 |        -2 |        -2
 -0.5 |        -1 |        -0
  0.5 |         1 |         0
  1.5 |         2 |         2
  2.5 |         3 |         2
  3.5 |         4 |         4
(8 rows)

8.1.3. Floating-Point Types

The data types real and double precision are inexact, variable-precision numeric types. On all currently supported platforms, these types are implementations of IEEE Standard 754 for Binary Floating-Point Arithmetic (single and double precision, respectively), to the extent that the underlying processor, operating system, and compiler support it.

Inexact means that some values cannot be converted exactly to the internal format and are stored as approximations, so that storing and retrieving a value might show slight discrepancies. Managing these errors and how they propagate through calculations is the subject of an entire branch of mathematics and computer science and will not be discussed here, except for the following points:

  • If you require exact storage and calculations (such as for monetary amounts), use the numeric type instead.

  • If you want to do complicated calculations with these types for anything important, especially if you rely on certain behavior in boundary cases (infinity, underflow), you should evaluate the implementation carefully.

  • Comparing two floating-point values for equality might not always work as expected.

On all currently supported platforms, the real type has a range of around 1E-37 to 1E+37 with a precision of at least 6 decimal digits. The double precision type has a range of around 1E-307 to 1E+308 with a precision of at least 15 digits. Values that are too large or too small will cause an error. Rounding might take place if the precision of an input number is too high. Numbers too close to zero that are not representable as distinct from zero will cause an underflow error.

By default, floating point values are output in text form in their shortest precise decimal representation; the decimal value produced is closer to the true stored binary value than to any other value representable in the same binary precision. (However, the output value is currently never exactly midway between two representable values, in order to avoid a widespread bug where input routines do not properly respect the round-to-nearest-even rule.) This value will use at most 17 significant decimal digits for float8 values, and at most 9 digits for float4 values.

Note

This shortest-precise output format is much faster to generate than the historical rounded format.

For compatibility with output generated by older versions of PostgreSQL, and to allow the output precision to be reduced, the extra_float_digits parameter can be used to select rounded decimal output instead. Setting a value of 0 restores the previous default of rounding the value to 6 (for float4) or 15 (for float8) significant decimal digits. Setting a negative value reduces the number of digits further; for example -2 would round output to 4 or 13 digits respectively.

Any value of extra_float_digits greater than 0 selects the shortest-precise format.

Note

Applications that wanted precise values have historically had to set extra_float_digits to 3 to obtain them. For maximum compatibility between versions, they should continue to do so.

In addition to ordinary numeric values, the floating-point types have several special values:

Infinity
-Infinity
NaN

These represent the IEEE 754 special values “infinity”, “negative infinity”, and “not-a-number”, respectively. When writing these values as constants in an SQL command, you must put quotes around them, for example UPDATE table SET x = '-Infinity'. On input, these strings are recognized in a case-insensitive manner. The infinity values can alternatively be spelled inf and -inf.

Note

IEEE 754 specifies that NaN should not compare equal to any other floating-point value (including NaN). In order to allow floating-point values to be sorted and used in tree-based indexes, PostgreSQL treats NaN values as equal, and greater than all non-NaN values.

PostgreSQL also supports the SQL-standard notations float and float(p) for specifying inexact numeric types. Here, p specifies the minimum acceptable precision in binary digits. PostgreSQL accepts float(1) to float(24) as selecting the real type, while float(25) to float(53) select double precision. Values of p outside the allowed range draw an error. float with no precision specified is taken to mean double precision.

8.1.4. Serial Types

Note

This section describes a PostgreSQL-specific way to create an autoincrementing column. Another way is to use the SQL-standard identity column feature, described at CREATE TABLE.

The data types smallserial, serial and bigserial are not true types, but merely a notational convenience for creating unique identifier columns (similar to the AUTO_INCREMENT property supported by some other databases). In the current implementation, specifying:

CREATE TABLE tablename (
    colname SERIAL
);

is equivalent to specifying:

CREATE SEQUENCE tablename_colname_seq AS integer;
CREATE TABLE tablename (
    colname integer NOT NULL DEFAULT nextval('tablename_colname_seq')
);
ALTER SEQUENCE tablename_colname_seq OWNED BY tablename.colname;

Thus, we have created an integer column and arranged for its default values to be assigned from a sequence generator. A NOT NULL constraint is applied to ensure that a null value cannot be inserted. (In most cases you would also want to attach a UNIQUE or PRIMARY KEY constraint to prevent duplicate values from being inserted by accident, but this is not automatic.) Lastly, the sequence is marked as “owned by” the column, so that it will be dropped if the column or table is dropped.

Note

Because smallserial, serial and bigserial are implemented using sequences, there may be "holes" or gaps in the sequence of values which appears in the column, even if no rows are ever deleted. A value allocated from the sequence is still "used up" even if a row containing that value is never successfully inserted into the table column. This may happen, for example, if the inserting transaction rolls back. See nextval() in Section 9.17 for details.

To insert the next value of the sequence into the serial column, specify that the serial column should be assigned its default value. This can be done either by excluding the column from the list of columns in the INSERT statement, or through the use of the DEFAULT key word.

The type names serial and serial4 are equivalent: both create integer columns. The type names bigserial and serial8 work the same way, except that they create a bigint column. bigserial should be used if you anticipate the use of more than 231 identifiers over the lifetime of the table. The type names smallserial and serial2 also work the same way, except that they create a smallint column.

The sequence created for a serial column is automatically dropped when the owning column is dropped. You can drop the sequence without dropping the column, but this will force removal of the column default expression.

Prev | Up | Next
Chapter 8. Data Types | Home | 8.2. Monetary Types


8.2. Monetary Types
Prev Up Chapter 8. Data Types Home Next

8.2. Monetary Types

The money type stores a currency amount with a fixed fractional precision; see Table 8.3. The fractional precision is determined by the database's lc_monetary setting. The range shown in the table assumes there are two fractional digits. Input is accepted in a variety of formats, including integer and floating-point literals, as well as typical currency formatting, such as '$1,000.00'. Output is generally in the latter form but depends on the locale.

Table 8.3. Monetary Types

Name | Storage Size | Description | Range
money | 8 bytes | currency amount | -92233720368547758.08 to +92233720368547758.07

Since the output of this data type is locale-sensitive, it might not work to load money data into a database that has a different setting of lc_monetary. To avoid problems, before restoring a dump into a new database make sure lc_monetary has the same or equivalent value as in the database that was dumped.

Values of the numeric, int, and bigint data types can be cast to money. Conversion from the real and double precision data types can be done by casting to numeric first, for example:

SELECT '12.34'::float8::numeric::money;

However, this is not recommended. Floating point numbers should not be used to handle money due to the potential for rounding errors.

A money value can be cast to numeric without loss of precision. Conversion to other types could potentially lose precision, and must also be done in two stages:

SELECT '52093.89'::money::numeric::float8;

Division of a money value by an integer value is performed with truncation of the fractional part towards zero. To get a rounded result, divide by a floating-point value, or cast the money value to numeric before dividing and back to money afterwards. (The latter is preferable to avoid risking precision loss.) When a money value is divided by another money value, the result is double precision (i.e., a pure number, not money); the currency units cancel each other out in the division.

Prev | Up | Next
8.1. Numeric Types | Home | 8.3. Character Types


8.3. Character Types
Prev Up Chapter 8. Data Types Home Next

8.3. Character Types

Table 8.4. Character Types

Name | Description
character varying(n), varchar(n) | variable-length with limit
character(n), char(n) | fixed-length, blank padded
text | variable unlimited length

Table 8.4 shows the general-purpose character types available in PostgreSQL.

SQL defines two primary character types: character varying(n) and character(n), where n is a positive integer. Both of these types can store strings up to n characters (not bytes) in length. An attempt to store a longer string into a column of these types will result in an error, unless the excess characters are all spaces, in which case the string will be truncated to the maximum length. (This somewhat bizarre exception is required by the SQL standard.) If the string to be stored is shorter than the declared length, values of type character will be space-padded; values of type character varying will simply store the shorter string.

If one explicitly casts a value to character varying(n) or character(n), then an over-length value will be truncated to n characters without raising an error. (This too is required by the SQL standard.)

The notations varchar(n) and char(n) are aliases for character varying(n) and character(n), respectively. character without length specifier is equivalent to character(1). If character varying is used without length specifier, the type accepts strings of any size. The latter is a PostgreSQL extension.

In addition, PostgreSQL provides the text type, which stores strings of any length. Although the type text is not in the SQL standard, several other SQL database management systems have it as well.

Values of type character are physically padded with spaces to the specified width n, and are stored and displayed that way. However, trailing spaces are treated as semantically insignificant and disregarded when comparing two values of type character. In collations where whitespace is significant, this behavior can produce unexpected results; for example SELECT 'a '::CHAR(2) collate "C" < E'a\n'::CHAR(2) returns true, even though C locale would consider a space to be greater than a newline. Trailing spaces are removed when converting a character value to one of the other string types. Note that trailing spaces are semantically significant in character varying and text values, and when using pattern matching, that is LIKE and regular expressions.

The characters that can be stored in any of these data types are determined by the database character set, which is selected when the database is created. Regardless of the specific character set, the character with code zero (sometimes called NUL) cannot be stored. For more information refer to Section 24.3.

The storage requirement for a short string (up to 126 bytes) is 1 byte plus the actual string, which includes the space padding in the case of character. Longer strings have 4 bytes of overhead instead of 1. Long strings are compressed by the system automatically, so the physical requirement on disk might be less. Very long values are also stored in background tables so that they do not interfere with rapid access to shorter column values. In any case, the longest possible character string that can be stored is about 1 GB. (The maximum value that will be allowed for n in the data type declaration is less than that. It wouldn't be useful to change this because with multibyte character encodings the number of characters and bytes can be quite different. If you desire to store long strings with no specific upper limit, use text or character varying without a length specifier, rather than making up an arbitrary length limit.)

Tip

There is no performance difference among these three types, apart from increased storage space when using the blank-padded type, and a few extra CPU cycles to check the length when storing into a length-constrained column. While character(n) has performance advantages in some other database systems, there is no such advantage in PostgreSQL; in fact character(n) is usually the slowest of the three because of its additional storage costs. In most situations text or character varying should be used instead.

Refer to Section 4.1.2.1 for information about the syntax of string literals, and to Chapter 9 for information about available operators and functions.

Example 8.1. Using the Character Types

CREATE TABLE test1 (a character(4));
INSERT INTO test1 VALUES ('ok');
SELECT a, char_length(a) FROM test1; -- (1)

  a   | char_length
------+-------------
 ok   |           2

CREATE TABLE test2 (b varchar(5));
INSERT INTO test2 VALUES ('ok');
INSERT INTO test2 VALUES ('good      ');
INSERT INTO test2 VALUES ('too long');
ERROR:  value too long for type character varying(5)
INSERT INTO test2 VALUES ('too long'::varchar(5)); -- explicit truncation
SELECT b, char_length(b) FROM test2;

   b   | char_length
-------+-------------
 ok    |           2
 good  |           5
 too l |           5

(1)

The char_length function is discussed in Section 9.4.

There are two other fixed-length character types in PostgreSQL, shown in Table 8.5. The name type exists only for the storage of identifiers in the internal system catalogs and is not intended for use by the general user. Its length is currently defined as 64 bytes (63 usable characters plus terminator) but should be referenced using the constant NAMEDATALEN in C source code. The length is set at compile time (and is therefore adjustable for special uses); the default maximum length might change in a future release. The type "char" (note the quotes) is different from char(1) in that it only uses one byte of storage. It is internally used in the system catalogs as a simplistic enumeration type.

Table 8.5. Special Character Types

Name | Storage Size | Description
"char" | 1 byte | single-byte internal type
name | 64 bytes | internal type for object names

Prev | Up | Next
8.2. Monetary Types | Home | 8.4. Binary Data Types


8.4. Binary Data Types
Prev Up Chapter 8. Data Types Home Next

8.4. Binary Data Types

8.4.1. bytea Hex Format
8.4.2. bytea Escape Format

The bytea data type allows storage of binary strings; see Table 8.6.

Table 8.6. Binary Data Types

Name | Storage Size | Description
bytea | 1 or 4 bytes plus the actual binary string | variable-length binary string

A binary string is a sequence of octets (or bytes). Binary strings are distinguished from character strings in two ways. First, binary strings specifically allow storing octets of value zero and other “non-printable” octets (usually, octets outside the decimal range 32 to 126). Character strings disallow zero octets, and also disallow any other octet values and sequences of octet values that are invalid according to the database's selected character set encoding. Second, operations on binary strings process the actual bytes, whereas the processing of character strings depends on locale settings. In short, binary strings are appropriate for storing data that the programmer thinks of as “raw bytes”, whereas character strings are appropriate for storing text.

The bytea type supports two formats for input and output: “hex” format and PostgreSQL's historical “escape” format. Both of these are always accepted on input. The output format depends on the configuration parameter bytea_output; the default is hex. (Note that the hex format was introduced in PostgreSQL 9.0; earlier versions and some tools don't understand it.)

The SQL standard defines a different binary string type, called BLOB or BINARY LARGE OBJECT. The input format is different from bytea, but the provided functions and operators are mostly the same.

8.4.1. bytea Hex Format

The “hex” format encodes binary data as 2 hexadecimal digits per byte, most significant nibble first. The entire string is preceded by the sequence \x (to distinguish it from the escape format). In some contexts, the initial backslash may need to be escaped by doubling it (see Section 4.1.2.1). For input, the hexadecimal digits can be either upper or lower case, and whitespace is permitted between digit pairs (but not within a digit pair nor in the starting \x sequence). The hex format is compatible with a wide range of external applications and protocols, and it tends to be faster to convert than the escape format, so its use is preferred.

Example:

SELECT '\xDEADBEEF';

8.4.2. bytea Escape Format

The “escape” format is the traditional PostgreSQL format for the bytea type. It takes the approach of representing a binary string as a sequence of ASCII characters, while converting those bytes that cannot be represented as an ASCII character into special escape sequences. If, from the point of view of the application, representing bytes as characters makes sense, then this representation can be convenient. But in practice it is usually confusing because it fuzzes up the distinction between binary strings and character strings, and also the particular escape mechanism that was chosen is somewhat unwieldy. Therefore, this format should probably be avoided for most new applications.

When entering bytea values in escape format, octets of certain values must be escaped, while all octet values can be escaped. In general, to escape an octet, convert it into its three-digit octal value and precede it by a backslash. Backslash itself (octet decimal value 92) can alternatively be represented by double backslashes. Table 8.7 shows the characters that must be escaped, and gives the alternative escape sequences where applicable.

Table 8.7. bytea Literal Escaped Octets

Decimal Octet Value | Description | Escaped Input Representation | Example | Hex Representation
0 | zero octet | '\000' | '\000'::bytea | \x00
39 | single quote | '''' or '\047' | ''''::bytea | \x27
92 | backslash | '\\' or '\134' | '\\'::bytea | \x5c
0 to 31 and 127 to 255 | “non-printable” octets | '\xxx' (octal value) | '\001'::bytea | \x01

The requirement to escape non-printable octets varies depending on locale settings. In some instances you can get away with leaving them unescaped.

The reason that single quotes must be doubled, as shown in Table 8.7, is that this is true for any string literal in an SQL command. The generic string-literal parser consumes the outermost single quotes and reduces any pair of single quotes to one data character. What the bytea input function sees is just one single quote, which it treats as a plain data character. However, the bytea input function treats backslashes as special, and the other behaviors shown in Table 8.7 are implemented by that function.

In some contexts, backslashes must be doubled compared to what is shown above, because the generic string-literal parser will also reduce pairs of backslashes to one data character; see Section 4.1.2.1.

Bytea octets are output in hex format by default. If you change bytea_output to escape, “non-printable” octets are converted to their equivalent three-digit octal value and preceded by one backslash. Most “printable” octets are output by their standard representation in the client character set, e.g.:

SET bytea_output = 'escape';

SELECT 'abc \153\154\155 \052\251\124'::bytea;
     bytea
----------------
 abc klm *\251T

The octet with decimal value 92 (backslash) is doubled in the output. Details are in Table 8.8.

Table 8.8. bytea Output Escaped Octets

Decimal Octet Value | Description | Escaped Output Representation | Example | Output Result
92 | backslash | \\ | '\134'::bytea | \\
0 to 31 and 127 to 255 | “non-printable” octets | \xxx (octal value) | '\001'::bytea | \001
32 to 126 | “printable” octets | client character set representation | '\176'::bytea | ~

Depending on the front end to PostgreSQL you use, you might have additional work to do in terms of escaping and unescaping bytea strings. For example, you might also have to escape line feeds and carriage returns if your interface automatically translates these.

Prev | Up | Next
8.3. Character Types | Home | 8.5. Date/Time Types


8.5. Date/Time Types
Prev Up Chapter 8. Data Types Home Next

8.5. Date/Time Types

8.5.1. Date/Time Input
8.5.2. Date/Time Output
8.5.3. Time Zones
8.5.4. Interval Input
8.5.5. Interval Output

PostgreSQL supports the full set of SQL date and time types, shown in Table 8.9. The operations available on these data types are described in Section 9.9. Dates are counted according to the Gregorian calendar, even in years before that calendar was introduced (see Section B.6 for more information).

Table 8.9. Date/Time Types

Name | Storage Size | Description | Low Value | High Value | Resolution
timestamp [ (p) ] [ without time zone ] | 8 bytes | both date and time (no time zone) | 4713 BC | 294276 AD | 1 microsecond
timestamp [ (p) ] with time zone | 8 bytes | both date and time, with time zone | 4713 BC | 294276 AD | 1 microsecond
date | 4 bytes | date (no time of day) | 4713 BC | 5874897 AD | 1 day
time [ (p) ] [ without time zone ] | 8 bytes | time of day (no date) | 00:00:00 | 24:00:00 | 1 microsecond
time [ (p) ] with time zone | 12 bytes | time of day (no date), with time zone | 00:00:00+1559 | 24:00:00-1559 | 1 microsecond
interval [ fields ] [ (p) ] | 16 bytes | time interval | -178000000 years | 178000000 years | 1 microsecond

Note

The SQL standard requires that writing just timestamp be equivalent to timestamp without time zone, and PostgreSQL honors that behavior. timestamptz is accepted as an abbreviation for timestamp with time zone; this is a PostgreSQL extension.

time, timestamp, and interval accept an optional precision value p which specifies the number of fractional digits retained in the seconds field. By default, there is no explicit bound on precision. The allowed range of p is from 0 to 6.

The interval type has an additional option, which is to restrict the set of stored fields by writing one of these phrases:

YEAR
MONTH
DAY
HOUR
MINUTE
SECOND
YEAR TO MONTH
DAY TO HOUR
DAY TO MINUTE
DAY TO SECOND
HOUR TO MINUTE
HOUR TO SECOND
MINUTE TO SECOND

Note that if both fields and p are specified, the fields must include SECOND, since the precision applies only to the seconds.

The type time with time zone is defined by the SQL standard, but the definition exhibits properties which lead to questionable usefulness. In most cases, a combination of date, time, timestamp without time zone, and timestamp with time zone should provide a complete range of date/time functionality required by any application.

8.5.1. Date/Time Input

Date and time input is accepted in almost any reasonable format, including ISO 8601, SQL-compatible, traditional POSTGRES, and others. For some formats, ordering of day, month, and year in date input is ambiguous and there is support for specifying the expected ordering of these fields. Set the DateStyle parameter to MDY to select month-day-year interpretation, DMY to select day-month-year interpretation, or YMD to select year-month-day interpretation.

PostgreSQL is more flexible in handling date/time input than the SQL standard requires. See Appendix B for the exact parsing rules of date/time input and for the recognized text fields including months, days of the week, and time zones.

Remember that any date or time literal input needs to be enclosed in single quotes, like text strings. Refer to Section 4.1.2.7 for more information. SQL requires the following syntax

type [ (p) ] 'value'

where p is an optional precision specification giving the number of fractional digits in the seconds field. Precision can be specified for time, timestamp, and interval types, and can range from 0 to 6. If no precision is specified in a constant specification, it defaults to the precision of the literal value (but not more than 6 digits).

8.5.1.1. Dates

Table 8.10 shows some possible inputs for the date type.

Table 8.10. Date Input

Example | Description
1999-01-08 | ISO 8601; January 8 in any mode (recommended format)
January 8, 1999 | unambiguous in any datestyle input mode
1/8/1999 | January 8 in MDY mode; August 1 in DMY mode
1/18/1999 | January 18 in MDY mode; rejected in other modes
01/02/03 | January 2, 2003 in MDY mode; February 1, 2003 in DMY mode; February 3, 2001 in YMD mode
1999-Jan-08 | January 8 in any mode
Jan-08-1999 | January 8 in any mode
08-Jan-1999 | January 8 in any mode
99-Jan-08 | January 8 in YMD mode, else error
08-Jan-99 | January 8, except error in YMD mode
Jan-08-99 | January 8, except error in YMD mode
19990108 | ISO 8601; January 8, 1999 in any mode
990108 | ISO 8601; January 8, 1999 in any mode
1999.008 | year and day of year
J2451187 | Julian date
January 8, 99 BC | year 99 BC

8.5.1.2. Times

The time-of-day types are time [ (p) ] without time zone and time [ (p) ] with time zone. time alone is equivalent to time without time zone.

Valid input for these types consists of a time of day followed by an optional time zone. (See Table 8.11 and Table 8.12.) If a time zone is specified in the input for time without time zone, it is silently ignored. You can also specify a date but it will be ignored, except when you use a time zone name that involves a daylight-savings rule, such as America/New_York. In this case specifying the date is required in order to determine whether standard or daylight-savings time applies. The appropriate time zone offset is recorded in the time with time zone value.

Table 8.11. Time Input

Example | Description
04:05:06.789 | ISO 8601
04:05:06 | ISO 8601
04:05 | ISO 8601
040506 | ISO 8601
04:05 AM | same as 04:05; AM does not affect value
04:05 PM | same as 16:05; input hour must be <= 12
04:05:06.789-8 | ISO 8601, with time zone as UTC offset
04:05:06-08:00 | ISO 8601, with time zone as UTC offset
04:05-08:00 | ISO 8601, with time zone as UTC offset
040506-08 | ISO 8601, with time zone as UTC offset
040506+0730 | ISO 8601, with fractional-hour time zone as UTC offset
040506+07:30:00 | UTC offset specified to seconds (not allowed in ISO 8601)
04:05:06 PST | time zone specified by abbreviation
2003-04-12 04:05:06 America/New_York | time zone specified by full name

Table 8.12. Time Zone Input

Example | Description
PST | Abbreviation (for Pacific Standard Time)
America/New_York | Full time zone name
PST8PDT | POSIX-style time zone specification
-8:00:00 | UTC offset for PST
-8:00 | UTC offset for PST (ISO 8601 extended format)
-800 | UTC offset for PST (ISO 8601 basic format)
-8 | UTC offset for PST (ISO 8601 basic format)
zulu | Military abbreviation for UTC
z | Short form of zulu (also in ISO 8601)

Refer to Section 8.5.3 for more information on how to specify time zones.

8.5.1.3. Time Stamps

Valid input for the time stamp types consists of the concatenation of a date and a time, followed by an optional time zone, followed by an optional AD or BC. (Alternatively, AD/BC can appear before the time zone, but this is not the preferred ordering.) Thus:

1999-01-08 04:05:06

and:

1999-01-08 04:05:06 -8:00

are valid values, which follow the ISO 8601 standard. In addition, the common format:

January 8 04:05:06 1999 PST

is supported.

The SQL standard differentiates timestamp without time zone and timestamp with time zone literals by the presence of a “+” or “-” symbol and time zone offset after the time. Hence, according to the standard,

TIMESTAMP '2004-10-19 10:23:54'

is a timestamp without time zone, while

TIMESTAMP '2004-10-19 10:23:54+02'

is a timestamp with time zone. PostgreSQL never examines the content of a literal string before determining its type, and therefore will treat both of the above as timestamp without time zone. To ensure that a literal is treated as timestamp with time zone, give it the correct explicit type:

TIMESTAMP WITH TIME ZONE '2004-10-19 10:23:54+02'

In a literal that has been determined to be timestamp without time zone, PostgreSQL will silently ignore any time zone indication. That is, the resulting value is derived from the date/time fields in the input value, and is not adjusted for time zone.

For timestamp with time zone, the internally stored value is always in UTC (Universal Coordinated Time, traditionally known as Greenwich Mean Time, GMT). An input value that has an explicit time zone specified is converted to UTC using the appropriate offset for that time zone. If no time zone is stated in the input string, then it is assumed to be in the time zone indicated by the system's TimeZone parameter, and is converted to UTC using the offset for the timezone zone.

When a timestamp with time zone value is output, it is always converted from UTC to the current timezone zone, and displayed as local time in that zone. To see the time in another time zone, either change timezone or use the AT TIME ZONE construct (see Section 9.9.4).

Conversions between timestamp without time zone and timestamp with time zone normally assume that the timestamp without time zone value should be taken or given as timezone local time. A different time zone can be specified for the conversion using AT TIME ZONE.

8.5.1.4. Special Values

PostgreSQL supports several special date/time input values for convenience, as shown in Table 8.13. The values infinity and -infinity are specially represented inside the system and will be displayed unchanged; but the others are simply notational shorthands that will be converted to ordinary date/time values when read. (In particular, now and related strings are converted to a specific time value as soon as they are read.) All of these values need to be enclosed in single quotes when used as constants in SQL commands.

Table 8.13. Special Date/Time Inputs

Input String | Valid Types | Description
epoch | date, timestamp | 1970-01-01 00:00:00+00 (Unix system time zero)
infinity | date, timestamp | later than all other time stamps
-infinity | date, timestamp | earlier than all other time stamps
now | date, time, timestamp | current transaction's start time
today | date, timestamp | midnight (00:00) today
tomorrow | date, timestamp | midnight (00:00) tomorrow
yesterday | date, timestamp | midnight (00:00) yesterday
allballs | time | 00:00:00.00 UTC

The following SQL-compatible functions can also be used to obtain the current time value for the corresponding data type: CURRENT_DATE, CURRENT_TIME, CURRENT_TIMESTAMP, LOCALTIME, LOCALTIMESTAMP. (See Section 9.9.5.) Note that these are SQL functions and are not recognized in data input strings.

Caution

While the input strings now, today, tomorrow, and yesterday are fine to use in interactive SQL commands, they can have surprising behavior when the command is saved to be executed later, for example in prepared statements, views, and function definitions. The string can be converted to a specific time value that continues to be used long after it becomes stale. Use one of the SQL functions instead in such contexts. For example, CURRENT_DATE + 1 is safer than 'tomorrow'::date.

8.5.2. Date/Time Output

The output format of the date/time types can be set to one of the four styles ISO 8601, SQL (Ingres), traditional POSTGRES (Unix date format), or German. The default is the ISO format. (The SQL standard requires the use of the ISO 8601 format. The name of the “SQL” output format is a historical accident.) Table 8.14 shows examples of each output style. The output of the date and time types is generally only the date or time part in accordance with the given examples. However, the POSTGRES style outputs date-only values in ISO format.

Table 8.14. Date/Time Output Styles

Style Specification | Description | Example
ISO | ISO 8601, SQL standard | 1997-12-17 07:37:16-08
SQL | traditional style | 12/17/1997 07:37:16.00 PST
Postgres | original style | Wed Dec 17 07:37:16 1997 PST
German | regional style | 17.12.1997 07:37:16.00 PST

Note

ISO 8601 specifies the use of uppercase letter T to separate the date and time. PostgreSQL accepts that format on input, but on output it uses a space rather than T, as shown above. This is for readability and for consistency with RFC 3339 as well as some other database systems.

In the SQL and POSTGRES styles, day appears before month if DMY field ordering has been specified, otherwise month appears before day. (See Section 8.5.1 for how this setting also affects interpretation of input values.) Table 8.15 shows examples.

Table 8.15. Date Order Conventions

datestyle Setting | Input Ordering | Example Output
SQL, DMY | day/month/year | 17/12/1997 15:37:16.00 CET
SQL, MDY | month/day/year | 12/17/1997 07:37:16.00 PST
Postgres, DMY | day/month/year | Wed 17 Dec 07:37:16 1997 PST

In the ISO style, the time zone is always shown as a signed numeric offset from UTC, with positive sign used for zones east of Greenwich. The offset will be shown as hh (hours only) if it is an integral number of hours, else as hh:mm if it is an integral number of minutes, else as hh:mm:ss. (The third case is not possible with any modern time zone standard, but it can appear when working with timestamps that predate the adoption of standardized time zones.) In the other date styles, the time zone is shown as an alphabetic abbreviation if one is in common use in the current zone. Otherwise it appears as a signed numeric offset in ISO 8601 basic format (hh or hhmm).

The date/time style can be selected by the user using the SET datestyle command, the DateStyle parameter in the postgresql.conf configuration file, or the PGDATESTYLE environment variable on the server or client.

The formatting function to_char (see Section 9.8) is also available as a more flexible way to format date/time output.

8.5.3. Time Zones

Time zones, and time-zone conventions, are influenced by political decisions, not just earth geometry. Time zones around the world became somewhat standardized during the 1900s, but continue to be prone to arbitrary changes, particularly with respect to daylight-savings rules. PostgreSQL uses the widely-used IANA (Olson) time zone database for information about historical time zone rules. For times in the future, the assumption is that the latest known rules for a given time zone will continue to be observed indefinitely far into the future.

PostgreSQL endeavors to be compatible with the SQL standard definitions for typical usage. However, the SQL standard has an odd mix of date and time types and capabilities. Two obvious problems are:

  • Although the date type cannot have an associated time zone, the time type can. Time zones in the real world have little meaning unless associated with a date as well as a time, since the offset can vary through the year with daylight-saving time boundaries.

  • The default time zone is specified as a constant numeric offset from UTC. It is therefore impossible to adapt to daylight-saving time when doing date/time arithmetic across DST boundaries.

To address these difficulties, we recommend using date/time types that contain both date and time when using time zones. We do not recommend using the type time with time zone (though it is supported by PostgreSQL for legacy applications and for compliance with the SQL standard). PostgreSQL assumes your local time zone for any type containing only date or time.

All timezone-aware dates and times are stored internally in UTC. They are converted to local time in the zone specified by the TimeZone configuration parameter before being displayed to the client.

PostgreSQL allows you to specify time zones in three different forms:

  • A full time zone name, for example America/New_York. The recognized time zone names are listed in the pg_timezone_names view (see Section 52.94). PostgreSQL uses the widely-used IANA time zone data for this purpose, so the same time zone names are also recognized by other software.

  • A time zone abbreviation, for example PST. Such a specification merely defines a particular offset from UTC, in contrast to full time zone names which can imply a set of daylight savings transition rules as well. The recognized abbreviations are listed in the pg_timezone_abbrevs view (see Section 52.93). You cannot set the configuration parameters TimeZone or log_timezone to a time zone abbreviation, but you can use abbreviations in date/time input values and with the AT TIME ZONE operator.

  • In addition to the timezone names and abbreviations, PostgreSQL will accept POSIX-style time zone specifications, as described in Section B.5. This option is not normally preferable to using a named time zone, but it may be necessary if no suitable IANA time zone entry is available.

In short, this is the difference between abbreviations and full names: abbreviations represent a specific offset from UTC, whereas many of the full names imply a local daylight-savings time rule, and so have two possible UTC offsets. As an example, 2014-06-04 12:00 America/New_York represents noon local time in New York, which for this particular date was Eastern Daylight Time (UTC-4). So 2014-06-04 12:00 EDT specifies that same time instant. But 2014-06-04 12:00 EST specifies noon Eastern Standard Time (UTC-5), regardless of whether daylight savings was nominally in effect on that date.

To complicate matters, some jurisdictions have used the same timezone abbreviation to mean different UTC offsets at different times; for example, in Moscow MSK has meant UTC+3 in some years and UTC+4 in others. PostgreSQL interprets such abbreviations according to whatever they meant (or had most recently meant) on the specified date; but, as with the EST example above, this is not necessarily the same as local civil time on that date.

In all cases, timezone names and abbreviations are recognized case-insensitively. (This is a change from PostgreSQL versions prior to 8.2, which were case-sensitive in some contexts but not others.)

Neither timezone names nor abbreviations are hard-wired into the server; they are obtained from configuration files stored under .../share/timezone/ and .../share/timezonesets/ of the installation directory (see Section B.4).

The TimeZone configuration parameter can be set in the file postgresql.conf, or in any of the other standard ways described in Chapter 20. There are also some special ways to set it:

  • The SQL command SET TIME ZONE sets the time zone for the session. This is an alternative spelling of SET TIMEZONE TO with a more SQL-spec-compatible syntax.

  • The PGTZ environment variable is used by libpq clients to send a SET TIME ZONE command to the server upon connection.

8.5.4. Interval Input

interval values can be written using the following verbose syntax:

[@] quantity unit [quantity unit...] [direction]

where quantity is a number (possibly signed); unit is microsecond, millisecond, second, minute, hour, day, week, month, year, decade, century, millennium, or abbreviations or plurals of these units; direction can be ago or empty. The at sign (@) is optional noise. The amounts of the different units are implicitly added with appropriate sign accounting. ago negates all the fields. This syntax is also used for interval output, if IntervalStyle is set to postgres_verbose.

Quantities of days, hours, minutes, and seconds can be specified without explicit unit markings. For example, '1 12:59:10' is read the same as '1 day 12 hours 59 min 10 sec'. Also, a combination of years and months can be specified with a dash; for example '200-10' is read the same as '200 years 10 months'. (These shorter forms are in fact the only ones allowed by the SQL standard, and are used for output when IntervalStyle is set to sql_standard.)

Interval values can also be written as ISO 8601 time intervals, using either the “format with designators” of the standard's section 4.4.3.2 or the “alternative format” of section 4.4.3.3. The format with designators looks like this:

P quantity unit [ quantity unit ...] [ T [ quantity unit ...]]

The string must start with a P, and may include a T that introduces the time-of-day units. The available unit abbreviations are given in Table 8.16. Units may be omitted, and may be specified in any order, but units smaller than a day must appear after T. In particular, the meaning of M depends on whether it is before or after T.

Table 8.16. ISO 8601 Interval Unit Abbreviations

Abbreviation | Meaning
Y | Years
M | Months (in the date part)
W | Weeks
D | Days
H | Hours
M | Minutes (in the time part)
S | Seconds

In the alternative format:

P [ years-months-days ] [ T hours:minutes:seconds ]

the string must begin with P, and a T separates the date and time parts of the interval. The values are given as numbers similar to ISO 8601 dates.

When writing an interval constant with a fields specification, or when assigning a string to an interval column that was defined with a fields specification, the interpretation of unmarked quantities depends on the fields. For example INTERVAL '1' YEAR is read as 1 year, whereas INTERVAL '1' means 1 second. Also, field values “to the right” of the least significant field allowed by the fields specification are silently discarded. For example, writing INTERVAL '1 day 2:03:04' HOUR TO MINUTE results in dropping the seconds field, but not the day field.

According to the SQL standard all fields of an interval value must have the same sign, so a leading negative sign applies to all fields; for example the negative sign in the interval literal '-1 2:03:04' applies to both the days and hour/minute/second parts. PostgreSQL allows the fields to have different signs, and traditionally treats each field in the textual representation as independently signed, so that the hour/minute/second part is considered positive in this example. If IntervalStyle is set to sql_standard then a leading sign is considered to apply to all fields (but only if no additional signs appear). Otherwise the traditional PostgreSQL interpretation is used. To avoid ambiguity, it's recommended to attach an explicit sign to each field if any field is negative.

Field values can have fractional parts: for example, '1.5 weeks' or '01:02:03.45'. However, because interval internally stores only three integer units (months, days, microseconds), fractional units must be spilled to smaller units. Fractional parts of units greater than months are truncated to be an integer number of months, e.g. '1.5 years' becomes '1 year 6 mons'. Fractional parts of weeks and days are computed to be an integer number of days and microseconds, assuming 30 days per month and 24 hours per day, e.g., '1.75 months' becomes 1 mon 22 days 12:00:00. Only seconds will ever be shown as fractional on output.

Table 8.17 shows some examples of valid interval input.

Table 8.17. Interval Input

Example | Description
1-2 | SQL standard format: 1 year 2 months
3 4:05:06 | SQL standard format: 3 days 4 hours 5 minutes 6 seconds
1 year 2 months 3 days 4 hours 5 minutes 6 seconds | Traditional Postgres format: 1 year 2 months 3 days 4 hours 5 minutes 6 seconds
P1Y2M3DT4H5M6S | ISO 8601 “format with designators”: same meaning as above
P0001-02-03T04:05:06 | ISO 8601 “alternative format”: same meaning as above

Internally interval values are stored as months, days, and microseconds. This is done because the number of days in a month varies, and a day can have 23 or 25 hours if a daylight savings time adjustment is involved. The months and days fields are integers while the microseconds field can store fractional seconds. Because intervals are usually created from constant strings or timestamp subtraction, this storage method works well in most cases, but can cause unexpected results:

SELECT EXTRACT(hours from '80 minutes'::interval);
 date_part
-----------
         1

SELECT EXTRACT(days from '80 hours'::interval);
 date_part
-----------
         0

Functions justify_days and justify_hours are available for adjusting days and hours that overflow their normal ranges.

8.5.5. Interval Output

The output format of the interval type can be set to one of the four styles sql_standard, postgres, postgres_verbose, or iso_8601, using the command SET intervalstyle. The default is the postgres format. Table 8.18 shows examples of each output style.

The sql_standard style produces output that conforms to the SQL standard's specification for interval literal strings, if the interval value meets the standard's restrictions (either year-month only or day-time only, with no mixing of positive and negative components). Otherwise the output looks like a standard year-month literal string followed by a day-time literal string, with explicit signs added to disambiguate mixed-sign intervals.

The output of the postgres style matches the output of PostgreSQL releases prior to 8.4 when the DateStyle parameter was set to ISO.

The output of the postgres_verbose style matches the output of PostgreSQL releases prior to 8.4 when the DateStyle parameter was set to non-ISO output.

The output of the iso_8601 style matches the “format with designators” described in section 4.4.3.2 of the ISO 8601 standard.

Table 8.18. Interval Output Style Examples

Style Specification | Year-Month Interval | Day-Time Interval | Mixed Interval
sql_standard | 1-2 | 3 4:05:06 | -1-2 +3 -4:05:06
postgres | 1 year 2 mons | 3 days 04:05:06 | -1 year -2 mons +3 days -04:05:06
postgres_verbose | @ 1 year 2 mons | @ 3 days 4 hours 5 mins 6 secs | @ 1 year 2 mons -3 days 4 hours 5 mins 6 secs ago
iso_8601 | P1Y2M | P3DT4H5M6S | P-1Y-2M3D​T-4H-5M-6S

Prev | Up | Next
8.4. Binary Data Types | Home | 8.6. Boolean Type


8.6. Boolean Type
Prev Up Chapter 8. Data Types Home Next

8.6. Boolean Type

PostgreSQL provides the standard SQL type boolean; see Table 8.19. The boolean type can have several states: “true”, “false”, and a third state, “unknown”, which is represented by the SQL null value.

Table 8.19. Boolean Data Type

Name | Storage Size | Description
boolean | 1 byte | state of true or false

Boolean constants can be represented in SQL queries by the SQL key words TRUE, FALSE, and NULL.

The datatype input function for type boolean accepts these string representations for the “true” state:

true
yes
on
1

and these representations for the “false” state:

false
no
off
0

Unique prefixes of these strings are also accepted, for example t or n. Leading or trailing whitespace is ignored, and case does not matter.

The datatype output function for type boolean always emits either t or f, as shown in Example 8.2.

Example 8.2. Using the boolean Type

CREATE TABLE test1 (a boolean, b text);
INSERT INTO test1 VALUES (TRUE, 'sic est');
INSERT INTO test1 VALUES (FALSE, 'non est');
SELECT * FROM test1;
 a |    b
---+---------
 t | sic est
 f | non est

SELECT * FROM test1 WHERE a;
 a |    b
---+---------
 t | sic est

The key words TRUE and FALSE are the preferred (SQL-compliant) method for writing Boolean constants in SQL queries. But you can also use the string representations by following the generic string-literal constant syntax described in Section 4.1.2.7, for example 'yes'::boolean.

Note that the parser automatically understands that TRUE and FALSE are of type boolean, but this is not so for NULL because that can have any type. So in some contexts you might have to cast NULL to boolean explicitly, for example NULL::boolean. Conversely, the cast can be omitted from a string-literal Boolean value in contexts where the parser can deduce that the literal must be of type boolean.

Prev | Up | Next
8.5. Date/Time Types | Home | 8.7. Enumerated Types


8.7. Enumerated Types
Prev Up Chapter 8. Data Types Home Next

8.7. Enumerated Types

8.7.1. Declaration of Enumerated Types
8.7.2. Ordering
8.7.3. Type Safety
8.7.4. Implementation Details

Enumerated (enum) types are data types that comprise a static, ordered set of values. They are equivalent to the enum types supported in a number of programming languages. An example of an enum type might be the days of the week, or a set of status values for a piece of data.

8.7.1. Declaration of Enumerated Types

Enum types are created using the CREATE TYPE command, for example:

CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy');

Once created, the enum type can be used in table and function definitions much like any other type:

CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy');
CREATE TABLE person (
    name text,
    current_mood mood
);
INSERT INTO person VALUES ('Moe', 'happy');
SELECT * FROM person WHERE current_mood = 'happy';
 name | current_mood
------+--------------
 Moe  | happy
(1 row)

8.7.2. Ordering

The ordering of the values in an enum type is the order in which the values were listed when the type was created. All standard comparison operators and related aggregate functions are supported for enums. For example:

INSERT INTO person VALUES ('Larry', 'sad');
INSERT INTO person VALUES ('Curly', 'ok');
SELECT * FROM person WHERE current_mood > 'sad';
 name  | current_mood
-------+--------------
 Moe   | happy
 Curly | ok
(2 rows)

SELECT * FROM person WHERE current_mood > 'sad' ORDER BY current_mood;
 name  | current_mood
-------+--------------
 Curly | ok
 Moe   | happy
(2 rows)

SELECT name
FROM person
WHERE current_mood = (SELECT MIN(current_mood) FROM person);
 name
-------
 Larry
(1 row)

8.7.3. Type Safety

Each enumerated data type is separate and cannot be compared with other enumerated types. See this example:

CREATE TYPE happiness AS ENUM ('happy', 'very happy', 'ecstatic');
CREATE TABLE holidays (
    num_weeks integer,
    happiness happiness
);
INSERT INTO holidays(num_weeks,happiness) VALUES (4, 'happy');
INSERT INTO holidays(num_weeks,happiness) VALUES (6, 'very happy');
INSERT INTO holidays(num_weeks,happiness) VALUES (8, 'ecstatic');
INSERT INTO holidays(num_weeks,happiness) VALUES (2, 'sad');
ERROR:  invalid input value for enum happiness: "sad"
SELECT person.name, holidays.num_weeks FROM person, holidays
  WHERE person.current_mood = holidays.happiness;
ERROR:  operator does not exist: mood = happiness

If you really need to do something like that, you can either write a custom operator or add explicit casts to your query:

SELECT person.name, holidays.num_weeks FROM person, holidays
  WHERE person.current_mood::text = holidays.happiness::text;
 name | num_weeks
------+-----------
 Moe  |         4
(1 row)

8.7.4. Implementation Details

Enum labels are case sensitive, so 'happy' is not the same as 'HAPPY'. White space in the labels is significant too.

Although enum types are primarily intended for static sets of values, there is support for adding new values to an existing enum type, and for renaming values (see ALTER TYPE). Existing values cannot be removed from an enum type, nor can the sort ordering of such values be changed, short of dropping and re-creating the enum type.

An enum value occupies four bytes on disk. The length of an enum value's textual label is limited by the NAMEDATALEN setting compiled into PostgreSQL; in standard builds this means at most 63 bytes.

The translations from internal enum values to textual labels are kept in the system catalog pg_enum. Querying this catalog directly can be useful.

Prev | Up | Next
8.6. Boolean Type | Home | 8.8. Geometric Types


8.8. Geometric Types
Prev Up Chapter 8. Data Types Home Next

8.8. Geometric Types

8.8.1. Points
8.8.2. Lines
8.8.3. Line Segments
8.8.4. Boxes
8.8.5. Paths
8.8.6. Polygons
8.8.7. Circles

Geometric data types represent two-dimensional spatial objects. Table 8.20 shows the geometric types available in PostgreSQL.

Table 8.20. Geometric Types

Name | Storage Size | Description | Representation
point | 16 bytes | Point on a plane | (x,y)
line | 32 bytes | Infinite line | {A,B,C}
lseg | 32 bytes | Finite line segment | ((x1,y1),(x2,y2))
box | 32 bytes | Rectangular box | ((x1,y1),(x2,y2))
path | 16+16n bytes | Closed path (similar to polygon) | ((x1,y1),...)
path | 16+16n bytes | Open path | [(x1,y1),...]
polygon | 40+16n bytes | Polygon (similar to closed path) | ((x1,y1),...)
circle | 24 bytes | Circle | <(x,y),r> (center point and radius)

A rich set of functions and operators is available to perform various geometric operations such as scaling, translation, rotation, and determining intersections. They are explained in Section 9.11.

8.8.1. Points

Points are the fundamental two-dimensional building block for geometric types. Values of type point are specified using either of the following syntaxes:

( x , y )
  x , y

where x and y are the respective coordinates, as floating-point numbers.

Points are output using the first syntax.

8.8.2. Lines

Lines are represented by the linear equation Ax + By + C = 0, where A and B are not both zero. Values of type line are input and output in the following form:

{ A, B, C }

Alternatively, any of the following forms can be used for input:

[ ( x1 , y1 ) , ( x2 , y2 ) ]
( ( x1 , y1 ) , ( x2 , y2 ) )
  ( x1 , y1 ) , ( x2 , y2 )
    x1 , y1   ,   x2 , y2

where (x1,y1) and (x2,y2) are two different points on the line.

8.8.3. Line Segments

Line segments are represented by pairs of points that are the endpoints of the segment. Values of type lseg are specified using any of the following syntaxes:

[ ( x1 , y1 ) , ( x2 , y2 ) ]
( ( x1 , y1 ) , ( x2 , y2 ) )
  ( x1 , y1 ) , ( x2 , y2 )
    x1 , y1   ,   x2 , y2

where (x1,y1) and (x2,y2) are the end points of the line segment.

Line segments are output using the first syntax.

8.8.4. Boxes

Boxes are represented by pairs of points that are opposite corners of the box. Values of type box are specified using any of the following syntaxes:

( ( x1 , y1 ) , ( x2 , y2 ) )
  ( x1 , y1 ) , ( x2 , y2 )
    x1 , y1   ,   x2 , y2

where (x1,y1) and (x2,y2) are any two opposite corners of the box.

Boxes are output using the second syntax.

Any two opposite corners can be supplied on input, but the values will be reordered as needed to store the upper right and lower left corners, in that order.

8.8.5. Paths

Paths are represented by lists of connected points. Paths can be open, where the first and last points in the list are considered not connected, or closed, where the first and last points are considered connected.

Values of type path are specified using any of the following syntaxes:

[ ( x1 , y1 ) , ... , ( xn , yn ) ]
( ( x1 , y1 ) , ... , ( xn , yn ) )
  ( x1 , y1 ) , ... , ( xn , yn )
  ( x1 , y1   , ... ,   xn , yn )
    x1 , y1   , ... ,   xn , yn

where the points are the end points of the line segments comprising the path. Square brackets ([]) indicate an open path, while parentheses (()) indicate a closed path. When the outermost parentheses are omitted, as in the third through fifth syntaxes, a closed path is assumed.

Paths are output using the first or second syntax, as appropriate.

8.8.6. Polygons

Polygons are represented by lists of points (the vertexes of the polygon). Polygons are very similar to closed paths, but are stored differently and have their own set of support routines.

Values of type polygon are specified using any of the following syntaxes:

( ( x1 , y1 ) , ... , ( xn , yn ) )
  ( x1 , y1 ) , ... , ( xn , yn )
  ( x1 , y1   , ... ,   xn , yn )
    x1 , y1   , ... ,   xn , yn

where the points are the end points of the line segments comprising the boundary of the polygon.

Polygons are output using the first syntax.

8.8.7. Circles

Circles are represented by a center point and radius. Values of type circle are specified using any of the following syntaxes:

< ( x , y ) , r >
( ( x , y ) , r )
  ( x , y ) , r
    x , y   , r

where (x,y) is the center point and r is the radius of the circle.

Circles are output using the first syntax.

Prev | Up | Next
8.7. Enumerated Types | Home | 8.9. Network Address Types


8.9. Network Address Types
Prev Up Chapter 8. Data Types Home Next

8.9. Network Address Types

8.9.1. inet
8.9.2. cidr
8.9.3. inet vs. cidr
8.9.4. macaddr
8.9.5. macaddr8

PostgreSQL offers data types to store IPv4, IPv6, and MAC addresses, as shown in Table 8.21. It is better to use these types instead of plain text types to store network addresses, because these types offer input error checking and specialized operators and functions (see Section 9.12).

Table 8.21. Network Address Types

Name | Storage Size | Description
cidr | 7 or 19 bytes | IPv4 and IPv6 networks
inet | 7 or 19 bytes | IPv4 and IPv6 hosts and networks
macaddr | 6 bytes | MAC addresses
macaddr8 | 8 bytes | MAC addresses (EUI-64 format)

When sorting inet or cidr data types, IPv4 addresses will always sort before IPv6 addresses, including IPv4 addresses encapsulated or mapped to IPv6 addresses, such as ::10.2.3.4 or ::ffff:10.4.3.2.

8.9.1. inet

The inet type holds an IPv4 or IPv6 host address, and optionally its subnet, all in one field. The subnet is represented by the number of network address bits present in the host address (the “netmask”). If the netmask is 32 and the address is IPv4, then the value does not indicate a subnet, only a single host. In IPv6, the address length is 128 bits, so 128 bits specify a unique host address. Note that if you want to accept only networks, you should use the cidr type rather than inet.

The input format for this type is address/y where address is an IPv4 or IPv6 address and y is the number of bits in the netmask. If the /y portion is omitted, the netmask is taken to be 32 for IPv4 or 128 for IPv6, so the value represents just a single host. On display, the /y portion is suppressed if the netmask specifies a single host.

8.9.2. cidr

The cidr type holds an IPv4 or IPv6 network specification. Input and output formats follow Classless Internet Domain Routing conventions. The format for specifying networks is address/y where address is the network's lowest address represented as an IPv4 or IPv6 address, and y is the number of bits in the netmask. If y is omitted, it is calculated using assumptions from the older classful network numbering system, except it will be at least large enough to include all of the octets written in the input. It is an error to specify a network address that has bits set to the right of the specified netmask.

Table 8.22 shows some examples.

Table 8.22. cidr Type Input Examples

cidr Input | cidr Output | abbrev(cidr)
192.168.100.128/25 | 192.168.100.128/25 | 192.168.100.128/25
192.168/24 | 192.168.0.0/24 | 192.168.0/24
192.168/25 | 192.168.0.0/25 | 192.168.0.0/25
192.168.1 | 192.168.1.0/24 | 192.168.1/24
192.168 | 192.168.0.0/24 | 192.168.0/24
128.1 | 128.1.0.0/16 | 128.1/16
128 | 128.0.0.0/16 | 128.0/16
128.1.2 | 128.1.2.0/24 | 128.1.2/24
10.1.2 | 10.1.2.0/24 | 10.1.2/24
10.1 | 10.1.0.0/16 | 10.1/16
10 | 10.0.0.0/8 | 10/8
10.1.2.3/32 | 10.1.2.3/32 | 10.1.2.3/32
2001:4f8:3:ba::/64 | 2001:4f8:3:ba::/64 | 2001:4f8:3:ba/64
2001:4f8:3:ba:​2e0:81ff:fe22:d1f1/128 | 2001:4f8:3:ba:​2e0:81ff:fe22:d1f1/128 | 2001:4f8:3:ba:​2e0:81ff:fe22:d1f1/128
::ffff:1.2.3.0/120 | ::ffff:1.2.3.0/120 | ::ffff:1.2.3/120
::ffff:1.2.3.0/128 | ::ffff:1.2.3.0/128 | ::ffff:1.2.3.0/128

8.9.3. inet vs. cidr

The essential difference between inet and cidr data types is that inet accepts values with nonzero bits to the right of the netmask, whereas cidr does not. For example, 192.168.0.1/24 is valid for inet but not for cidr.

Tip

If you do not like the output format for inet or cidr values, try the functions host, text, and abbrev.

8.9.4. macaddr

The macaddr type stores MAC addresses, known for example from Ethernet card hardware addresses (although MAC addresses are used for other purposes as well). Input is accepted in the following formats:

'08:00:2b:01:02:03'
'08-00-2b-01-02-03'
'08002b:010203'
'08002b-010203'
'0800.2b01.0203'
'0800-2b01-0203'
'08002b010203'

These examples all specify the same address. Upper and lower case is accepted for the digits a through f. Output is always in the first of the forms shown.

IEEE Standard 802-2001 specifies the second form shown (with hyphens) as the canonical form for MAC addresses, and specifies the first form (with colons) as used with bit-reversed, MSB-first notation, so that 08-00-2b-01-02-03 = 10:00:D4:80:40:C0. This convention is widely ignored nowadays, and it is relevant only for obsolete network protocols (such as Token Ring). PostgreSQL makes no provisions for bit reversal; all accepted formats use the canonical LSB order.

The remaining five input formats are not part of any standard.

8.9.5. macaddr8

The macaddr8 type stores MAC addresses in EUI-64 format, known for example from Ethernet card hardware addresses (although MAC addresses are used for other purposes as well). This type can accept both 6 and 8 byte length MAC addresses and stores them in 8 byte length format. MAC addresses given in 6 byte format will be stored in 8 byte length format with the 4th and 5th bytes set to FF and FE, respectively. Note that IPv6 uses a modified EUI-64 format where the 7th bit should be set to one after the conversion from EUI-48. The function macaddr8_set7bit is provided to make this change. Generally speaking, any input which is comprised of pairs of hex digits (on byte boundaries), optionally separated consistently by one of ':', '-' or '.', is accepted. The number of hex digits must be either 16 (8 bytes) or 12 (6 bytes). Leading and trailing whitespace is ignored. The following are examples of input formats that are accepted:

'08:00:2b:01:02:03:04:05'
'08-00-2b-01-02-03-04-05'
'08002b:0102030405'
'08002b-0102030405'
'0800.2b01.0203.0405'
'0800-2b01-0203-0405'
'08002b01:02030405'
'08002b0102030405'

These examples all specify the same address. Upper and lower case is accepted for the digits a through f. Output is always in the first of the forms shown.

The last six input formats shown above are not part of any standard.

To convert a traditional 48 bit MAC address in EUI-48 format to modified EUI-64 format to be included as the host portion of an IPv6 address, use macaddr8_set7bit as shown:

SELECT macaddr8_set7bit('08:00:2b:01:02:03');

    macaddr8_set7bit
-------------------------
 0a:00:2b:ff:fe:01:02:03
(1 row)

Prev | Up | Next
8.8. Geometric Types | Home | 8.10. Bit String Types


8.10. Bit String Types
Prev Up Chapter 8. Data Types Home Next

8.10. Bit String Types

Bit strings are strings of 1's and 0's. They can be used to store or visualize bit masks. There are two SQL bit types: bit(n) and bit varying(n), where n is a positive integer.

bit type data must match the length n exactly; it is an error to attempt to store shorter or longer bit strings. bit varying data is of variable length up to the maximum length n; longer strings will be rejected. Writing bit without a length is equivalent to bit(1), while bit varying without a length specification means unlimited length.

Note

If one explicitly casts a bit-string value to bit(n), it will be truncated or zero-padded on the right to be exactly n bits, without raising an error. Similarly, if one explicitly casts a bit-string value to bit varying(n), it will be truncated on the right if it is more than n bits.

Refer to Section 4.1.2.5 for information about the syntax of bit string constants. Bit-logical operators and string manipulation functions are available; see Section 9.6.

Example 8.3. Using the Bit String Types

CREATE TABLE test (a BIT(3), b BIT VARYING(5));
INSERT INTO test VALUES (B'101', B'00');
INSERT INTO test VALUES (B'10', B'101');

ERROR:  bit string length 2 does not match type bit(3)

INSERT INTO test VALUES (B'10'::bit(3), B'101');
SELECT * FROM test;

  a  |  b
-----+-----
 101 | 00
 100 | 101

A bit string value requires 1 byte for each group of 8 bits, plus 5 or 8 bytes overhead depending on the length of the string (but long values may be compressed or moved out-of-line, as explained in Section 8.3 for character strings).

Prev | Up | Next
8.9. Network Address Types | Home | 8.11. Text Search Types


8.11. Text Search Types
Prev Up Chapter 8. Data Types Home Next

8.11. Text Search Types

8.11.1. tsvector
8.11.2. tsquery

PostgreSQL provides two data types that are designed to support full text search, which is the activity of searching through a collection of natural-language documents to locate those that best match a query. The tsvector type represents a document in a form optimized for text search; the tsquery type similarly represents a text query. Chapter 12 provides a detailed explanation of this facility, and Section 9.13 summarizes the related functions and operators.

8.11.1. tsvector

A tsvector value is a sorted list of distinct lexemes, which are words that have been normalized to merge different variants of the same word (see Chapter 12 for details). Sorting and duplicate-elimination are done automatically during input, as shown in this example:

SELECT 'a fat cat sat on a mat and ate a fat rat'::tsvector;
                      tsvector
----------------------------------------------------
 'a' 'and' 'ate' 'cat' 'fat' 'mat' 'on' 'rat' 'sat'

To represent lexemes containing whitespace or punctuation, surround them with quotes:

SELECT $$the lexeme '    ' contains spaces$$::tsvector;
                 tsvector
-------------------------------------------
 '    ' 'contains' 'lexeme' 'spaces' 'the'

(We use dollar-quoted string literals in this example and the next one to avoid the confusion of having to double quote marks within the literals.) Embedded quotes and backslashes must be doubled:

SELECT $$the lexeme 'Joe''s' contains a quote$$::tsvector;
                    tsvector
------------------------------------------------
 'Joe''s' 'a' 'contains' 'lexeme' 'quote' 'the'

Optionally, integer positions can be attached to lexemes:

SELECT 'a:1 fat:2 cat:3 sat:4 on:5 a:6 mat:7 and:8 ate:9 a:10 fat:11 rat:12'::tsvector;
                                  tsvector
-------------------------------------------------------------------​------------
 'a':1,6,10 'and':8 'ate':9 'cat':3 'fat':2,11 'mat':7 'on':5 'rat':12 'sat':4

A position normally indicates the source word's location in the document. Positional information can be used for proximity ranking. Position values can range from 1 to 16383; larger numbers are silently set to 16383. Duplicate positions for the same lexeme are discarded.

Lexemes that have positions can further be labeled with a weight, which can be A, B, C, or D. D is the default and hence is not shown on output:

SELECT 'a:1A fat:2B,4C cat:5D'::tsvector;
          tsvector
----------------------------
 'a':1A 'cat':5 'fat':2B,4C

Weights are typically used to reflect document structure, for example by marking title words differently from body words. Text search ranking functions can assign different priorities to the different weight markers.

It is important to understand that the tsvector type itself does not perform any word normalization; it assumes the words it is given are normalized appropriately for the application. For example,

SELECT 'The Fat Rats'::tsvector;
      tsvector
--------------------
 'Fat' 'Rats' 'The'

For most English-text-searching applications the above words would be considered non-normalized, but tsvector doesn't care. Raw document text should usually be passed through to_tsvector to normalize the words appropriately for searching:

SELECT to_tsvector('english', 'The Fat Rats');
   to_tsvector
-----------------
 'fat':2 'rat':3

Again, see Chapter 12 for more detail.

8.11.2. tsquery

A tsquery value stores lexemes that are to be searched for, and can combine them using the Boolean operators & (AND), | (OR), and ! (NOT), as well as the phrase search operator <-> (FOLLOWED BY). There is also a variant <N> of the FOLLOWED BY operator, where N is an integer constant that specifies the distance between the two lexemes being searched for. <-> is equivalent to <1>.

Parentheses can be used to enforce grouping of these operators. In the absence of parentheses, ! (NOT) binds most tightly, <-> (FOLLOWED BY) next most tightly, then & (AND), with | (OR) binding the least tightly.

Here are some examples:

SELECT 'fat & rat'::tsquery;
    tsquery
---------------
 'fat' & 'rat'

SELECT 'fat & (rat | cat)'::tsquery;
          tsquery
---------------------------
 'fat' & ( 'rat' | 'cat' )

SELECT 'fat & rat & ! cat'::tsquery;
        tsquery
------------------------
 'fat' & 'rat' & !'cat'

Optionally, lexemes in a tsquery can be labeled with one or more weight letters, which restricts them to match only tsvector lexemes with one of those weights:

SELECT 'fat:ab & cat'::tsquery;
    tsquery
------------------
 'fat':AB & 'cat'

Also, lexemes in a tsquery can be labeled with * to specify prefix matching:

SELECT 'super:*'::tsquery;
  tsquery
-----------
 'super':*

This query will match any word in a tsvector that begins with “super”.

Quoting rules for lexemes are the same as described previously for lexemes in tsvector; and, as with tsvector, any required normalization of words must be done before converting to the tsquery type. The to_tsquery function is convenient for performing such normalization:

SELECT to_tsquery('Fat:ab & Cats');
    to_tsquery
------------------
 'fat':AB & 'cat'

Note that to_tsquery will process prefixes in the same way as other words, which means this comparison returns true:

SELECT to_tsvector( 'postgraduate' ) @@ to_tsquery( 'postgres:*' );
 ?column?
----------
 t

because postgres gets stemmed to postgr:

SELECT to_tsvector( 'postgraduate' ), to_tsquery( 'postgres:*' );
  to_tsvector  | to_tsquery
---------------+------------
 'postgradu':1 | 'postgr':*

which will match the stemmed form of postgraduate.

Prev | Up | Next
8.10. Bit String Types | Home | 8.12. UUID Type


8.12. UUID Type
Prev Up Chapter 8. Data Types Home Next

8.12. UUID Type

The data type uuid stores Universally Unique Identifiers (UUID) as defined by RFC 4122, ISO/IEC 9834-8:2005, and related standards. (Some systems refer to this data type as a globally unique identifier, or GUID, instead.) This identifier is a 128-bit quantity that is generated by an algorithm chosen to make it very unlikely that the same identifier will be generated by anyone else in the known universe using the same algorithm. Therefore, for distributed systems, these identifiers provide a better uniqueness guarantee than sequence generators, which are only unique within a single database.

A UUID is written as a sequence of lower-case hexadecimal digits, in several groups separated by hyphens, specifically a group of 8 digits followed by three groups of 4 digits followed by a group of 12 digits, for a total of 32 digits representing the 128 bits. An example of a UUID in this standard form is:

a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11

PostgreSQL also accepts the following alternative forms for input: use of upper-case digits, the standard format surrounded by braces, omitting some or all hyphens, adding a hyphen after any group of four digits. Examples are:

A0EEBC99-9C0B-4EF8-BB6D-6BB9BD380A11
{a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11}
a0eebc999c0b4ef8bb6d6bb9bd380a11
a0ee-bc99-9c0b-4ef8-bb6d-6bb9-bd38-0a11
{a0eebc99-9c0b4ef8-bb6d6bb9-bd380a11}

Output is always in the standard form.

See Section 9.14 for how to generate a UUID in PostgreSQL.

Prev | Up | Next
8.11. Text Search Types | Home | 8.13. XML Type


8.13. XML Type
Prev Up Chapter 8. Data Types Home Next

8.13. XML Type

8.13.1. Creating XML Values
8.13.2. Encoding Handling
8.13.3. Accessing XML Values

The xml data type can be used to store XML data. Its advantage over storing XML data in a text field is that it checks the input values for well-formedness, and there are support functions to perform type-safe operations on it; see Section 9.15. Use of this data type requires the installation to have been built with configure --with-libxml.

The xml type can store well-formed “documents”, as defined by the XML standard, as well as “content” fragments, which are defined by reference to the more permissive “document node” of the XQuery and XPath data model. Roughly, this means that content fragments can have more than one top-level element or character node. The expression xmlvalue IS DOCUMENT can be used to evaluate whether a particular xml value is a full document or only a content fragment.

Limits and compatibility notes for the xml data type can be found in Section D.3.

8.13.1. Creating XML Values

To produce a value of type xml from character data, use the function xmlparse:

XMLPARSE ( { DOCUMENT | CONTENT } value)

Examples:

XMLPARSE (DOCUMENT '<?xml version="1.0"?><book><title>Manual</title><chapter>...</chapter></book>')
XMLPARSE (CONTENT 'abc<foo>bar</foo><bar>foo</bar>')

While this is the only way to convert character strings into XML values according to the SQL standard, the PostgreSQL-specific syntaxes:

xml '<foo>bar</foo>'
'<foo>bar</foo>'::xml

can also be used.

The xml type does not validate input values against a document type declaration (DTD), even when the input value specifies a DTD. There is also currently no built-in support for validating against other XML schema languages such as XML Schema.

The inverse operation, producing a character string value from xml, uses the function xmlserialize:

XMLSERIALIZE ( { DOCUMENT | CONTENT } value AS type )

type can be character, character varying, or text (or an alias for one of those). Again, according to the SQL standard, this is the only way to convert between type xml and character types, but PostgreSQL also allows you to simply cast the value.

When a character string value is cast to or from type xml without going through XMLPARSE or XMLSERIALIZE, respectively, the choice of DOCUMENT versus CONTENT is determined by the “XML option” session configuration parameter, which can be set using the standard command:

SET XML OPTION { DOCUMENT | CONTENT };

or the more PostgreSQL-like syntax

SET xmloption TO { DOCUMENT | CONTENT };

The default is CONTENT, so all forms of XML data are allowed.

8.13.2. Encoding Handling

Care must be taken when dealing with multiple character encodings on the client, server, and in the XML data passed through them. When using the text mode to pass queries to the server and query results to the client (which is the normal mode), PostgreSQL converts all character data passed between the client and the server and vice versa to the character encoding of the respective end; see Section 24.3. This includes string representations of XML values, such as in the above examples. This would ordinarily mean that encoding declarations contained in XML data can become invalid as the character data is converted to other encodings while traveling between client and server, because the embedded encoding declaration is not changed. To cope with this behavior, encoding declarations contained in character strings presented for input to the xml type are ignored, and content is assumed to be in the current server encoding. Consequently, for correct processing, character strings of XML data must be sent from the client in the current client encoding. It is the responsibility of the client to either convert documents to the current client encoding before sending them to the server, or to adjust the client encoding appropriately. On output, values of type xml will not have an encoding declaration, and clients should assume all data is in the current client encoding.

When using binary mode to pass query parameters to the server and query results back to the client, no encoding conversion is performed, so the situation is different. In this case, an encoding declaration in the XML data will be observed, and if it is absent, the data will be assumed to be in UTF-8 (as required by the XML standard; note that PostgreSQL does not support UTF-16). On output, data will have an encoding declaration specifying the client encoding, unless the client encoding is UTF-8, in which case it will be omitted.

Needless to say, processing XML data with PostgreSQL will be less error-prone and more efficient if the XML data encoding, client encoding, and server encoding are the same. Since XML data is internally processed in UTF-8, computations will be most efficient if the server encoding is also UTF-8.

Caution

Some XML-related functions may not work at all on non-ASCII data when the server encoding is not UTF-8. This is known to be an issue for xmltable() and xpath() in particular.

8.13.3. Accessing XML Values

The xml data type is unusual in that it does not provide any comparison operators. This is because there is no well-defined and universally useful comparison algorithm for XML data. One consequence of this is that you cannot retrieve rows by comparing an xml column against a search value. XML values should therefore typically be accompanied by a separate key field such as an ID. An alternative solution for comparing XML values is to convert them to character strings first, but note that character string comparison has little to do with a useful XML comparison method.

Since there are no comparison operators for the xml data type, it is not possible to create an index directly on a column of this type. If speedy searches in XML data are desired, possible workarounds include casting the expression to a character string type and indexing that, or indexing an XPath expression. Of course, the actual query would have to be adjusted to search by the indexed expression.

The text-search functionality in PostgreSQL can also be used to speed up full-document searches of XML data. The necessary preprocessing support is, however, not yet available in the PostgreSQL distribution.

Prev | Up | Next
8.12. UUID Type | Home | 8.14. JSON Types


8.14. JSON Types
Prev Up Chapter 8. Data Types Home Next

8.14. JSON Types

8.14.1. JSON Input and Output Syntax
8.14.2. Designing JSON Documents
8.14.3. jsonb Containment and Existence
8.14.4. jsonb Indexing
8.14.5. jsonb Subscripting
8.14.6. Transforms
8.14.7. jsonpath Type

JSON data types are for storing JSON (JavaScript Object Notation) data, as specified in RFC 7159. Such data can also be stored as text, but the JSON data types have the advantage of enforcing that each stored value is valid according to the JSON rules. There are also assorted JSON-specific functions and operators available for data stored in these data types; see Section 9.16.

PostgreSQL offers two types for storing JSON data: json and jsonb. To implement efficient query mechanisms for these data types, PostgreSQL also provides the jsonpath data type described in Section 8.14.7.

The json and jsonb data types accept almost identical sets of values as input. The major practical difference is one of efficiency. The json data type stores an exact copy of the input text, which processing functions must reparse on each execution; while jsonb data is stored in a decomposed binary format that makes it slightly slower to input due to added conversion overhead, but significantly faster to process, since no reparsing is needed. jsonb also supports indexing, which can be a significant advantage.

Because the json type stores an exact copy of the input text, it will preserve semantically-insignificant white space between tokens, as well as the order of keys within JSON objects. Also, if a JSON object within the value contains the same key more than once, all the key/value pairs are kept. (The processing functions consider the last value as the operative one.) By contrast, jsonb does not preserve white space, does not preserve the order of object keys, and does not keep duplicate object keys. If duplicate keys are specified in the input, only the last value is kept.

In general, most applications should prefer to store JSON data as jsonb, unless there are quite specialized needs, such as legacy assumptions about ordering of object keys.

RFC 7159 specifies that JSON strings should be encoded in UTF8. It is therefore not possible for the JSON types to conform rigidly to the JSON specification unless the database encoding is UTF8. Attempts to directly include characters that cannot be represented in the database encoding will fail; conversely, characters that can be represented in the database encoding but not in UTF8 will be allowed.

RFC 7159 permits JSON strings to contain Unicode escape sequences denoted by \uXXXX. In the input function for the json type, Unicode escapes are allowed regardless of the database encoding, and are checked only for syntactic correctness (that is, that four hex digits follow \u). However, the input function for jsonb is stricter: it disallows Unicode escapes for characters that cannot be represented in the database encoding. The jsonb type also rejects \u0000 (because that cannot be represented in PostgreSQL's text type), and it insists that any use of Unicode surrogate pairs to designate characters outside the Unicode Basic Multilingual Plane be correct. Valid Unicode escapes are converted to the equivalent single character for storage; this includes folding surrogate pairs into a single character.

Note

Many of the JSON processing functions described in Section 9.16 will convert Unicode escapes to regular characters, and will therefore throw the same types of errors just described even if their input is of type json not jsonb. The fact that the json input function does not make these checks may be considered a historical artifact, although it does allow for simple storage (without processing) of JSON Unicode escapes in a database encoding that does not support the represented characters.

When converting textual JSON input into jsonb, the primitive types described by RFC 7159 are effectively mapped onto native PostgreSQL types, as shown in Table 8.23. Therefore, there are some minor additional constraints on what constitutes valid jsonb data that do not apply to the json type, nor to JSON in the abstract, corresponding to limits on what can be represented by the underlying data type. Notably, jsonb will reject numbers that are outside the range of the PostgreSQL numeric data type, while json will not. Such implementation-defined restrictions are permitted by RFC 7159. However, in practice such problems are far more likely to occur in other implementations, as it is common to represent JSON's number primitive type as IEEE 754 double precision floating point (which RFC 7159 explicitly anticipates and allows for). When using JSON as an interchange format with such systems, the danger of losing numeric precision compared to data originally stored by PostgreSQL should be considered.

Conversely, as noted in the table there are some minor restrictions on the input format of JSON primitive types that do not apply to the corresponding PostgreSQL types.

Table 8.23. JSON Primitive Types and Corresponding PostgreSQL Types

JSON primitive type | PostgreSQL type | Notes
string | text | \u0000 is disallowed, as are Unicode escapes representing characters not available in the database encoding
number | numeric | NaN and infinity values are disallowed
boolean | boolean | Only lowercase true and false spellings are accepted
null | (none) | SQL NULL is a different concept

8.14.1. JSON Input and Output Syntax

The input/output syntax for the JSON data types is as specified in RFC 7159.

The following are all valid json (or jsonb) expressions:

-- Simple scalar/primitive value
-- Primitive values can be numbers, quoted strings, true, false, or null
SELECT '5'::json;

-- Array of zero or more elements (elements need not be of same type)
SELECT '[1, 2, "foo", null]'::json;

-- Object containing pairs of keys and values
-- Note that object keys must always be quoted strings
SELECT '{"bar": "baz", "balance": 7.77, "active": false}'::json;

-- Arrays and objects can be nested arbitrarily
SELECT '{"foo": [true, "bar"], "tags": {"a": 1, "b": null}}'::json;

As previously stated, when a JSON value is input and then printed without any additional processing, json outputs the same text that was input, while jsonb does not preserve semantically-insignificant details such as whitespace. For example, note the differences here:

SELECT '{"bar": "baz", "balance": 7.77, "active":false}'::json;
                      json
-------------------------------------------------
 {"bar": "baz", "balance": 7.77, "active":false}
(1 row)

SELECT '{"bar": "baz", "balance": 7.77, "active":false}'::jsonb;
                      jsonb
--------------------------------------------------
 {"bar": "baz", "active": false, "balance": 7.77}
(1 row)

One semantically-insignificant detail worth noting is that in jsonb, numbers will be printed according to the behavior of the underlying numeric type. In practice this means that numbers entered with E notation will be printed without it, for example:

SELECT '{"reading": 1.230e-5}'::json, '{"reading": 1.230e-5}'::jsonb;
         json          |          jsonb
-----------------------+-------------------------
 {"reading": 1.230e-5} | {"reading": 0.00001230}
(1 row)

However, jsonb will preserve trailing fractional zeroes, as seen in this example, even though those are semantically insignificant for purposes such as equality checks.

For the list of built-in functions and operators available for constructing and processing JSON values, see Section 9.16.

8.14.2. Designing JSON Documents

Representing data as JSON can be considerably more flexible than the traditional relational data model, which is compelling in environments where requirements are fluid. It is quite possible for both approaches to co-exist and complement each other within the same application. However, even for applications where maximal flexibility is desired, it is still recommended that JSON documents have a somewhat fixed structure. The structure is typically unenforced (though enforcing some business rules declaratively is possible), but having a predictable structure makes it easier to write queries that usefully summarize a set of “documents” (datums) in a table.

JSON data is subject to the same concurrency-control considerations as any other data type when stored in a table. Although storing large documents is practicable, keep in mind that any update acquires a row-level lock on the whole row. Consider limiting JSON documents to a manageable size in order to decrease lock contention among updating transactions. Ideally, JSON documents should each represent an atomic datum that business rules dictate cannot reasonably be further subdivided into smaller datums that could be modified independently.

8.14.3. jsonb Containment and Existence

Testing containment is an important capability of jsonb. There is no parallel set of facilities for the json type. Containment tests whether one jsonb document has contained within it another one. These examples return true except as noted:

-- Simple scalar/primitive values contain only the identical value:
SELECT '"foo"'::jsonb @> '"foo"'::jsonb;

-- The array on the right side is contained within the one on the left:
SELECT '[1, 2, 3]'::jsonb @> '[1, 3]'::jsonb;

-- Order of array elements is not significant, so this is also true:
SELECT '[1, 2, 3]'::jsonb @> '[3, 1]'::jsonb;

-- Duplicate array elements don't matter either:
SELECT '[1, 2, 3]'::jsonb @> '[1, 2, 2]'::jsonb;

-- The object with a single pair on the right side is contained
-- within the object on the left side:
SELECT '{"product": "PostgreSQL", "version": 9.4, "jsonb": true}'::jsonb @> '{"version": 9.4}'::jsonb;

-- The array on the right side is not considered contained within the
-- array on the left, even though a similar array is nested within it:
SELECT '[1, 2, [1, 3]]'::jsonb @> '[1, 3]'::jsonb;  -- yields false

-- But with a layer of nesting, it is contained:
SELECT '[1, 2, [1, 3]]'::jsonb @> '[[1, 3]]'::jsonb;

-- Similarly, containment is not reported here:
SELECT '{"foo": {"bar": "baz"}}'::jsonb @> '{"bar": "baz"}'::jsonb;  -- yields false

-- A top-level key and an empty object is contained:
SELECT '{"foo": {"bar": "baz"}}'::jsonb @> '{"foo": {}}'::jsonb;

The general principle is that the contained object must match the containing object as to structure and data contents, possibly after discarding some non-matching array elements or object key/value pairs from the containing object. But remember that the order of array elements is not significant when doing a containment match, and duplicate array elements are effectively considered only once.

As a special exception to the general principle that the structures must match, an array may contain a primitive value:

-- This array contains the primitive string value:
SELECT '["foo", "bar"]'::jsonb @> '"bar"'::jsonb;

-- This exception is not reciprocal -- non-containment is reported here:
SELECT '"bar"'::jsonb @> '["bar"]'::jsonb;  -- yields false

jsonb also has an existence operator, which is a variation on the theme of containment: it tests whether a string (given as a text value) appears as an object key or array element at the top level of the jsonb value. These examples return true except as noted:

-- String exists as array element:
SELECT '["foo", "bar", "baz"]'::jsonb ? 'bar';

-- String exists as object key:
SELECT '{"foo": "bar"}'::jsonb ? 'foo';

-- Object values are not considered:
SELECT '{"foo": "bar"}'::jsonb ? 'bar';  -- yields false

-- As with containment, existence must match at the top level:
SELECT '{"foo": {"bar": "baz"}}'::jsonb ? 'bar'; -- yields false

-- A string is considered to exist if it matches a primitive JSON string:
SELECT '"foo"'::jsonb ? 'foo';

JSON objects are better suited than arrays for testing containment or existence when there are many keys or elements involved, because unlike arrays they are internally optimized for searching, and do not need to be searched linearly.

Tip

Because JSON containment is nested, an appropriate query can skip explicit selection of sub-objects. As an example, suppose that we have a doc column containing objects at the top level, with most objects containing tags fields that contain arrays of sub-objects. This query finds entries in which sub-objects containing both "term":"paris" and "term":"food" appear, while ignoring any such keys outside the tags array:

SELECT doc->'site_name' FROM websites
  WHERE doc @> '{"tags":[{"term":"paris"}, {"term":"food"}]}';

One could accomplish the same thing with, say,

SELECT doc->'site_name' FROM websites
  WHERE doc->'tags' @> '[{"term":"paris"}, {"term":"food"}]';

but that approach is less flexible, and often less efficient as well.

On the other hand, the JSON existence operator is not nested: it will only look for the specified key or array element at top level of the JSON value.

The various containment and existence operators, along with all other JSON operators and functions are documented in Section 9.16.

8.14.4. jsonb Indexing

GIN indexes can be used to efficiently search for keys or key/value pairs occurring within a large number of jsonb documents (datums). Two GIN “operator classes” are provided, offering different performance and flexibility trade-offs.

The default GIN operator class for jsonb supports queries with the key-exists operators ?, ?| and ?&, the containment operator @>, and the jsonpath match operators @? and @@. (For details of the semantics that these operators implement, see Table 9.45.) An example of creating an index with this operator class is:

CREATE INDEX idxgin ON api USING GIN (jdoc);

The non-default GIN operator class jsonb_path_ops does not support the key-exists operators, but it does support @>, @? and @@. An example of creating an index with this operator class is:

CREATE INDEX idxginp ON api USING GIN (jdoc jsonb_path_ops);

Consider the example of a table that stores JSON documents retrieved from a third-party web service, with a documented schema definition. A typical document is:

{
    "guid": "9c36adc1-7fb5-4d5b-83b4-90356a46061a",
    "name": "Angela Barton",
    "is_active": true,
    "company": "Magnafone",
    "address": "178 Howard Place, Gulf, Washington, 702",
    "registered": "2009-11-07T08:53:22 +08:00",
    "latitude": 19.793713,
    "longitude": 86.513373,
    "tags": [
        "enim",
        "aliquip",
        "qui"
    ]
}

We store these documents in a table named api, in a jsonb column named jdoc. If a GIN index is created on this column, queries like the following can make use of the index:

-- Find documents in which the key "company" has value "Magnafone"
SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"company": "Magnafone"}';

However, the index could not be used for queries like the following, because though the operator ? is indexable, it is not applied directly to the indexed column jdoc:

-- Find documents in which the key "tags" contains key or array element "qui"
SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc -> 'tags' ? 'qui';

Still, with appropriate use of expression indexes, the above query can use an index. If querying for particular items within the "tags" key is common, defining an index like this may be worthwhile:

CREATE INDEX idxgintags ON api USING GIN ((jdoc -> 'tags'));

Now, the WHERE clause jdoc -> 'tags' ? 'qui' will be recognized as an application of the indexable operator ? to the indexed expression jdoc -> 'tags'. (More information on expression indexes can be found in Section 11.7.)

Another approach to querying is to exploit containment, for example:

-- Find documents in which the key "tags" contains array element "qui"
SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qui"]}';

A simple GIN index on the jdoc column can support this query. But note that such an index will store copies of every key and value in the jdoc column, whereas the expression index of the previous example stores only data found under the tags key. While the simple-index approach is far more flexible (since it supports queries about any key), targeted expression indexes are likely to be smaller and faster to search than a simple index.

GIN indexes also support the @? and @@ operators, which perform jsonpath matching. Examples are

SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @? '$.tags[*] ? (@ == "qui")';

SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @@ '$.tags[*] == "qui"';

For these operators, a GIN index extracts clauses of the form accessors_chain = constant out of the jsonpath pattern, and does the index search based on the keys and values mentioned in these clauses. The accessors chain may include .key, [*], and [index] accessors. The jsonb_ops operator class also supports .* and .** accessors, but the jsonb_path_ops operator class does not.

Although the jsonb_path_ops operator class supports only queries with the @>, @? and @@ operators, it has notable performance advantages over the default operator class jsonb_ops. A jsonb_path_ops index is usually much smaller than a jsonb_ops index over the same data, and the specificity of searches is better, particularly when queries contain keys that appear frequently in the data. Therefore search operations typically perform better than with the default operator class.

The technical difference between a jsonb_ops and a jsonb_path_ops GIN index is that the former creates independent index items for each key and value in the data, while the latter creates index items only for each value in the data. [7] Basically, each jsonb_path_ops index item is a hash of the value and the key(s) leading to it; for example to index {"foo": {"bar": "baz"}}, a single index item would be created incorporating all three of foo, bar, and baz into the hash value. Thus a containment query looking for this structure would result in an extremely specific index search; but there is no way at all to find out whether foo appears as a key. On the other hand, a jsonb_ops index would create three index items representing foo, bar, and baz separately; then to do the containment query, it would look for rows containing all three of these items. While GIN indexes can perform such an AND search fairly efficiently, it will still be less specific and slower than the equivalent jsonb_path_ops search, especially if there are a very large number of rows containing any single one of the three index items.

A disadvantage of the jsonb_path_ops approach is that it produces no index entries for JSON structures not containing any values, such as {"a": {}}. If a search for documents containing such a structure is requested, it will require a full-index scan, which is quite slow. jsonb_path_ops is therefore ill-suited for applications that often perform such searches.

jsonb also supports btree and hash indexes. These are usually useful only if it's important to check equality of complete JSON documents. The btree ordering for jsonb datums is seldom of great interest, but for completeness it is:

Object > Array > Boolean > Number > String > Null

Object with n pairs > object with n - 1 pairs

Array with n elements > array with n - 1 elements

Objects with equal numbers of pairs are compared in the order:

key-1, value-1, key-2 ...

Note that object keys are compared in their storage order; in particular, since shorter keys are stored before longer keys, this can lead to results that might be unintuitive, such as:

{ "aa": 1, "c": 1} > {"b": 1, "d": 1}

Similarly, arrays with equal numbers of elements are compared in the order:

element-1, element-2 ...

Primitive JSON values are compared using the same comparison rules as for the underlying PostgreSQL data type. Strings are compared using the default database collation.

8.14.5. jsonb Subscripting

The jsonb data type supports array-style subscripting expressions to extract and modify elements. Nested values can be indicated by chaining subscripting expressions, following the same rules as the path argument in the jsonb_set function. If a jsonb value is an array, numeric subscripts start at zero, and negative integers count backwards from the last element of the array. Slice expressions are not supported. The result of a subscripting expression is always of the jsonb data type.

UPDATE statements may use subscripting in the SET clause to modify jsonb values. Subscript paths must be traversable for all affected values insofar as they exist. For instance, the path val['a']['b']['c'] can be traversed all the way to c if every val, val['a'], and val['a']['b'] is an object. If any val['a'] or val['a']['b'] is not defined, it will be created as an empty object and filled as necessary. However, if any val itself or one of the intermediary values is defined as a non-object such as a string, number, or jsonb null, traversal cannot proceed so an error is raised and the transaction aborted.

An example of subscripting syntax:

-- Extract object value by key
SELECT ('{"a": 1}'::jsonb)['a'];

-- Extract nested object value by key path
SELECT ('{"a": {"b": {"c": 1}}}'::jsonb)['a']['b']['c'];

-- Extract array element by index
SELECT ('[1, "2", null]'::jsonb)[1];

-- Update object value by key. Note the quotes around '1': the assigned
-- value must be of the jsonb type as well
UPDATE table_name SET jsonb_field['key'] = '1';

-- This will raise an error if any record's jsonb_field['a']['b'] is something
-- other than an object. For example, the value {"a": 1} has a numeric value
-- of the key 'a'.
UPDATE table_name SET jsonb_field['a']['b']['c'] = '1';

-- Filter records using a WHERE clause with subscripting. Since the result of
-- subscripting is jsonb, the value we compare it against must also be jsonb.
-- The double quotes make "value" also a valid jsonb string.
SELECT * FROM table_name WHERE jsonb_field['key'] = '"value"';

jsonb assignment via subscripting handles a few edge cases differently from jsonb_set. When a source jsonb value is NULL, assignment via subscripting will proceed as if it was an empty JSON value of the type (object or array) implied by the subscript key:

-- Where jsonb_field was NULL, it is now {"a": 1}
UPDATE table_name SET jsonb_field['a'] = '1';

-- Where jsonb_field was NULL, it is now [1]
UPDATE table_name SET jsonb_field[0] = '1';

If an index is specified for an array containing too few elements, NULL elements will be appended until the index is reachable and the value can be set.

-- Where jsonb_field was [], it is now [null, null, 2];
-- where jsonb_field was [0], it is now [0, null, 2]
UPDATE table_name SET jsonb_field[2] = '2';

A jsonb value will accept assignments to nonexistent subscript paths as long as the last existing element to be traversed is an object or array, as implied by the corresponding subscript (the element indicated by the last subscript in the path is not traversed and may be anything). Nested array and object structures will be created, and in the former case null-padded, as specified by the subscript path until the assigned value can be placed.

-- Where jsonb_field was {}, it is now {'a': [{'b': 1}]}
UPDATE table_name SET jsonb_field['a'][0]['b'] = '1';

-- Where jsonb_field was [], it is now [null, {'a': 1}]
UPDATE table_name SET jsonb_field[1]['a'] = '1';

8.14.6. Transforms

Additional extensions are available that implement transforms for the jsonb type for different procedural languages.

The extensions for PL/Perl are called jsonb_plperl and jsonb_plperlu. If you use them, jsonb values are mapped to Perl arrays, hashes, and scalars, as appropriate.

The extensions for PL/Python are called jsonb_plpythonu, jsonb_plpython2u, and jsonb_plpython3u (see Section 46.1 for the PL/Python naming convention). If you use them, jsonb values are mapped to Python dictionaries, lists, and scalars, as appropriate.

Of these extensions, jsonb_plperl is considered “trusted”, that is, it can be installed by non-superusers who have CREATE privilege on the current database. The rest require superuser privilege to install.

8.14.7. jsonpath Type

The jsonpath type implements support for the SQL/JSON path language in PostgreSQL to efficiently query JSON data. It provides a binary representation of the parsed SQL/JSON path expression that specifies the items to be retrieved by the path engine from the JSON data for further processing with the SQL/JSON query functions.

The semantics of SQL/JSON path predicates and operators generally follow SQL. At the same time, to provide a natural way of working with JSON data, SQL/JSON path syntax uses some JavaScript conventions:

  • Dot (.) is used for member access.

  • Square brackets ([]) are used for array access.

  • SQL/JSON arrays are 0-relative, unlike regular SQL arrays that start from 1.

An SQL/JSON path expression is typically written in an SQL query as an SQL character string literal, so it must be enclosed in single quotes, and any single quotes desired within the value must be doubled (see Section 4.1.2.1). Some forms of path expressions require string literals within them. These embedded string literals follow JavaScript/ECMAScript conventions: they must be surrounded by double quotes, and backslash escapes may be used within them to represent otherwise-hard-to-type characters. In particular, the way to write a double quote within an embedded string literal is \", and to write a backslash itself, you must write \\. Other special backslash sequences include those recognized in JSON strings: \b, \f, \n, \r, \t, \v for various ASCII control characters, and \uNNNN for a Unicode character identified by its 4-hex-digit code point. The backslash syntax also includes two cases not allowed by JSON: \xNN for a character code written with only two hex digits, and \u{N...} for a character code written with 1 to 6 hex digits.

A path expression consists of a sequence of path elements, which can be any of the following:

  • Path literals of JSON primitive types: Unicode text, numeric, true, false, or null.

  • Path variables listed in Table 8.24.

  • Accessor operators listed in Table 8.25.

  • jsonpath operators and methods listed in Section 9.16.2.2.

  • Parentheses, which can be used to provide filter expressions or define the order of path evaluation.

For details on using jsonpath expressions with SQL/JSON query functions, see Section 9.16.2.

Table 8.24. jsonpath Variables

Variable | Description
$ | A variable representing the JSON value being queried (the context item).
$varname | A named variable. Its value can be set by the parameter vars of several JSON processing functions; see Table 9.47 for details.
@ | A variable representing the result of path evaluation in filter expressions.

Table 8.25. jsonpath Accessors

Accessor Operator Description

.key

."$varname"

Member accessor that returns an object member with the specified key. If the key name matches some named variable starting with $ or does not meet the JavaScript rules for an identifier, it must be enclosed in double quotes to make it a string literal.

.*

Wildcard member accessor that returns the values of all members located at the top level of the current object.

.**

Recursive wildcard member accessor that processes all levels of the JSON hierarchy of the current object and returns all the member values, regardless of their nesting level. This is a PostgreSQL extension of the SQL/JSON standard.

.**{level}

.**{start_level to end_level}

Like .**, but selects only the specified levels of the JSON hierarchy. Nesting levels are specified as integers. Level zero corresponds to the current object. To access the lowest nesting level, you can use the last keyword. This is a PostgreSQL extension of the SQL/JSON standard.

[subscript, ...]

Array element accessor. subscript can be given in two forms: index or start_index to end_index. The first form returns a single array element by its index. The second form returns an array slice by the range of indexes, including the elements that correspond to the provided start_index and end_index.

The specified index can be an integer, as well as an expression returning a single numeric value, which is automatically cast to integer. Index zero corresponds to the first array element. You can also use the last keyword to denote the last array element, which is useful for handling arrays of unknown length.

[*]

Wildcard array element accessor that returns all array elements.

[7] For this purpose, the term “value” includes array elements, though JSON terminology sometimes considers array elements distinct from values within objects.

Prev | Up | Next
8.13. XML Type | Home | 8.15. Arrays


8.15. Arrays
Prev Up Chapter 8. Data Types Home Next

8.15. Arrays

8.15.1. Declaration of Array Types
8.15.2. Array Value Input
8.15.3. Accessing Arrays
8.15.4. Modifying Arrays
8.15.5. Searching in Arrays
8.15.6. Array Input and Output Syntax

PostgreSQL allows columns of a table to be defined as variable-length multidimensional arrays. Arrays of any built-in or user-defined base type, enum type, composite type, range type, or domain can be created.

8.15.1. Declaration of Array Types

To illustrate the use of array types, we create this table:

CREATE TABLE sal_emp (
    name            text,
    pay_by_quarter  integer[],
    schedule        text[][]
);

As shown, an array data type is named by appending square brackets ([]) to the data type name of the array elements. The above command will create a table named sal_emp with a column of type text (name), a one-dimensional array of type integer (pay_by_quarter), which represents the employee's salary by quarter, and a two-dimensional array of text (schedule), which represents the employee's weekly schedule.

The syntax for CREATE TABLE allows the exact size of arrays to be specified, for example:

CREATE TABLE tictactoe (
    squares   integer[3][3]
);

However, the current implementation ignores any supplied array size limits, i.e., the behavior is the same as for arrays of unspecified length.

The current implementation does not enforce the declared number of dimensions either. Arrays of a particular element type are all considered to be of the same type, regardless of size or number of dimensions. So, declaring the array size or number of dimensions in CREATE TABLE is simply documentation; it does not affect run-time behavior.

An alternative syntax, which conforms to the SQL standard by using the keyword ARRAY, can be used for one-dimensional arrays. pay_by_quarter could have been defined as:

    pay_by_quarter  integer ARRAY[4],

Or, if no array size is to be specified:

    pay_by_quarter  integer ARRAY,

As before, however, PostgreSQL does not enforce the size restriction in any case.

8.15.2. Array Value Input

To write an array value as a literal constant, enclose the element values within curly braces and separate them by commas. (If you know C, this is not unlike the C syntax for initializing structures.) You can put double quotes around any element value, and must do so if it contains commas or curly braces. (More details appear below.) Thus, the general format of an array constant is the following:

'{ val1 delim val2 delim ... }'

where delim is the delimiter character for the type, as recorded in its pg_type entry. Among the standard data types provided in the PostgreSQL distribution, all use a comma (,), except for type box which uses a semicolon (;). Each val is either a constant of the array element type, or a subarray. An example of an array constant is:

'{{1,2,3},{4,5,6},{7,8,9}}'

This constant is a two-dimensional, 3-by-3 array consisting of three subarrays of integers.

To set an element of an array constant to NULL, write NULL for the element value. (Any upper- or lower-case variant of NULL will do.) If you want an actual string value “NULL”, you must put double quotes around it.

(These kinds of array constants are actually only a special case of the generic type constants discussed in Section 4.1.2.7. The constant is initially treated as a string and passed to the array input conversion routine. An explicit type specification might be necessary.)

Now we can show some INSERT statements:

INSERT INTO sal_emp
    VALUES ('Bill',
    '{10000, 10000, 10000, 10000}',
    '{{"meeting", "lunch"}, {"training", "presentation"}}');

INSERT INTO sal_emp
    VALUES ('Carol',
    '{20000, 25000, 25000, 25000}',
    '{{"breakfast", "consulting"}, {"meeting", "lunch"}}');

The result of the previous two inserts looks like this:

SELECT * FROM sal_emp;
 name  |      pay_by_quarter       |                 schedule
-------+---------------------------+-------------------------------------------
 Bill  | {10000,10000,10000,10000} | {{meeting,lunch},{training,presentation}}
 Carol | {20000,25000,25000,25000} | {{breakfast,consulting},{meeting,lunch}}
(2 rows)

Multidimensional arrays must have matching extents for each dimension. A mismatch causes an error, for example:

INSERT INTO sal_emp
    VALUES ('Bill',
    '{10000, 10000, 10000, 10000}',
    '{{"meeting", "lunch"}, {"meeting"}}');
ERROR:  multidimensional arrays must have array expressions with matching dimensions

The ARRAY constructor syntax can also be used:

INSERT INTO sal_emp
    VALUES ('Bill',
    ARRAY[10000, 10000, 10000, 10000],
    ARRAY[['meeting', 'lunch'], ['training', 'presentation']]);

INSERT INTO sal_emp
    VALUES ('Carol',
    ARRAY[20000, 25000, 25000, 25000],
    ARRAY[['breakfast', 'consulting'], ['meeting', 'lunch']]);

Notice that the array elements are ordinary SQL constants or expressions; for instance, string literals are single quoted, instead of double quoted as they would be in an array literal. The ARRAY constructor syntax is discussed in more detail in Section 4.2.12.

8.15.3. Accessing Arrays

Now, we can run some queries on the table. First, we show how to access a single element of an array. This query retrieves the names of the employees whose pay changed in the second quarter:

SELECT name FROM sal_emp WHERE pay_by_quarter[1] <> pay_by_quarter[2];

 name
-------
 Carol
(1 row)

The array subscript numbers are written within square brackets. By default PostgreSQL uses a one-based numbering convention for arrays, that is, an array of n elements starts with array[1] and ends with array[n].

This query retrieves the third quarter pay of all employees:

SELECT pay_by_quarter[3] FROM sal_emp;

 pay_by_quarter
----------------
          10000
          25000
(2 rows)

We can also access arbitrary rectangular slices of an array, or subarrays. An array slice is denoted by writing lower-bound:upper-bound for one or more array dimensions. For example, this query retrieves the first item on Bill's schedule for the first two days of the week:

SELECT schedule[1:2][1:1] FROM sal_emp WHERE name = 'Bill';

        schedule
------------------------
 {{meeting},{training}}
(1 row)

If any dimension is written as a slice, i.e., contains a colon, then all dimensions are treated as slices. Any dimension that has only a single number (no colon) is treated as being from 1 to the number specified. For example, [2] is treated as [1:2], as in this example:

SELECT schedule[1:2][2] FROM sal_emp WHERE name = 'Bill';

                 schedule
-------------------------------------------
 {{meeting,lunch},{training,presentation}}
(1 row)

To avoid confusion with the non-slice case, it's best to use slice syntax for all dimensions, e.g., [1:2][1:1], not [2][1:1].

It is possible to omit the lower-bound and/or upper-bound of a slice specifier; the missing bound is replaced by the lower or upper limit of the array's subscripts. For example:

SELECT schedule[:2][2:] FROM sal_emp WHERE name = 'Bill';

        schedule
------------------------
 {{lunch},{presentation}}
(1 row)

SELECT schedule[:][1:1] FROM sal_emp WHERE name = 'Bill';

        schedule
------------------------
 {{meeting},{training}}
(1 row)

An array subscript expression will return null if either the array itself or any of the subscript expressions are null. Also, null is returned if a subscript is outside the array bounds (this case does not raise an error). For example, if schedule currently has the dimensions [1:3][1:2] then referencing schedule[3][3] yields NULL. Similarly, an array reference with the wrong number of subscripts yields a null rather than an error.

An array slice expression likewise yields null if the array itself or any of the subscript expressions are null. However, in other cases such as selecting an array slice that is completely outside the current array bounds, a slice expression yields an empty (zero-dimensional) array instead of null. (This does not match non-slice behavior and is done for historical reasons.) If the requested slice partially overlaps the array bounds, then it is silently reduced to just the overlapping region instead of returning null.

The current dimensions of any array value can be retrieved with the array_dims function:

SELECT array_dims(schedule) FROM sal_emp WHERE name = 'Carol';

 array_dims
------------
 [1:2][1:2]
(1 row)

array_dims produces a text result, which is convenient for people to read but perhaps inconvenient for programs. Dimensions can also be retrieved with array_upper and array_lower, which return the upper and lower bound of a specified array dimension, respectively:

SELECT array_upper(schedule, 1) FROM sal_emp WHERE name = 'Carol';

 array_upper
-------------
           2
(1 row)

array_length will return the length of a specified array dimension:

SELECT array_length(schedule, 1) FROM sal_emp WHERE name = 'Carol';

 array_length
--------------
            2
(1 row)

cardinality returns the total number of elements in an array across all dimensions. It is effectively the number of rows a call to unnest would yield:

SELECT cardinality(schedule) FROM sal_emp WHERE name = 'Carol';

 cardinality
-------------
           4
(1 row)

8.15.4. Modifying Arrays

An array value can be replaced completely:

UPDATE sal_emp SET pay_by_quarter = '{25000,25000,27000,27000}'
    WHERE name = 'Carol';

or using the ARRAY expression syntax:

UPDATE sal_emp SET pay_by_quarter = ARRAY[25000,25000,27000,27000]
    WHERE name = 'Carol';

An array can also be updated at a single element:

UPDATE sal_emp SET pay_by_quarter[4] = 15000
    WHERE name = 'Bill';

or updated in a slice:

UPDATE sal_emp SET pay_by_quarter[1:2] = '{27000,27000}'
    WHERE name = 'Carol';

The slice syntaxes with omitted lower-bound and/or upper-bound can be used too, but only when updating an array value that is not NULL or zero-dimensional (otherwise, there is no existing subscript limit to substitute).

A stored array value can be enlarged by assigning to elements not already present. Any positions between those previously present and the newly assigned elements will be filled with nulls. For example, if array myarray currently has 4 elements, it will have six elements after an update that assigns to myarray[6]; myarray[5] will contain null. Currently, enlargement in this fashion is only allowed for one-dimensional arrays, not multidimensional arrays.

Subscripted assignment allows creation of arrays that do not use one-based subscripts. For example one might assign to myarray[-2:7] to create an array with subscript values from -2 to 7.

New array values can also be constructed using the concatenation operator, ||:

SELECT ARRAY[1,2] || ARRAY[3,4];
 ?column?
-----------
 {1,2,3,4}
(1 row)

SELECT ARRAY[5,6] || ARRAY[[1,2],[3,4]];
      ?column?
---------------------
 {{5,6},{1,2},{3,4}}
(1 row)

The concatenation operator allows a single element to be pushed onto the beginning or end of a one-dimensional array. It also accepts two N-dimensional arrays, or an N-dimensional and an N+1-dimensional array.

When a single element is pushed onto either the beginning or end of a one-dimensional array, the result is an array with the same lower bound subscript as the array operand. For example:

SELECT array_dims(1 || '[0:1]={2,3}'::int[]);
 array_dims
------------
 [0:2]
(1 row)

SELECT array_dims(ARRAY[1,2] || 3);
 array_dims
------------
 [1:3]
(1 row)

When two arrays with an equal number of dimensions are concatenated, the result retains the lower bound subscript of the left-hand operand's outer dimension. The result is an array comprising every element of the left-hand operand followed by every element of the right-hand operand. For example:

SELECT array_dims(ARRAY[1,2] || ARRAY[3,4,5]);
 array_dims
------------
 [1:5]
(1 row)

SELECT array_dims(ARRAY[[1,2],[3,4]] || ARRAY[[5,6],[7,8],[9,0]]);
 array_dims
------------
 [1:5][1:2]
(1 row)

When an N-dimensional array is pushed onto the beginning or end of an N+1-dimensional array, the result is analogous to the element-array case above. Each N-dimensional sub-array is essentially an element of the N+1-dimensional array's outer dimension. For example:

SELECT array_dims(ARRAY[1,2] || ARRAY[[3,4],[5,6]]);
 array_dims
------------
 [1:3][1:2]
(1 row)

An array can also be constructed by using the functions array_prepend, array_append, or array_cat. The first two only support one-dimensional arrays, but array_cat supports multidimensional arrays. Some examples:

SELECT array_prepend(1, ARRAY[2,3]);
 array_prepend
---------------
 {1,2,3}
(1 row)

SELECT array_append(ARRAY[1,2], 3);
 array_append
--------------
 {1,2,3}
(1 row)

SELECT array_cat(ARRAY[1,2], ARRAY[3,4]);
 array_cat
-----------
 {1,2,3,4}
(1 row)

SELECT array_cat(ARRAY[[1,2],[3,4]], ARRAY[5,6]);
      array_cat
---------------------
 {{1,2},{3,4},{5,6}}
(1 row)

SELECT array_cat(ARRAY[5,6], ARRAY[[1,2],[3,4]]);
      array_cat
---------------------
 {{5,6},{1,2},{3,4}}

In simple cases, the concatenation operator discussed above is preferred over direct use of these functions. However, because the concatenation operator is overloaded to serve all three cases, there are situations where use of one of the functions is helpful to avoid ambiguity. For example consider:

SELECT ARRAY[1, 2] || '{3, 4}';  -- the untyped literal is taken as an array
 ?column?
-----------
 {1,2,3,4}

SELECT ARRAY[1, 2] || '7';                 -- so is this one
ERROR:  malformed array literal: "7"

SELECT ARRAY[1, 2] || NULL;                -- so is an undecorated NULL
 ?column?
----------
 {1,2}
(1 row)

SELECT array_append(ARRAY[1, 2], NULL);    -- this might have been meant
 array_append
--------------
 {1,2,NULL}

In the examples above, the parser sees an integer array on one side of the concatenation operator, and a constant of undetermined type on the other. The heuristic it uses to resolve the constant's type is to assume it's of the same type as the operator's other input — in this case, integer array. So the concatenation operator is presumed to represent array_cat, not array_append. When that's the wrong choice, it could be fixed by casting the constant to the array's element type; but explicit use of array_append might be a preferable solution.

8.15.5. Searching in Arrays

To search for a value in an array, each value must be checked. This can be done manually, if you know the size of the array. For example:

SELECT * FROM sal_emp WHERE pay_by_quarter[1] = 10000 OR
                            pay_by_quarter[2] = 10000 OR
                            pay_by_quarter[3] = 10000 OR
                            pay_by_quarter[4] = 10000;

However, this quickly becomes tedious for large arrays, and is not helpful if the size of the array is unknown. An alternative method is described in Section 9.24. The above query could be replaced by:

SELECT * FROM sal_emp WHERE 10000 = ANY (pay_by_quarter);

In addition, you can find rows where the array has all values equal to 10000 with:

SELECT * FROM sal_emp WHERE 10000 = ALL (pay_by_quarter);

Alternatively, the generate_subscripts function can be used. For example:

SELECT * FROM
   (SELECT pay_by_quarter,
           generate_subscripts(pay_by_quarter, 1) AS s
      FROM sal_emp) AS foo
 WHERE pay_by_quarter[s] = 10000;

This function is described in Table 9.64.

You can also search an array using the && operator, which checks whether the left operand overlaps with the right operand. For instance:

SELECT * FROM sal_emp WHERE pay_by_quarter && ARRAY[10000];

This and other array operators are further described in Section 9.19. It can be accelerated by an appropriate index, as described in Section 11.2.

You can also search for specific values in an array using the array_position and array_positions functions. The former returns the subscript of the first occurrence of a value in an array; the latter returns an array with the subscripts of all occurrences of the value in the array. For example:

SELECT array_position(ARRAY['sun','mon','tue','wed','thu','fri','sat'], 'mon');
 array_position
----------------
              2
(1 row)

SELECT array_positions(ARRAY[1, 4, 3, 1, 3, 4, 2, 1], 1);
 array_positions
-----------------
 {1,4,8}
(1 row)

Tip

Arrays are not sets; searching for specific array elements can be a sign of database misdesign. Consider using a separate table with a row for each item that would be an array element. This will be easier to search, and is likely to scale better for a large number of elements.

8.15.6. Array Input and Output Syntax

The external text representation of an array value consists of items that are interpreted according to the I/O conversion rules for the array's element type, plus decoration that indicates the array structure. The decoration consists of curly braces ({ and }) around the array value plus delimiter characters between adjacent items. The delimiter character is usually a comma (,) but can be something else: it is determined by the typdelim setting for the array's element type. Among the standard data types provided in the PostgreSQL distribution, all use a comma, except for type box, which uses a semicolon (;). In a multidimensional array, each dimension (row, plane, cube, etc.) gets its own level of curly braces, and delimiters must be written between adjacent curly-braced entities of the same level.

The array output routine will put double quotes around element values if they are empty strings, contain curly braces, delimiter characters, double quotes, backslashes, or white space, or match the word NULL. Double quotes and backslashes embedded in element values will be backslash-escaped. For numeric data types it is safe to assume that double quotes will never appear, but for textual data types one should be prepared to cope with either the presence or absence of quotes.

By default, the lower bound index value of an array's dimensions is set to one. To represent arrays with other lower bounds, the array subscript ranges can be specified explicitly before writing the array contents. This decoration consists of square brackets ([]) around each array dimension's lower and upper bounds, with a colon (:) delimiter character in between. The array dimension decoration is followed by an equal sign (=). For example:

SELECT f1[1][-2][3] AS e1, f1[1][-1][5] AS e2
 FROM (SELECT '[1:1][-2:-1][3:5]={{{1,2,3},{4,5,6}}}'::int[] AS f1) AS ss;

 e1 | e2
----+----
  1 |  6
(1 row)

The array output routine will include explicit dimensions in its result only when there are one or more lower bounds different from one.

If the value written for an element is NULL (in any case variant), the element is taken to be NULL. The presence of any quotes or backslashes disables this and allows the literal string value “NULL” to be entered. Also, for backward compatibility with pre-8.2 versions of PostgreSQL, the array_nulls configuration parameter can be turned off to suppress recognition of NULL as a NULL.

As shown previously, when writing an array value you can use double quotes around any individual array element. You must do so if the element value would otherwise confuse the array-value parser. For example, elements containing curly braces, commas (or the data type's delimiter character), double quotes, backslashes, or leading or trailing whitespace must be double-quoted. Empty strings and strings matching the word NULL must be quoted, too. To put a double quote or backslash in a quoted array element value, precede it with a backslash. Alternatively, you can avoid quotes and use backslash-escaping to protect all data characters that would otherwise be taken as array syntax.

You can add whitespace before a left brace or after a right brace. You can also add whitespace before or after any individual item string. In all of these cases the whitespace will be ignored. However, whitespace within double-quoted elements, or surrounded on both sides by non-whitespace characters of an element, is not ignored.

Tip

The ARRAY constructor syntax (see Section 4.2.12) is often easier to work with than the array-literal syntax when writing array values in SQL commands. In ARRAY, individual element values are written the same way they would be written when not members of an array.

Prev | Up | Next
8.14. JSON Types | Home | 8.16. Composite Types


8.17. Range Types
Prev Up Chapter 8. Data Types Home Next

8.17. Range Types

8.17.1. Built-in Range and Multirange Types
8.17.2. Examples
8.17.3. Inclusive and Exclusive Bounds
8.17.4. Infinite (Unbounded) Ranges
8.17.5. Range Input/Output
8.17.6. Constructing Ranges and Multiranges
8.17.7. Discrete Range Types
8.17.8. Defining New Range Types
8.17.9. Indexing
8.17.10. Constraints on Ranges

Range types are data types representing a range of values of some element type (called the range's subtype). For instance, ranges of timestamp might be used to represent the ranges of time that a meeting room is reserved. In this case the data type is tsrange (short for “timestamp range”), and timestamp is the subtype. The subtype must have a total order so that it is well-defined whether element values are within, before, or after a range of values.

Range types are useful because they represent many element values in a single range value, and because concepts such as overlapping ranges can be expressed clearly. The use of time and date ranges for scheduling purposes is the clearest example; but price ranges, measurement ranges from an instrument, and so forth can also be useful.

Every range type has a corresponding multirange type. A multirange is an ordered list of non-contiguous, non-empty, non-null ranges. Most range operators also work on multiranges, and they have a few functions of their own.

8.17.1. Built-in Range and Multirange Types

PostgreSQL comes with the following built-in range types:

  • int4range — Range of integer, int4multirange — corresponding Multirange

  • int8range — Range of bigint, int8multirange — corresponding Multirange

  • numrange — Range of numeric, nummultirange — corresponding Multirange

  • tsrange — Range of timestamp without time zone, tsmultirange — corresponding Multirange

  • tstzrange — Range of timestamp with time zone, tstzmultirange — corresponding Multirange

  • daterange — Range of date, datemultirange — corresponding Multirange

In addition, you can define your own range types; see CREATE TYPE for more information.

8.17.2. Examples

CREATE TABLE reservation (room int, during tsrange);
INSERT INTO reservation VALUES
    (1108, '[2010-01-01 14:30, 2010-01-01 15:30)');

-- Containment
SELECT int4range(10, 20) @> 3;

-- Overlaps
SELECT numrange(11.1, 22.2) && numrange(20.0, 30.0);

-- Extract the upper bound
SELECT upper(int8range(15, 25));

-- Compute the intersection
SELECT int4range(10, 20) * int4range(15, 25);

-- Is the range empty?
SELECT isempty(numrange(1, 5));

See Table 9.53 and Table 9.55 for complete lists of operators and functions on range types.

8.17.3. Inclusive and Exclusive Bounds

Every non-empty range has two bounds, the lower bound and the upper bound. All points between these values are included in the range. An inclusive bound means that the boundary point itself is included in the range as well, while an exclusive bound means that the boundary point is not included in the range.

In the text form of a range, an inclusive lower bound is represented by “[” while an exclusive lower bound is represented by “(”. Likewise, an inclusive upper bound is represented by “]”, while an exclusive upper bound is represented by “)”. (See Section 8.17.5 for more details.)

The functions lower_inc and upper_inc test the inclusivity of the lower and upper bounds of a range value, respectively.

8.17.4. Infinite (Unbounded) Ranges

The lower bound of a range can be omitted, meaning that all values less than the upper bound are included in the range, e.g., (,3]. Likewise, if the upper bound of the range is omitted, then all values greater than the lower bound are included in the range. If both lower and upper bounds are omitted, all values of the element type are considered to be in the range. Specifying a missing bound as inclusive is automatically converted to exclusive, e.g., [,] is converted to (,). You can think of these missing values as +/-infinity, but they are special range type values and are considered to be beyond any range element type's +/-infinity values.

Element types that have the notion of “infinity” can use them as explicit bound values. For example, with timestamp ranges, [today,infinity) excludes the special timestamp value infinity, while [today,infinity] include it, as does [today,) and [today,].

The functions lower_inf and upper_inf test for infinite lower and upper bounds of a range, respectively.

8.17.5. Range Input/Output

The input for a range value must follow one of the following patterns:

(lower-bound,upper-bound)
(lower-bound,upper-bound]
[lower-bound,upper-bound)
[lower-bound,upper-bound]
empty

The parentheses or brackets indicate whether the lower and upper bounds are exclusive or inclusive, as described previously. Notice that the final pattern is empty, which represents an empty range (a range that contains no points).

The lower-bound may be either a string that is valid input for the subtype, or empty to indicate no lower bound. Likewise, upper-bound may be either a string that is valid input for the subtype, or empty to indicate no upper bound.

Each bound value can be quoted using " (double quote) characters. This is necessary if the bound value contains parentheses, brackets, commas, double quotes, or backslashes, since these characters would otherwise be taken as part of the range syntax. To put a double quote or backslash in a quoted bound value, precede it with a backslash. (Also, a pair of double quotes within a double-quoted bound value is taken to represent a double quote character, analogously to the rules for single quotes in SQL literal strings.) Alternatively, you can avoid quoting and use backslash-escaping to protect all data characters that would otherwise be taken as range syntax. Also, to write a bound value that is an empty string, write "", since writing nothing means an infinite bound.

Whitespace is allowed before and after the range value, but any whitespace between the parentheses or brackets is taken as part of the lower or upper bound value. (Depending on the element type, it might or might not be significant.)

Note

These rules are very similar to those for writing field values in composite-type literals. See Section 8.16.6 for additional commentary.

Examples:

-- includes 3, does not include 7, and does include all points in between
SELECT '[3,7)'::int4range;

-- does not include either 3 or 7, but includes all points in between
SELECT '(3,7)'::int4range;

-- includes only the single point 4
SELECT '[4,4]'::int4range;

-- includes no points (and will be normalized to 'empty')
SELECT '[4,4)'::int4range;

The input for a multirange is curly brackets ({ and }) containing zero or more valid ranges, separated by commas. Whitespace is permitted around the brackets and commas. This is intended to be reminiscent of array syntax, although multiranges are much simpler: they have just one dimension and there is no need to quote their contents. (The bounds of their ranges may be quoted as above however.)

Examples:

SELECT '{}'::int4multirange;
SELECT '{[3,7)}'::int4multirange;
SELECT '{[3,7), [8,9)}'::int4multirange;

8.17.6. Constructing Ranges and Multiranges

Each range type has a constructor function with the same name as the range type. Using the constructor function is frequently more convenient than writing a range literal constant, since it avoids the need for extra quoting of the bound values. The constructor function accepts two or three arguments. The two-argument form constructs a range in standard form (lower bound inclusive, upper bound exclusive), while the three-argument form constructs a range with bounds of the form specified by the third argument. The third argument must be one of the strings “()”, “(]”, “[)”, or “[]”. For example:

-- The full form is: lower bound, upper bound, and text argument indicating
-- inclusivity/exclusivity of bounds.
SELECT numrange(1.0, 14.0, '(]');

-- If the third argument is omitted, '[)' is assumed.
SELECT numrange(1.0, 14.0);

-- Although '(]' is specified here, on display the value will be converted to
-- canonical form, since int8range is a discrete range type (see below).
SELECT int8range(1, 14, '(]');

-- Using NULL for either bound causes the range to be unbounded on that side.
SELECT numrange(NULL, 2.2);

Each range type also has a multirange constructor with the same name as the multirange type. The constructor function takes zero or more arguments which are all ranges of the appropriate type. For example:

SELECT nummultirange();
SELECT nummultirange(numrange(1.0, 14.0));
SELECT nummultirange(numrange(1.0, 14.0), numrange(20.0, 25.0));

8.17.7. Discrete Range Types

A discrete range is one whose element type has a well-defined “step”, such as integer or date. In these types two elements can be said to be adjacent, when there are no valid values between them. This contrasts with continuous ranges, where it's always (or almost always) possible to identify other element values between two given values. For example, a range over the numeric type is continuous, as is a range over timestamp. (Even though timestamp has limited precision, and so could theoretically be treated as discrete, it's better to consider it continuous since the step size is normally not of interest.)

Another way to think about a discrete range type is that there is a clear idea of a “next” or “previous” value for each element value. Knowing that, it is possible to convert between inclusive and exclusive representations of a range's bounds, by choosing the next or previous element value instead of the one originally given. For example, in an integer range type [4,8] and (3,9) denote the same set of values; but this would not be so for a range over numeric.

A discrete range type should have a canonicalization function that is aware of the desired step size for the element type. The canonicalization function is charged with converting equivalent values of the range type to have identical representations, in particular consistently inclusive or exclusive bounds. If a canonicalization function is not specified, then ranges with different formatting will always be treated as unequal, even though they might represent the same set of values in reality.

The built-in range types int4range, int8range, and daterange all use a canonical form that includes the lower bound and excludes the upper bound; that is, [). User-defined range types can use other conventions, however.

8.17.8. Defining New Range Types

Users can define their own range types. The most common reason to do this is to use ranges over subtypes not provided among the built-in range types. For example, to define a new range type of subtype float8:

CREATE TYPE floatrange AS RANGE (
    subtype = float8,
    subtype_diff = float8mi
);

SELECT '[1.234, 5.678]'::floatrange;

Because float8 has no meaningful “step”, we do not define a canonicalization function in this example.

When you define your own range you automatically get a corresponding multirange type.

Defining your own range type also allows you to specify a different subtype B-tree operator class or collation to use, so as to change the sort ordering that determines which values fall into a given range.

If the subtype is considered to have discrete rather than continuous values, the CREATE TYPE command should specify a canonical function. The canonicalization function takes an input range value, and must return an equivalent range value that may have different bounds and formatting. The canonical output for two ranges that represent the same set of values, for example the integer ranges [1, 7] and [1, 8), must be identical. It doesn't matter which representation you choose to be the canonical one, so long as two equivalent values with different formattings are always mapped to the same value with the same formatting. In addition to adjusting the inclusive/exclusive bounds format, a canonicalization function might round off boundary values, in case the desired step size is larger than what the subtype is capable of storing. For instance, a range type over timestamp could be defined to have a step size of an hour, in which case the canonicalization function would need to round off bounds that weren't a multiple of an hour, or perhaps throw an error instead.

In addition, any range type that is meant to be used with GiST or SP-GiST indexes should define a subtype difference, or subtype_diff, function. (The index will still work without subtype_diff, but it is likely to be considerably less efficient than if a difference function is provided.) The subtype difference function takes two input values of the subtype, and returns their difference (i.e., X minus Y) represented as a float8 value. In our example above, the function float8mi that underlies the regular float8 minus operator can be used; but for any other subtype, some type conversion would be necessary. Some creative thought about how to represent differences as numbers might be needed, too. To the greatest extent possible, the subtype_diff function should agree with the sort ordering implied by the selected operator class and collation; that is, its result should be positive whenever its first argument is greater than its second according to the sort ordering.

A less-oversimplified example of a subtype_diff function is:

CREATE FUNCTION time_subtype_diff(x time, y time) RETURNS float8 AS
'SELECT EXTRACT(EPOCH FROM (x - y))' LANGUAGE sql STRICT IMMUTABLE;

CREATE TYPE timerange AS RANGE (
    subtype = time,
    subtype_diff = time_subtype_diff
);

SELECT '[11:10, 23:00]'::timerange;

See CREATE TYPE for more information about creating range types.

8.17.9. Indexing

GiST and SP-GiST indexes can be created for table columns of range types. GiST indexes can be also created for table columns of multirange types. For instance, to create a GiST index:

CREATE INDEX reservation_idx ON reservation USING GIST (during);

A GiST or SP-GiST index on ranges can accelerate queries involving these range operators: =, &&, <@, @>, <<, >>, -|-, &<, and &>. A GiST index on multiranges can accelerate queries involving the same set of multirange operators. A GiST index on ranges and GiST index on multiranges can also accelerate queries involving these cross-type range to multirange and multirange to range operators correspondingly: &&, <@, @>, <<, >>, -|-, &<, and &>. See Table 9.53 for more information.

In addition, B-tree and hash indexes can be created for table columns of range types. For these index types, basically the only useful range operation is equality. There is a B-tree sort ordering defined for range values, with corresponding < and > operators, but the ordering is rather arbitrary and not usually useful in the real world. Range types' B-tree and hash support is primarily meant to allow sorting and hashing internally in queries, rather than creation of actual indexes.

8.17.10. Constraints on Ranges

While UNIQUE is a natural constraint for scalar values, it is usually unsuitable for range types. Instead, an exclusion constraint is often more appropriate (see CREATE TABLE ... CONSTRAINT ... EXCLUDE). Exclusion constraints allow the specification of constraints such as “non-overlapping” on a range type. For example:

CREATE TABLE reservation (
    during tsrange,
    EXCLUDE USING GIST (during WITH &&)
);

That constraint will prevent any overlapping values from existing in the table at the same time:

INSERT INTO reservation VALUES
    ('[2010-01-01 11:30, 2010-01-01 15:00)');
INSERT 0 1

INSERT INTO reservation VALUES
    ('[2010-01-01 14:45, 2010-01-01 15:45)');
ERROR:  conflicting key value violates exclusion constraint "reservation_during_excl"
DETAIL:  Key (during)=(["2010-01-01 14:45:00","2010-01-01 15:45:00")) conflicts
with existing key (during)=(["2010-01-01 11:30:00","2010-01-01 15:00:00")).

You can use the btree_gist extension to define exclusion constraints on plain scalar data types, which can then be combined with range exclusions for maximum flexibility. For example, after btree_gist is installed, the following constraint will reject overlapping ranges only if the meeting room numbers are equal:

CREATE EXTENSION btree_gist;
CREATE TABLE room_reservation (
    room text,
    during tsrange,
    EXCLUDE USING GIST (room WITH =, during WITH &&)
);

INSERT INTO room_reservation VALUES
    ('123A', '[2010-01-01 14:00, 2010-01-01 15:00)');
INSERT 0 1

INSERT INTO room_reservation VALUES
    ('123A', '[2010-01-01 14:30, 2010-01-01 15:30)');
ERROR:  conflicting key value violates exclusion constraint "room_reservation_room_during_excl"
DETAIL:  Key (room, during)=(123A, ["2010-01-01 14:30:00","2010-01-01 15:30:00")) conflicts
with existing key (room, during)=(123A, ["2010-01-01 14:00:00","2010-01-01 15:00:00")).

INSERT INTO room_reservation VALUES
    ('123B', '[2010-01-01 14:30, 2010-01-01 15:30)');
INSERT 0 1

Prev | Up | Next
8.16. Composite Types | Home | 8.18. Domain Types


8.18. Domain Types
Prev Up Chapter 8. Data Types Home Next

8.18. Domain Types

A domain is a user-defined data type that is based on another underlying type. Optionally, it can have constraints that restrict its valid values to a subset of what the underlying type would allow. Otherwise it behaves like the underlying type — for example, any operator or function that can be applied to the underlying type will work on the domain type. The underlying type can be any built-in or user-defined base type, enum type, array type, composite type, range type, or another domain.

For example, we could create a domain over integers that accepts only positive integers:

CREATE DOMAIN posint AS integer CHECK (VALUE > 0);
CREATE TABLE mytable (id posint);
INSERT INTO mytable VALUES(1);   -- works
INSERT INTO mytable VALUES(-1);  -- fails

When an operator or function of the underlying type is applied to a domain value, the domain is automatically down-cast to the underlying type. Thus, for example, the result of mytable.id - 1 is considered to be of type integer not posint. We could write (mytable.id - 1)::posint to cast the result back to posint, causing the domain's constraints to be rechecked. In this case, that would result in an error if the expression had been applied to an id value of 1. Assigning a value of the underlying type to a field or variable of the domain type is allowed without writing an explicit cast, but the domain's constraints will be checked.

For additional information see CREATE DOMAIN.

Prev | Up | Next
8.17. Range Types | Home | 8.19. Object Identifier Types


8.19. Object Identifier Types
Prev Up Chapter 8. Data Types Home Next

8.19. Object Identifier Types

Object identifiers (OIDs) are used internally by PostgreSQL as primary keys for various system tables. Type oid represents an object identifier. There are also several alias types for oid, each named regsomething. Table 8.26 shows an overview.

The oid type is currently implemented as an unsigned four-byte integer. Therefore, it is not large enough to provide database-wide uniqueness in large databases, or even in large individual tables.

The oid type itself has few operations beyond comparison. It can be cast to integer, however, and then manipulated using the standard integer operators. (Beware of possible signed-versus-unsigned confusion if you do this.)

The OID alias types have no operations of their own except for specialized input and output routines. These routines are able to accept and display symbolic names for system objects, rather than the raw numeric value that type oid would use. The alias types allow simplified lookup of OID values for objects. For example, to examine the pg_attribute rows related to a table mytable, one could write:

SELECT * FROM pg_attribute WHERE attrelid = 'mytable'::regclass;

rather than:

SELECT * FROM pg_attribute
  WHERE attrelid = (SELECT oid FROM pg_class WHERE relname = 'mytable');

While that doesn't look all that bad by itself, it's still oversimplified. A far more complicated sub-select would be needed to select the right OID if there are multiple tables named mytable in different schemas. The regclass input converter handles the table lookup according to the schema path setting, and so it does the “right thing” automatically. Similarly, casting a table's OID to regclass is handy for symbolic display of a numeric OID.

Table 8.26. Object Identifier Types

Name | References | Description | Value Example
oid | any | numeric object identifier | 564182
regclass | pg_class | relation name | pg_type
regcollation | pg_collation | collation name | "POSIX"
regconfig | pg_ts_config | text search configuration | english
regdictionary | pg_ts_dict | text search dictionary | simple
regnamespace | pg_namespace | namespace name | pg_catalog
regoper | pg_operator | operator name | +
regoperator | pg_operator | operator with argument types | *(integer,​integer) or -(NONE,​integer)
regproc | pg_proc | function name | sum
regprocedure | pg_proc | function with argument types | sum(int4)
regrole | pg_authid | role name | smithee
regtype | pg_type | data type name | integer

All of the OID alias types for objects that are grouped by namespace accept schema-qualified names, and will display schema-qualified names on output if the object would not be found in the current search path without being qualified. For example, myschema.mytable is acceptable input for regclass (if there is such a table). That value might be output as myschema.mytable, or just mytable, depending on the current search path. The regproc and regoper alias types will only accept input names that are unique (not overloaded), so they are of limited use; for most uses regprocedure or regoperator are more appropriate. For regoperator, unary operators are identified by writing NONE for the unused operand.

The input functions for these types allow whitespace between tokens, and will fold upper-case letters to lower case, except within double quotes; this is done to make the syntax rules similar to the way object names are written in SQL. Conversely, the output functions will use double quotes if needed to make the output be a valid SQL identifier. For example, the OID of a function named Foo (with upper case F) taking two integer arguments could be entered as ' "Foo" ( int, integer ) '::regprocedure. The output would look like "Foo"(integer,integer). Both the function name and the argument type names could be schema-qualified, too.

Many built-in PostgreSQL functions accept the OID of a table, or another kind of database object, and for convenience are declared as taking regclass (or the appropriate OID alias type). This means you do not have to look up the object's OID by hand, but can just enter its name as a string literal. For example, the nextval(regclass) function takes a sequence relation's OID, so you could call it like this:

nextval('foo')              operates on sequence foo
nextval('FOO')              same as above
nextval('"Foo"')            operates on sequence Foo
nextval('myschema.foo')     operates on myschema.foo
nextval('"myschema".foo')   same as above
nextval('foo')              searches search path for foo

Note

When you write the argument of such a function as an unadorned literal string, it becomes a constant of type regclass (or the appropriate type). Since this is really just an OID, it will track the originally identified object despite later renaming, schema reassignment, etc. This “early binding” behavior is usually desirable for object references in column defaults and views. But sometimes you might want “late binding” where the object reference is resolved at run time. To get late-binding behavior, force the constant to be stored as a text constant instead of regclass:

nextval('foo'::text)      foo is looked up at runtime

The to_regclass() function and its siblings can also be used to perform run-time lookups. See Table 9.70.

Another practical example of use of regclass is to look up the OID of a table listed in the information_schema views, which don't supply such OIDs directly. One might for example wish to call the pg_relation_size() function, which requires the table OID. Taking the above rules into account, the correct way to do that is

SELECT table_schema, table_name,
       pg_relation_size((quote_ident(table_schema) || '.' ||
                         quote_ident(table_name))::regclass)
FROM information_schema.tables
WHERE ...

The quote_ident() function will take care of double-quoting the identifiers where needed. The seemingly easier

SELECT pg_relation_size(table_name)
FROM information_schema.tables
WHERE ...

is not recommended, because it will fail for tables that are outside your search path or have names that require quoting.

An additional property of most of the OID alias types is the creation of dependencies. If a constant of one of these types appears in a stored expression (such as a column default expression or view), it creates a dependency on the referenced object. For example, if a column has a default expression nextval('my_seq'::regclass), PostgreSQL understands that the default expression depends on the sequence my_seq, so the system will not let the sequence be dropped without first removing the default expression. The alternative of nextval('my_seq'::text) does not create a dependency. (regrole is an exception to this property. Constants of this type are not allowed in stored expressions.)

Another identifier type used by the system is xid, or transaction (abbreviated xact) identifier. This is the data type of the system columns xmin and xmax. Transaction identifiers are 32-bit quantities. In some contexts, a 64-bit variant xid8 is used. Unlike xid values, xid8 values increase strictly monotonically and cannot be reused in the lifetime of a database cluster.

A third identifier type used by the system is cid, or command identifier. This is the data type of the system columns cmin and cmax. Command identifiers are also 32-bit quantities.

A final identifier type used by the system is tid, or tuple identifier (row identifier). This is the data type of the system column ctid. A tuple ID is a pair (block number, tuple index within block) that identifies the physical location of the row within its table.

(The system columns are further explained in Section 5.5.)

Prev | Up | Next
8.18. Domain Types | Home | 8.20. pg_lsn Type


8.20. pg_lsn Type
Prev Up Chapter 8. Data Types Home Next

8.20. pg_lsn Type

The pg_lsn data type can be used to store LSN (Log Sequence Number) data which is a pointer to a location in the WAL. This type is a representation of XLogRecPtr and an internal system type of PostgreSQL.

Internally, an LSN is a 64-bit integer, representing a byte position in the write-ahead log stream. It is printed as two hexadecimal numbers of up to 8 digits each, separated by a slash; for example, 16/B374D848. The pg_lsn type supports the standard comparison operators, like = and >. Two LSNs can be subtracted using the - operator; the result is the number of bytes separating those write-ahead log locations. Also the number of bytes can be added into and subtracted from LSN using the +(pg_lsn,numeric) and -(pg_lsn,numeric) operators, respectively. Note that the calculated LSN should be in the range of pg_lsn type, i.e., between 0/0 and FFFFFFFF/FFFFFFFF.

Prev | Up | Next
8.19. Object Identifier Types | Home | 8.21. Pseudo-Types


8.21. Pseudo-Types
Prev Up Chapter 8. Data Types Home Next

8.21. Pseudo-Types

The PostgreSQL type system contains a number of special-purpose entries that are collectively called pseudo-types. A pseudo-type cannot be used as a column data type, but it can be used to declare a function's argument or result type. Each of the available pseudo-types is useful in situations where a function's behavior does not correspond to simply taking or returning a value of a specific SQL data type. Table 8.27 lists the existing pseudo-types.

Table 8.27. Pseudo-Types

Name | Description
any | Indicates that a function accepts any input data type.
anyelement | Indicates that a function accepts any data type (see Section 38.2.5).
anyarray | Indicates that a function accepts any array data type (see Section 38.2.5).
anynonarray | Indicates that a function accepts any non-array data type (see Section 38.2.5).
anyenum | Indicates that a function accepts any enum data type (see Section 38.2.5 and Section 8.7).
anyrange | Indicates that a function accepts any range data type (see Section 38.2.5 and Section 8.17).
anymultirange | Indicates that a function accepts any multirange data type (see Section 38.2.5 and Section 8.17).
anycompatible | Indicates that a function accepts any data type, with automatic promotion of multiple arguments to a common data type (see Section 38.2.5).
anycompatiblearray | Indicates that a function accepts any array data type, with automatic promotion of multiple arguments to a common data type (see Section 38.2.5).
anycompatiblenonarray | Indicates that a function accepts any non-array data type, with automatic promotion of multiple arguments to a common data type (see Section 38.2.5).
anycompatiblerange | Indicates that a function accepts any range data type, with automatic promotion of multiple arguments to a common data type (see Section 38.2.5 and Section 8.17).
anycompatiblemultirange | Indicates that a function accepts any multirange data type, with automatic promotion of multiple arguments to a common data type (see Section 38.2.5 and Section 8.17).
cstring | Indicates that a function accepts or returns a null-terminated C string.
internal | Indicates that a function accepts or returns a server-internal data type.
language_handler | A procedural language call handler is declared to return language_handler.
fdw_handler | A foreign-data wrapper handler is declared to return fdw_handler.
table_am_handler | A table access method handler is declared to return table_am_handler.
index_am_handler | An index access method handler is declared to return index_am_handler.
tsm_handler | A tablesample method handler is declared to return tsm_handler.
record | Identifies a function taking or returning an unspecified row type.
trigger | A trigger function is declared to return trigger.
event_trigger | An event trigger function is declared to return event_trigger.
pg_ddl_command | Identifies a representation of DDL commands that is available to event triggers.
void | Indicates that a function returns no value.
unknown | Identifies a not-yet-resolved type, e.g., of an undecorated string literal.

Functions coded in C (whether built-in or dynamically loaded) can be declared to accept or return any of these pseudo-types. It is up to the function author to ensure that the function will behave safely when a pseudo-type is used as an argument type.

Functions coded in procedural languages can use pseudo-types only as allowed by their implementation languages. At present most procedural languages forbid use of a pseudo-type as an argument type, and allow only void and record as a result type (plus trigger or event_trigger when the function is used as a trigger or event trigger). Some also support polymorphic functions using the polymorphic pseudo-types, which are shown above and discussed in detail in Section 38.2.5.

The internal pseudo-type is used to declare functions that are meant only to be called internally by the database system, and not by direct invocation in an SQL query. If a function has at least one internal-type argument then it cannot be called from SQL. To preserve the type safety of this restriction it is important to follow this coding rule: do not create any function that is declared to return internal unless it has at least one internal argument.

Prev | Up | Next
8.20. pg_lsn Type | Home | Chapter 9. Functions and Operators


Chapter 9. Functions and Operators
Prev Up Part II. The SQL Language Home Next

Chapter 9. Functions and Operators

Table of Contents

9.1. Logical Operators
9.2. Comparison Functions and Operators
9.3. Mathematical Functions and Operators
9.4. String Functions and Operators
9.4.1. format
9.5. Binary String Functions and Operators
9.6. Bit String Functions and Operators
9.7. Pattern Matching
9.7.1. LIKE
9.7.2. SIMILAR TO Regular Expressions
9.7.3. POSIX Regular Expressions
9.8. Data Type Formatting Functions
9.9. Date/Time Functions and Operators
9.9.1. EXTRACT, date_part
9.9.2. date_trunc
9.9.3. date_bin
9.9.4. AT TIME ZONE
9.9.5. Current Date/Time
9.9.6. Delaying Execution
9.10. Enum Support Functions
9.11. Geometric Functions and Operators
9.12. Network Address Functions and Operators
9.13. Text Search Functions and Operators
9.14. UUID Functions
9.15. XML Functions
9.15.1. Producing XML Content
9.15.2. XML Predicates
9.15.3. Processing XML
9.15.4. Mapping Tables to XML
9.16. JSON Functions and Operators
9.16.1. Processing and Creating JSON Data
9.16.2. The SQL/JSON Path Language
9.17. Sequence Manipulation Functions
9.18. Conditional Expressions
9.18.1. CASE
9.18.2. COALESCE
9.18.3. NULLIF
9.18.4. GREATEST and LEAST
9.19. Array Functions and Operators
9.20. Range/Multirange Functions and Operators
9.21. Aggregate Functions
9.22. Window Functions
9.23. Subquery Expressions
9.23.1. EXISTS
9.23.2. IN
9.23.3. NOT IN
9.23.4. ANY/SOME
9.23.5. ALL
9.23.6. Single-Row Comparison
9.24. Row and Array Comparisons
9.24.1. IN
9.24.2. NOT IN
9.24.3. ANY/SOME (array)
9.24.4. ALL (array)
9.24.5. Row Constructor Comparison
9.24.6. Composite Type Comparison
9.25. Set Returning Functions
9.26. System Information Functions and Operators
9.27. System Administration Functions
9.27.1. Configuration Settings Functions
9.27.2. Server Signaling Functions
9.27.3. Backup Control Functions
9.27.4. Recovery Control Functions
9.27.5. Snapshot Synchronization Functions
9.27.6. Replication Management Functions
9.27.7. Database Object Management Functions
9.27.8. Index Maintenance Functions
9.27.9. Generic File Access Functions
9.27.10. Advisory Lock Functions
9.28. Trigger Functions
9.29. Event Trigger Functions
9.29.1. Capturing Changes at Command End
9.29.2. Processing Objects Dropped by a DDL Command
9.29.3. Handling a Table Rewrite Event
9.30. Statistics Information Functions
9.30.1. Inspecting MCV Lists

PostgreSQL provides a large number of functions and operators for the built-in data types. This chapter describes most of them, although additional special-purpose functions appear in relevant sections of the manual. Users can also define their own functions and operators, as described in Part V. The psql commands \df and \do can be used to list all available functions and operators, respectively.

The notation used throughout this chapter to describe the argument and result data types of a function or operator is like this:

repeat ( text, integer ) → text

which says that the function repeat takes one text and one integer argument and returns a result of type text. The right arrow is also used to indicate the result of an example, thus:

repeat('Pg', 4) → PgPgPgPg

If you are concerned about portability then note that most of the functions and operators described in this chapter, with the exception of the most trivial arithmetic and comparison operators and some explicitly marked functions, are not specified by the SQL standard. Some of this extended functionality is present in other SQL database management systems, and in many cases this functionality is compatible and consistent between the various implementations.

Prev | Up | Next
8.21. Pseudo-Types | Home | 9.1. Logical Operators


9.1. Logical Operators
Prev Up Chapter 9. Functions and Operators Home Next

9.1. Logical Operators

The usual logical operators are available:

boolean AND boolean → boolean
boolean OR boolean → boolean
NOT boolean → boolean

SQL uses a three-valued logic system with true, false, and null, which represents “unknown”. Observe the following truth tables:

a | b | a AND b | a OR b
TRUE | TRUE | TRUE | TRUE
TRUE | FALSE | FALSE | TRUE
TRUE | NULL | NULL | TRUE
FALSE | FALSE | FALSE | FALSE
FALSE | NULL | FALSE | NULL
NULL | NULL | NULL | NULL

a | NOT a
TRUE | FALSE
FALSE | TRUE
NULL | NULL

The operators AND and OR are commutative, that is, you can switch the left and right operands without affecting the result. (However, it is not guaranteed that the left operand is evaluated before the right operand. See Section 4.2.14 for more information about the order of evaluation of subexpressions.)

Prev | Up | Next
Chapter 9. Functions and Operators | Home | 9.2. Comparison Functions and Operators


9.2. Comparison Functions and Operators
Prev Up Chapter 9. Functions and Operators Home Next

9.2. Comparison Functions and Operators

The usual comparison operators are available, as shown in Table 9.1.

Table 9.1. Comparison Operators

Operator | Description
datatype < datatype → boolean | Less than
datatype > datatype → boolean | Greater than
datatype <= datatype → boolean | Less than or equal to
datatype >= datatype → boolean | Greater than or equal to
datatype = datatype → boolean | Equal
datatype <> datatype → boolean | Not equal
datatype != datatype → boolean | Not equal

Note

<> is the standard SQL notation for “not equal”. != is an alias, which is converted to <> at a very early stage of parsing. Hence, it is not possible to implement != and <> operators that do different things.

These comparison operators are available for all built-in data types that have a natural ordering, including numeric, string, and date/time types. In addition, arrays, composite types, and ranges can be compared if their component data types are comparable.

It is usually possible to compare values of related data types as well; for example integer > bigint will work. Some cases of this sort are implemented directly by “cross-type” comparison operators, but if no such operator is available, the parser will coerce the less-general type to the more-general type and apply the latter's comparison operator.

As shown above, all comparison operators are binary operators that return values of type boolean. Thus, expressions like 1 < 2 < 3 are not valid (because there is no < operator to compare a Boolean value with 3). Use the BETWEEN predicates shown below to perform range tests.

There are also some comparison predicates, as shown in Table 9.2. These behave much like operators, but have special syntax mandated by the SQL standard.

Table 9.2. Comparison Predicates

Predicate

Description

Example(s)

datatype BETWEEN datatype AND datatype → boolean

Between (inclusive of the range endpoints).

2 BETWEEN 1 AND 3 → t

2 BETWEEN 3 AND 1 → f

datatype NOT BETWEEN datatype AND datatype → boolean

Not between (the negation of BETWEEN).

2 NOT BETWEEN 1 AND 3 → f

datatype BETWEEN SYMMETRIC datatype AND datatype → boolean

Between, after sorting the two endpoint values.

2 BETWEEN SYMMETRIC 3 AND 1 → t

datatype NOT BETWEEN SYMMETRIC datatype AND datatype → boolean

Not between, after sorting the two endpoint values.

2 NOT BETWEEN SYMMETRIC 3 AND 1 → f

datatype IS DISTINCT FROM datatype → boolean

Not equal, treating null as a comparable value.

1 IS DISTINCT FROM NULL → t (rather than NULL)

NULL IS DISTINCT FROM NULL → f (rather than NULL)

datatype IS NOT DISTINCT FROM datatype → boolean

Equal, treating null as a comparable value.

1 IS NOT DISTINCT FROM NULL → f (rather than NULL)

NULL IS NOT DISTINCT FROM NULL → t (rather than NULL)

datatype IS NULL → boolean

Test whether value is null.

1.5 IS NULL → f

datatype IS NOT NULL → boolean

Test whether value is not null.

'null' IS NOT NULL → t

datatype ISNULL → boolean

Test whether value is null (nonstandard syntax).

datatype NOTNULL → boolean

Test whether value is not null (nonstandard syntax).

boolean IS TRUE → boolean

Test whether boolean expression yields true.

true IS TRUE → t

NULL::boolean IS TRUE → f (rather than NULL)

boolean IS NOT TRUE → boolean

Test whether boolean expression yields false or unknown.

true IS NOT TRUE → f

NULL::boolean IS NOT TRUE → t (rather than NULL)

boolean IS FALSE → boolean

Test whether boolean expression yields false.

true IS FALSE → f

NULL::boolean IS FALSE → f (rather than NULL)

boolean IS NOT FALSE → boolean

Test whether boolean expression yields true or unknown.

true IS NOT FALSE → t

NULL::boolean IS NOT FALSE → t (rather than NULL)

boolean IS UNKNOWN → boolean

Test whether boolean expression yields unknown.

true IS UNKNOWN → f

NULL::boolean IS UNKNOWN → t (rather than NULL)

boolean IS NOT UNKNOWN → boolean

Test whether boolean expression yields true or false.

true IS NOT UNKNOWN → t

NULL::boolean IS NOT UNKNOWN → f (rather than NULL)

The BETWEEN predicate simplifies range tests:

a BETWEEN x AND y

is equivalent to

a >= x AND a <= y

Notice that BETWEEN treats the endpoint values as included in the range. BETWEEN SYMMETRIC is like BETWEEN except there is no requirement that the argument to the left of AND be less than or equal to the argument on the right. If it is not, those two arguments are automatically swapped, so that a nonempty range is always implied.

The various variants of BETWEEN are implemented in terms of the ordinary comparison operators, and therefore will work for any data type(s) that can be compared.

Note

The use of AND in the BETWEEN syntax creates an ambiguity with the use of AND as a logical operator. To resolve this, only a limited set of expression types are allowed as the second argument of a BETWEEN clause. If you need to write a more complex sub-expression in BETWEEN, write parentheses around the sub-expression.

Ordinary comparison operators yield null (signifying “unknown”), not true or false, when either input is null. For example, 7 = NULL yields null, as does 7 <> NULL. When this behavior is not suitable, use the IS [ NOT ] DISTINCT FROM predicates:

a IS DISTINCT FROM b
a IS NOT DISTINCT FROM b

For non-null inputs, IS DISTINCT FROM is the same as the <> operator. However, if both inputs are null it returns false, and if only one input is null it returns true. Similarly, IS NOT DISTINCT FROM is identical to = for non-null inputs, but it returns true when both inputs are null, and false when only one input is null. Thus, these predicates effectively act as though null were a normal data value, rather than “unknown”.

To check whether a value is or is not null, use the predicates:

expression IS NULL
expression IS NOT NULL

or the equivalent, but nonstandard, predicates:

expression ISNULL
expression NOTNULL

Do not write expression = NULL because NULL is not “equal to” NULL. (The null value represents an unknown value, and it is not known whether two unknown values are equal.)

Tip

Some applications might expect that expression = NULL returns true if expression evaluates to the null value. It is highly recommended that these applications be modified to comply with the SQL standard. However, if that cannot be done the transform_null_equals configuration variable is available. If it is enabled, PostgreSQL will convert x = NULL clauses to x IS NULL.

If the expression is row-valued, then IS NULL is true when the row expression itself is null or when all the row's fields are null, while IS NOT NULL is true when the row expression itself is non-null and all the row's fields are non-null. Because of this behavior, IS NULL and IS NOT NULL do not always return inverse results for row-valued expressions; in particular, a row-valued expression that contains both null and non-null fields will return false for both tests. In some cases, it may be preferable to write row IS DISTINCT FROM NULL or row IS NOT DISTINCT FROM NULL, which will simply check whether the overall row value is null without any additional tests on the row fields.

Boolean values can also be tested using the predicates

boolean_expression IS TRUE
boolean_expression IS NOT TRUE
boolean_expression IS FALSE
boolean_expression IS NOT FALSE
boolean_expression IS UNKNOWN
boolean_expression IS NOT UNKNOWN

These will always return true or false, never a null value, even when the operand is null. A null input is treated as the logical value “unknown”. Notice that IS UNKNOWN and IS NOT UNKNOWN are effectively the same as IS NULL and IS NOT NULL, respectively, except that the input expression must be of Boolean type.

Some comparison-related functions are also available, as shown in Table 9.3.

Table 9.3. Comparison Functions

Function

Description

Example(s)

num_nonnulls ( VARIADIC "any" ) → integer

Returns the number of non-null arguments.

num_nonnulls(1, NULL, 2) → 2

num_nulls ( VARIADIC "any" ) → integer

Returns the number of null arguments.

num_nulls(1, NULL, 2) → 1

Prev | Up | Next
9.1. Logical Operators | Home | 9.3. Mathematical Functions and Operators


9.3. Mathematical Functions and Operators
Prev Up Chapter 9. Functions and Operators Home Next

9.3. Mathematical Functions and Operators

Mathematical operators are provided for many PostgreSQL types. For types without standard mathematical conventions (e.g., date/time types) we describe the actual behavior in subsequent sections.

Table 9.4 shows the mathematical operators that are available for the standard numeric types. Unless otherwise noted, operators shown as accepting numeric_type are available for all the types smallint, integer, bigint, numeric, real, and double precision. Operators shown as accepting integral_type are available for the types smallint, integer, and bigint. Except where noted, each form of an operator returns the same data type as its argument(s). Calls involving multiple argument data types, such as integer + numeric, are resolved by using the type appearing later in these lists.

Table 9.4. Mathematical Operators

Operator

Description

Example(s)

numeric_type + numeric_type → numeric_type

Addition

2 + 3 → 5

+ numeric_type → numeric_type

Unary plus (no operation)

+ 3.5 → 3.5

numeric_type - numeric_type → numeric_type

Subtraction

2 - 3 → -1

- numeric_type → numeric_type

Negation

- (-4) → 4

numeric_type * numeric_type → numeric_type

Multiplication

2 * 3 → 6

numeric_type / numeric_type → numeric_type

Division (for integral types, division truncates the result towards zero)

5.0 / 2 → 2.5000000000000000

5 / 2 → 2

(-5) / 2 → -2

numeric_type % numeric_type → numeric_type

Modulo (remainder); available for smallint, integer, bigint, and numeric

5 % 4 → 1

numeric ^ numeric → numeric

double precision ^ double precision → double precision

Exponentiation

2 ^ 3 → 8

Unlike typical mathematical practice, multiple uses of ^ will associate left to right by default:

2 ^ 3 ^ 3 → 512

2 ^ (3 ^ 3) → 134217728

|/ double precision → double precision

Square root

|/ 25.0 → 5

||/ double precision → double precision

Cube root

||/ 64.0 → 4

@ numeric_type → numeric_type

Absolute value

@ -5.0 → 5.0

integral_type & integral_type → integral_type

Bitwise AND

91 & 15 → 11

integral_type | integral_type → integral_type

Bitwise OR

32 | 3 → 35

integral_type # integral_type → integral_type

Bitwise exclusive OR

17 # 5 → 20

~ integral_type → integral_type

Bitwise NOT

~1 → -2

integral_type << integer → integral_type

Bitwise shift left

1 << 4 → 16

integral_type >> integer → integral_type

Bitwise shift right

8 >> 2 → 2

Table 9.5 shows the available mathematical functions. Many of these functions are provided in multiple forms with different argument types. Except where noted, any given form of a function returns the same data type as its argument(s); cross-type cases are resolved in the same way as explained above for operators. The functions working with double precision data are mostly implemented on top of the host system's C library; accuracy and behavior in boundary cases can therefore vary depending on the host system.

Table 9.5. Mathematical Functions

Function

Description

Example(s)

abs ( numeric_type ) → numeric_type

Absolute value

abs(-17.4) → 17.4

cbrt ( double precision ) → double precision

Cube root

cbrt(64.0) → 4

ceil ( numeric ) → numeric

ceil ( double precision ) → double precision

Nearest integer greater than or equal to argument

ceil(42.2) → 43

ceil(-42.8) → -42

ceiling ( numeric ) → numeric

ceiling ( double precision ) → double precision

Nearest integer greater than or equal to argument (same as ceil)

ceiling(95.3) → 96

degrees ( double precision ) → double precision

Converts radians to degrees

degrees(0.5) → 28.64788975654116

div ( y numeric, x numeric ) → numeric

Integer quotient of y/x (truncates towards zero)

div(9, 4) → 2

exp ( numeric ) → numeric

exp ( double precision ) → double precision

Exponential (e raised to the given power)

exp(1.0) → 2.7182818284590452

factorial ( bigint ) → numeric

Factorial

factorial(5) → 120

floor ( numeric ) → numeric

floor ( double precision ) → double precision

Nearest integer less than or equal to argument

floor(42.8) → 42

floor(-42.8) → -43

gcd ( numeric_type, numeric_type ) → numeric_type

Greatest common divisor (the largest positive number that divides both inputs with no remainder); returns 0 if both inputs are zero; available for integer, bigint, and numeric

gcd(1071, 462) → 21

lcm ( numeric_type, numeric_type ) → numeric_type

Least common multiple (the smallest strictly positive number that is an integral multiple of both inputs); returns 0 if either input is zero; available for integer, bigint, and numeric

lcm(1071, 462) → 23562

ln ( numeric ) → numeric

ln ( double precision ) → double precision

Natural logarithm

ln(2.0) → 0.6931471805599453

log ( numeric ) → numeric

log ( double precision ) → double precision

Base 10 logarithm

log(100) → 2

log10 ( numeric ) → numeric

log10 ( double precision ) → double precision

Base 10 logarithm (same as log)

log10(1000) → 3

log ( b numeric, x numeric ) → numeric

Logarithm of x to base b

log(2.0, 64.0) → 6.0000000000000000

min_scale ( numeric ) → integer

Minimum scale (number of fractional decimal digits) needed to represent the supplied value precisely

min_scale(8.4100) → 2

mod ( y numeric_type, x numeric_type ) → numeric_type

Remainder of y/x; available for smallint, integer, bigint, and numeric

mod(9, 4) → 1

pi ( ) → double precision

Approximate value of π

pi() → 3.141592653589793

power ( a numeric, b numeric ) → numeric

power ( a double precision, b double precision ) → double precision

a raised to the power of b

power(9, 3) → 729

radians ( double precision ) → double precision

Converts degrees to radians

radians(45.0) → 0.7853981633974483

round ( numeric ) → numeric

round ( double precision ) → double precision

Rounds to nearest integer. For numeric, ties are broken by rounding away from zero. For double precision, the tie-breaking behavior is platform dependent, but “round to nearest even” is the most common rule.

round(42.4) → 42

round ( v numeric, s integer ) → numeric

Rounds v to s decimal places. Ties are broken by rounding away from zero.

round(42.4382, 2) → 42.44

scale ( numeric ) → integer

Scale of the argument (the number of decimal digits in the fractional part)

scale(8.4100) → 4

sign ( numeric ) → numeric

sign ( double precision ) → double precision

Sign of the argument (-1, 0, or +1)

sign(-8.4) → -1

sqrt ( numeric ) → numeric

sqrt ( double precision ) → double precision

Square root

sqrt(2) → 1.4142135623730951

trim_scale ( numeric ) → numeric

Reduces the value's scale (number of fractional decimal digits) by removing trailing zeroes

trim_scale(8.4100) → 8.41

trunc ( numeric ) → numeric

trunc ( double precision ) → double precision

Truncates to integer (towards zero)

trunc(42.8) → 42

trunc(-42.8) → -42

trunc ( v numeric, s integer ) → numeric

Truncates v to s decimal places

trunc(42.4382, 2) → 42.43

width_bucket ( operand numeric, low numeric, high numeric, count integer ) → integer

width_bucket ( operand double precision, low double precision, high double precision, count integer ) → integer

Returns the number of the bucket in which operand falls in a histogram having count equal-width buckets spanning the range low to high. Returns 0 or count+1 for an input outside that range.

width_bucket(5.35, 0.024, 10.06, 5) → 3

width_bucket ( operand anycompatible, thresholds anycompatiblearray ) → integer

Returns the number of the bucket in which operand falls given an array listing the lower bounds of the buckets. Returns 0 for an input less than the first lower bound. operand and the array elements can be of any type having standard comparison operators. The thresholds array must be sorted, smallest first, or unexpected results will be obtained.

width_bucket(now(), array['yesterday', 'today', 'tomorrow']::timestamptz[]) → 2

Table 9.6 shows functions for generating random numbers.

Table 9.6. Random Functions

Function

Description

Example(s)

random ( ) → double precision

Returns a random value in the range 0.0 <= x < 1.0

random() → 0.897124072839091

setseed ( double precision ) → void

Sets the seed for subsequent random() calls; argument must be between -1.0 and 1.0, inclusive

setseed(0.12345)

The random() function uses a simple linear congruential algorithm. It is fast but not suitable for cryptographic applications; see the pgcrypto module for a more secure alternative. If setseed() is called, the series of results of subsequent random() calls in the current session can be repeated by re-issuing setseed() with the same argument.

Table 9.7 shows the available trigonometric functions. Each of these functions comes in two variants, one that measures angles in radians and one that measures angles in degrees.

Table 9.7. Trigonometric Functions

Function

Description

Example(s)

acos ( double precision ) → double precision

Inverse cosine, result in radians

acos(1) → 0

acosd ( double precision ) → double precision

Inverse cosine, result in degrees

acosd(0.5) → 60

asin ( double precision ) → double precision

Inverse sine, result in radians

asin(1) → 1.5707963267948966

asind ( double precision ) → double precision

Inverse sine, result in degrees

asind(0.5) → 30

atan ( double precision ) → double precision

Inverse tangent, result in radians

atan(1) → 0.7853981633974483

atand ( double precision ) → double precision

Inverse tangent, result in degrees

atand(1) → 45

atan2 ( y double precision, x double precision ) → double precision

Inverse tangent of y/x, result in radians

atan2(1, 0) → 1.5707963267948966

atan2d ( y double precision, x double precision ) → double precision

Inverse tangent of y/x, result in degrees

atan2d(1, 0) → 90

cos ( double precision ) → double precision

Cosine, argument in radians

cos(0) → 1

cosd ( double precision ) → double precision

Cosine, argument in degrees

cosd(60) → 0.5

cot ( double precision ) → double precision

Cotangent, argument in radians

cot(0.5) → 1.830487721712452

cotd ( double precision ) → double precision

Cotangent, argument in degrees

cotd(45) → 1

sin ( double precision ) → double precision

Sine, argument in radians

sin(1) → 0.8414709848078965

sind ( double precision ) → double precision

Sine, argument in degrees

sind(30) → 0.5

tan ( double precision ) → double precision

Tangent, argument in radians

tan(1) → 1.5574077246549023

tand ( double precision ) → double precision

Tangent, argument in degrees

tand(45) → 1

Note

Another way to work with angles measured in degrees is to use the unit transformation functions radians() and degrees() shown earlier. However, using the degree-based trigonometric functions is preferred, as that way avoids round-off error for special cases such as sind(30).

Table 9.8 shows the available hyperbolic functions.

Table 9.8. Hyperbolic Functions

Function

Description

Example(s)

sinh ( double precision ) → double precision

Hyperbolic sine

sinh(1) → 1.1752011936438014

cosh ( double precision ) → double precision

Hyperbolic cosine

cosh(0) → 1

tanh ( double precision ) → double precision

Hyperbolic tangent

tanh(1) → 0.7615941559557649

asinh ( double precision ) → double precision

Inverse hyperbolic sine

asinh(1) → 0.881373587019543

acosh ( double precision ) → double precision

Inverse hyperbolic cosine

acosh(1) → 0

atanh ( double precision ) → double precision

Inverse hyperbolic tangent

atanh(0.5) → 0.5493061443340548

Prev | Up | Next
9.2. Comparison Functions and Operators | Home | 9.4. String Functions and Operators


9.4. String Functions and Operators
Prev Up Chapter 9. Functions and Operators Home Next

9.4. String Functions and Operators

9.4.1. format

This section describes functions and operators for examining and manipulating string values. Strings in this context include values of the types character, character varying, and text. Except where noted, these functions and operators are declared to accept and return type text. They will interchangeably accept character varying arguments. Values of type character will be converted to text before the function or operator is applied, resulting in stripping any trailing spaces in the character value.

SQL defines some string functions that use key words, rather than commas, to separate arguments. Details are in Table 9.9. PostgreSQL also provides versions of these functions that use the regular function invocation syntax (see Table 9.10).

Note

The string concatenation operator (||) will accept non-string input, so long as at least one input is of string type, as shown in Table 9.9. For other cases, inserting an explicit coercion to text can be used to have non-string input accepted.

Table 9.9. SQL String Functions and Operators

Function/Operator

Description

Example(s)

text || text → text

Concatenates the two strings.

'Post' || 'greSQL' → PostgreSQL

text || anynonarray → text

anynonarray || text → text

Converts the non-string input to text, then concatenates the two strings. (The non-string input cannot be of an array type, because that would create ambiguity with the array || operators. If you want to concatenate an array's text equivalent, cast it to text explicitly.)

'Value: ' || 42 → Value: 42

text IS [NOT] [form] NORMALIZED → boolean

Checks whether the string is in the specified Unicode normalization form. The optional form key word specifies the form: NFC (the default), NFD, NFKC, or NFKD. This expression can only be used when the server encoding is UTF8. Note that checking for normalization using this expression is often faster than normalizing possibly already normalized strings.

U&'\0061\0308bc' IS NFD NORMALIZED → t

bit_length ( text ) → integer

Returns number of bits in the string (8 times the octet_length).

bit_length('jose') → 32

char_length ( text ) → integer

character_length ( text ) → integer

Returns number of characters in the string.

char_length('josé') → 4

lower ( text ) → text

Converts the string to all lower case, according to the rules of the database's locale.

lower('TOM') → tom

normalize ( text [, form ] ) → text

Converts the string to the specified Unicode normalization form. The optional form key word specifies the form: NFC (the default), NFD, NFKC, or NFKD. This function can only be used when the server encoding is UTF8.

normalize(U&'\0061\0308bc', NFC) → U&'\00E4bc'

octet_length ( text ) → integer

Returns number of bytes in the string.

octet_length('josé') → 5 (if server encoding is UTF8)

octet_length ( character ) → integer

Returns number of bytes in the string. Since this version of the function accepts type character directly, it will not strip trailing spaces.

octet_length('abc '::character(4)) → 4

overlay ( string text PLACING newsubstring text FROM start integer [ FOR count integer ] ) → text

Replaces the substring of string that starts at the start'th character and extends for count characters with newsubstring. If count is omitted, it defaults to the length of newsubstring.

overlay('Txxxxas' placing 'hom' from 2 for 4) → Thomas

position ( substring text IN string text ) → integer

Returns first starting index of the specified substring within string, or zero if it's not present.

position('om' in 'Thomas') → 3

substring ( string text [ FROM start integer ] [ FOR count integer ] ) → text

Extracts the substring of string starting at the start'th character if that is specified, and stopping after count characters if that is specified. Provide at least one of start and count.

substring('Thomas' from 2 for 3) → hom

substring('Thomas' from 3) → omas

substring('Thomas' for 2) → Th

substring ( string text FROM pattern text ) → text

Extracts the first substring matching POSIX regular expression; see Section 9.7.3.

substring('Thomas' from '...$') → mas

substring ( string text SIMILAR pattern text ESCAPE escape text ) → text

substring ( string text FROM pattern text FOR escape text ) → text

Extracts the first substring matching SQL regular expression; see Section 9.7.2. The first form has been specified since SQL:2003; the second form was only in SQL:1999 and should be considered obsolete.

substring('Thomas' similar '%#"o_a#"_' escape '#') → oma

trim ( [ LEADING | TRAILING | BOTH ] [ characters text ] FROM string text ) → text

Removes the longest string containing only characters in characters (a space by default) from the start, end, or both ends (BOTH is the default) of string.

trim(both 'xyz' from 'yxTomxx') → Tom

trim ( [ LEADING | TRAILING | BOTH ] [ FROM ] string text [, characters text ] ) → text

This is a non-standard syntax for trim().

trim(both from 'yxTomxx', 'xyz') → Tom

upper ( text ) → text

Converts the string to all upper case, according to the rules of the database's locale.

upper('tom') → TOM

Additional string manipulation functions are available and are listed in Table 9.10. Some of them are used internally to implement the SQL-standard string functions listed in Table 9.9.

Table 9.10. Other String Functions

Function

Description

Example(s)

ascii ( text ) → integer

Returns the numeric code of the first character of the argument. In UTF8 encoding, returns the Unicode code point of the character. In other multibyte encodings, the argument must be an ASCII character.

ascii('x') → 120

btrim ( string text [, characters text ] ) → text

Removes the longest string containing only characters in characters (a space by default) from the start and end of string.

btrim('xyxtrimyyx', 'xyz') → trim

chr ( integer ) → text

Returns the character with the given code. In UTF8 encoding the argument is treated as a Unicode code point. In other multibyte encodings the argument must designate an ASCII character. chr(0) is disallowed because text data types cannot store that character.

chr(65) → A

concat ( val1 "any" [, val2 "any" [, ...] ] ) → text

Concatenates the text representations of all the arguments. NULL arguments are ignored.

concat('abcde', 2, NULL, 22) → abcde222

concat_ws ( sep text, val1 "any" [, val2 "any" [, ...] ] ) → text

Concatenates all but the first argument, with separators. The first argument is used as the separator string, and should not be NULL. Other NULL arguments are ignored.

concat_ws(',', 'abcde', 2, NULL, 22) → abcde,2,22

format ( formatstr text [, formatarg "any" [, ...] ] ) → text

Formats arguments according to a format string; see Section 9.4.1. This function is similar to the C function sprintf.

format('Hello %s, %1$s', 'World') → Hello World, World

initcap ( text ) → text

Converts the first letter of each word to upper case and the rest to lower case. Words are sequences of alphanumeric characters separated by non-alphanumeric characters.

initcap('hi THOMAS') → Hi Thomas

left ( string text, n integer ) → text

Returns first n characters in the string, or when n is negative, returns all but last |n| characters.

left('abcde', 2) → ab

length ( text ) → integer

Returns the number of characters in the string.

length('jose') → 4

lpad ( string text, length integer [, fill text ] ) → text

Extends the string to length length by prepending the characters fill (a space by default). If the string is already longer than length then it is truncated (on the right).

lpad('hi', 5, 'xy') → xyxhi

ltrim ( string text [, characters text ] ) → text

Removes the longest string containing only characters in characters (a space by default) from the start of string.

ltrim('zzzytest', 'xyz') → test

md5 ( text ) → text

Computes the MD5 hash of the argument, with the result written in hexadecimal.

md5('abc') → 900150983cd24fb0​d6963f7d28e17f72

parse_ident ( qualified_identifier text [, strict_mode boolean DEFAULT true ] ) → text[]

Splits qualified_identifier into an array of identifiers, removing any quoting of individual identifiers. By default, extra characters after the last identifier are considered an error; but if the second parameter is false, then such extra characters are ignored. (This behavior is useful for parsing names for objects like functions.) Note that this function does not truncate over-length identifiers. If you want truncation you can cast the result to name[].

parse_ident('"SomeSchema".someTable') → {SomeSchema,sometable}

pg_client_encoding ( ) → name

Returns current client encoding name.

pg_client_encoding() → UTF8

quote_ident ( text ) → text

Returns the given string suitably quoted to be used as an identifier in an SQL statement string. Quotes are added only if necessary (i.e., if the string contains non-identifier characters or would be case-folded). Embedded quotes are properly doubled. See also Example 43.1.

quote_ident('Foo bar') → "Foo bar"

quote_literal ( text ) → text

Returns the given string suitably quoted to be used as a string literal in an SQL statement string. Embedded single-quotes and backslashes are properly doubled. Note that quote_literal returns null on null input; if the argument might be null, quote_nullable is often more suitable. See also Example 43.1.

quote_literal(E'O\'Reilly') → 'O''Reilly'

quote_literal ( anyelement ) → text

Converts the given value to text and then quotes it as a literal. Embedded single-quotes and backslashes are properly doubled.

quote_literal(42.5) → '42.5'

quote_nullable ( text ) → text

Returns the given string suitably quoted to be used as a string literal in an SQL statement string; or, if the argument is null, returns NULL. Embedded single-quotes and backslashes are properly doubled. See also Example 43.1.

quote_nullable(NULL) → NULL

quote_nullable ( anyelement ) → text

Converts the given value to text and then quotes it as a literal; or, if the argument is null, returns NULL. Embedded single-quotes and backslashes are properly doubled.

quote_nullable(42.5) → '42.5'

regexp_match ( string text, pattern text [, flags text ] ) → text[]

Returns captured substrings resulting from the first match of a POSIX regular expression to the string; see Section 9.7.3.

regexp_match('foobarbequebaz', '(bar)(beque)') → {bar,beque}

regexp_matches ( string text, pattern text [, flags text ] ) → setof text[]

Returns captured substrings resulting from the first match of a POSIX regular expression to the string, or multiple matches if the g flag is used; see Section 9.7.3.

regexp_matches('foobarbequebaz', 'ba.', 'g') →

 {bar}
 {baz}

regexp_replace ( string text, pattern text, replacement text [, flags text ] ) → text

Replaces substrings resulting from the first match of a POSIX regular expression, or multiple substring matches if the g flag is used; see Section 9.7.3.

regexp_replace('Thomas', '.[mN]a.', 'M') → ThM

regexp_split_to_array ( string text, pattern text [, flags text ] ) → text[]

Splits string using a POSIX regular expression as the delimiter, producing an array of results; see Section 9.7.3.

regexp_split_to_array('hello world', '\s+') → {hello,world}

regexp_split_to_table ( string text, pattern text [, flags text ] ) → setof text

Splits string using a POSIX regular expression as the delimiter, producing a set of results; see Section 9.7.3.

regexp_split_to_table('hello world', '\s+') →

 hello
 world

repeat ( string text, number integer ) → text

Repeats string the specified number of times.

repeat('Pg', 4) → PgPgPgPg

replace ( string text, from text, to text ) → text

Replaces all occurrences in string of substring from with substring to.

replace('abcdefabcdef', 'cd', 'XX') → abXXefabXXef

reverse ( text ) → text

Reverses the order of the characters in the string.

reverse('abcde') → edcba

right ( string text, n integer ) → text

Returns last n characters in the string, or when n is negative, returns all but first |n| characters.

right('abcde', 2) → de

rpad ( string text, length integer [, fill text ] ) → text

Extends the string to length length by appending the characters fill (a space by default). If the string is already longer than length then it is truncated.

rpad('hi', 5, 'xy') → hixyx

rtrim ( string text [, characters text ] ) → text

Removes the longest string containing only characters in characters (a space by default) from the end of string.

rtrim('testxxzx', 'xyz') → test

split_part ( string text, delimiter text, n integer ) → text

Splits string at occurrences of delimiter and returns the n'th field (counting from one), or when n is negative, returns the |n|'th-from-last field.

split_part('abc~@~def~@~ghi', '~@~', 2) → def

split_part('abc,def,ghi,jkl', ',', -2) → ghi

strpos ( string text, substring text ) → integer

Returns first starting index of the specified substring within string, or zero if it's not present. (Same as position(substring in string), but note the reversed argument order.)

strpos('high', 'ig') → 2

substr ( string text, start integer [, count integer ] ) → text

Extracts the substring of string starting at the start'th character, and extending for count characters if that is specified. (Same as substring(string from start for count).)

substr('alphabet', 3) → phabet

substr('alphabet', 3, 2) → ph

starts_with ( string text, prefix text ) → boolean

Returns true if string starts with prefix.

starts_with('alphabet', 'alph') → t

string_to_array ( string text, delimiter text [, null_string text ] ) → text[]

Splits the string at occurrences of delimiter and forms the resulting fields into a text array. If delimiter is NULL, each character in the string will become a separate element in the array. If delimiter is an empty string, then the string is treated as a single field. If null_string is supplied and is not NULL, fields matching that string are replaced by NULL.

string_to_array('xx~~yy~~zz', '~~', 'yy') → {xx,NULL,zz}

string_to_table ( string text, delimiter text [, null_string text ] ) → setof text

Splits the string at occurrences of delimiter and returns the resulting fields as a set of text rows. If delimiter is NULL, each character in the string will become a separate row of the result. If delimiter is an empty string, then the string is treated as a single field. If null_string is supplied and is not NULL, fields matching that string are replaced by NULL.

string_to_table('xx~^~yy~^~zz', '~^~', 'yy') →

 xx
 NULL
 zz

to_ascii ( string text ) → text

to_ascii ( string text, encoding name ) → text

to_ascii ( string text, encoding integer ) → text

Converts string to ASCII from another encoding, which may be identified by name or number. If encoding is omitted the database encoding is assumed (which in practice is the only useful case). The conversion consists primarily of dropping accents. Conversion is only supported from LATIN1, LATIN2, LATIN9, and WIN1250 encodings. (See the unaccent module for another, more flexible solution.)

to_ascii('Karél') → Karel

to_hex ( integer ) → text

to_hex ( bigint ) → text

Converts the number to its equivalent hexadecimal representation.

to_hex(2147483647) → 7fffffff

translate ( string text, from text, to text ) → text

Replaces each character in string that matches a character in the from set with the corresponding character in the to set. If from is longer than to, occurrences of the extra characters in from are deleted.

translate('12345', '143', 'ax') → a2x5

unistr ( text ) → text

Evaluate escaped Unicode characters in the argument. Unicode characters can be specified as \XXXX (4 hexadecimal digits), \+XXXXXX (6 hexadecimal digits), \uXXXX (4 hexadecimal digits), or \UXXXXXXXX (8 hexadecimal digits). To specify a backslash, write two backslashes. All other characters are taken literally.

If the server encoding is not UTF-8, the Unicode code point identified by one of these escape sequences is converted to the actual server encoding; an error is reported if that's not possible.

This function provides a (non-standard) alternative to string constants with Unicode escapes (see Section 4.1.2.3).

unistr('d\0061t\+000061') → data

unistr('d\u0061t\U00000061') → data

The concat, concat_ws and format functions are variadic, so it is possible to pass the values to be concatenated or formatted as an array marked with the VARIADIC keyword (see Section 38.5.6). The array's elements are treated as if they were separate ordinary arguments to the function. If the variadic array argument is NULL, concat and concat_ws return NULL, but format treats a NULL as a zero-element array.

See also the aggregate function string_agg in Section 9.21, and the functions for converting between strings and the bytea type in Table 9.13.

9.4.1. format

The function format produces output formatted according to a format string, in a style similar to the C function sprintf.

format(formatstr text [, formatarg "any" [, ...] ])

formatstr is a format string that specifies how the result should be formatted. Text in the format string is copied directly to the result, except where format specifiers are used. Format specifiers act as placeholders in the string, defining how subsequent function arguments should be formatted and inserted into the result. Each formatarg argument is converted to text according to the usual output rules for its data type, and then formatted and inserted into the result string according to the format specifier(s).

Format specifiers are introduced by a % character and have the form

%[position][flags][width]type

where the component fields are:

position (optional)

A string of the form n$ where n is the index of the argument to print. Index 1 means the first argument after formatstr. If the position is omitted, the default is to use the next argument in sequence.

flags (optional)

Additional options controlling how the format specifier's output is formatted. Currently the only supported flag is a minus sign (-) which will cause the format specifier's output to be left-justified. This has no effect unless the width field is also specified.

width (optional)

Specifies the minimum number of characters to use to display the format specifier's output. The output is padded on the left or right (depending on the - flag) with spaces as needed to fill the width. A too-small width does not cause truncation of the output, but is simply ignored. The width may be specified using any of the following: a positive integer; an asterisk (*) to use the next function argument as the width; or a string of the form *n$ to use the nth function argument as the width.

If the width comes from a function argument, that argument is consumed before the argument that is used for the format specifier's value. If the width argument is negative, the result is left aligned (as if the - flag had been specified) within a field of length abs(width).

type (required)

The type of format conversion to use to produce the format specifier's output. The following types are supported:

  • s formats the argument value as a simple string. A null value is treated as an empty string.

  • I treats the argument value as an SQL identifier, double-quoting it if necessary. It is an error for the value to be null (equivalent to quote_ident).

  • L quotes the argument value as an SQL literal. A null value is displayed as the string NULL, without quotes (equivalent to quote_nullable).

In addition to the format specifiers described above, the special sequence %% may be used to output a literal % character.

Here are some examples of the basic format conversions:

SELECT format('Hello %s', 'World');
Result: Hello World

SELECT format('Testing %s, %s, %s, %%', 'one', 'two', 'three');
Result: Testing one, two, three, %

SELECT format('INSERT INTO %I VALUES(%L)', 'Foo bar', E'O\'Reilly');
Result: INSERT INTO "Foo bar" VALUES('O''Reilly')

SELECT format('INSERT INTO %I VALUES(%L)', 'locations', 'C:\Program Files');
Result: INSERT INTO locations VALUES('C:\Program Files')

Here are examples using width fields and the - flag:

SELECT format('|%10s|', 'foo');
Result: |       foo|

SELECT format('|%-10s|', 'foo');
Result: |foo       |

SELECT format('|%*s|', 10, 'foo');
Result: |       foo|

SELECT format('|%*s|', -10, 'foo');
Result: |foo       |

SELECT format('|%-*s|', 10, 'foo');
Result: |foo       |

SELECT format('|%-*s|', -10, 'foo');
Result: |foo       |

These examples show use of position fields:

SELECT format('Testing %3$s, %2$s, %1$s', 'one', 'two', 'three');
Result: Testing three, two, one

SELECT format('|%*2$s|', 'foo', 10, 'bar');
Result: |       bar|

SELECT format('|%1$*2$s|', 'foo', 10, 'bar');
Result: |       foo|

Unlike the standard C function sprintf, PostgreSQL's format function allows format specifiers with and without position fields to be mixed in the same format string. A format specifier without a position field always uses the next argument after the last argument consumed. In addition, the format function does not require all function arguments to be used in the format string. For example:

SELECT format('Testing %3$s, %2$s, %s', 'one', 'two', 'three');
Result: Testing three, two, three

The %I and %L format specifiers are particularly useful for safely constructing dynamic SQL statements. See Example 43.1.

Prev | Up | Next
9.3. Mathematical Functions and Operators | Home | 9.5. Binary String Functions and Operators


9.5. Binary String Functions and Operators
Prev Up Chapter 9. Functions and Operators Home Next

9.5. Binary String Functions and Operators

This section describes functions and operators for examining and manipulating binary strings, that is values of type bytea. Many of these are equivalent, in purpose and syntax, to the text-string functions described in the previous section.

SQL defines some string functions that use key words, rather than commas, to separate arguments. Details are in Table 9.11. PostgreSQL also provides versions of these functions that use the regular function invocation syntax (see Table 9.12).

Table 9.11. SQL Binary String Functions and Operators

Function/Operator

Description

Example(s)

bytea || bytea → bytea

Concatenates the two binary strings.

'\x123456'::bytea || '\x789a00bcde'::bytea → \x123456789a00bcde

bit_length ( bytea ) → integer

Returns number of bits in the binary string (8 times the octet_length).

bit_length('\x123456'::bytea) → 24

octet_length ( bytea ) → integer

Returns number of bytes in the binary string.

octet_length('\x123456'::bytea) → 3

overlay ( bytes bytea PLACING newsubstring bytea FROM start integer [ FOR count integer ] ) → bytea

Replaces the substring of bytes that starts at the start'th byte and extends for count bytes with newsubstring. If count is omitted, it defaults to the length of newsubstring.

overlay('\x1234567890'::bytea placing '\002\003'::bytea from 2 for 3) → \x12020390

position ( substring bytea IN bytes bytea ) → integer

Returns first starting index of the specified substring within bytes, or zero if it's not present.

position('\x5678'::bytea in '\x1234567890'::bytea) → 3

substring ( bytes bytea [ FROM start integer ] [ FOR count integer ] ) → bytea

Extracts the substring of bytes starting at the start'th byte if that is specified, and stopping after count bytes if that is specified. Provide at least one of start and count.

substring('\x1234567890'::bytea from 3 for 2) → \x5678

trim ( [ LEADING | TRAILING | BOTH ] bytesremoved bytea FROM bytes bytea ) → bytea

Removes the longest string containing only bytes appearing in bytesremoved from the start, end, or both ends (BOTH is the default) of bytes.

trim('\x9012'::bytea from '\x1234567890'::bytea) → \x345678

trim ( [ LEADING | TRAILING | BOTH ] [ FROM ] bytes bytea, bytesremoved bytea ) → bytea

This is a non-standard syntax for trim().

trim(both from '\x1234567890'::bytea, '\x9012'::bytea) → \x345678

Additional binary string manipulation functions are available and are listed in Table 9.12. Some of them are used internally to implement the SQL-standard string functions listed in Table 9.11.

Table 9.12. Other Binary String Functions

Function

Description

Example(s)

bit_count ( bytes bytea ) → bigint

Returns the number of bits set in the binary string (also known as “popcount”).

bit_count('\x1234567890'::bytea) → 15

btrim ( bytes bytea, bytesremoved bytea ) → bytea

Removes the longest string containing only bytes appearing in bytesremoved from the start and end of bytes.

btrim('\x1234567890'::bytea, '\x9012'::bytea) → \x345678

get_bit ( bytes bytea, n bigint ) → integer

Extracts n'th bit from binary string.

get_bit('\x1234567890'::bytea, 30) → 1

get_byte ( bytes bytea, n integer ) → integer

Extracts n'th byte from binary string.

get_byte('\x1234567890'::bytea, 4) → 144

length ( bytea ) → integer

Returns the number of bytes in the binary string.

length('\x1234567890'::bytea) → 5

length ( bytes bytea, encoding name ) → integer

Returns the number of characters in the binary string, assuming that it is text in the given encoding.

length('jose'::bytea, 'UTF8') → 4

ltrim ( bytes bytea, bytesremoved bytea ) → bytea

Removes the longest string containing only bytes appearing in bytesremoved from the start of bytes.

ltrim('\x1234567890'::bytea, '\x9012'::bytea) → \x34567890

md5 ( bytea ) → text

Computes the MD5 hash of the binary string, with the result written in hexadecimal.

md5('Th\000omas'::bytea) → 8ab2d3c9689aaf18​b4958c334c82d8b1

rtrim ( bytes bytea, bytesremoved bytea ) → bytea

Removes the longest string containing only bytes appearing in bytesremoved from the end of bytes.

rtrim('\x1234567890'::bytea, '\x9012'::bytea) → \x12345678

set_bit ( bytes bytea, n bigint, newvalue integer ) → bytea

Sets n'th bit in binary string to newvalue.

set_bit('\x1234567890'::bytea, 30, 0) → \x1234563890

set_byte ( bytes bytea, n integer, newvalue integer ) → bytea

Sets n'th byte in binary string to newvalue.

set_byte('\x1234567890'::bytea, 4, 64) → \x1234567840

sha224 ( bytea ) → bytea

Computes the SHA-224 hash of the binary string.

sha224('abc'::bytea) → \x23097d223405d8228642a477bda2​55b32aadbce4bda0b3f7e36c9da7

sha256 ( bytea ) → bytea

Computes the SHA-256 hash of the binary string.

sha256('abc'::bytea) → \xba7816bf8f01cfea414140de5dae2223​b00361a396177a9cb410ff61f20015ad

sha384 ( bytea ) → bytea

Computes the SHA-384 hash of the binary string.

sha384('abc'::bytea) → \xcb00753f45a35e8bb5a03d699ac65007​272c32ab0eded1631a8b605a43ff5bed​8086072ba1e7cc2358baeca134c825a7

sha512 ( bytea ) → bytea

Computes the SHA-512 hash of the binary string.

sha512('abc'::bytea) → \xddaf35a193617abacc417349ae204131​12e6fa4e89a97ea20a9eeee64b55d39a​2192992a274fc1a836ba3c23a3feebbd​454d4423643ce80e2a9ac94fa54ca49f

substr ( bytes bytea, start integer [, count integer ] ) → bytea

Extracts the substring of bytes starting at the start'th byte, and extending for count bytes if that is specified. (Same as substring(bytes from start for count).)

substr('\x1234567890'::bytea, 3, 2) → \x5678

Functions get_byte and set_byte number the first byte of a binary string as byte 0. Functions get_bit and set_bit number bits from the right within each byte; for example bit 0 is the least significant bit of the first byte, and bit 15 is the most significant bit of the second byte.

For historical reasons, the function md5 returns a hex-encoded value of type text whereas the SHA-2 functions return type bytea. Use the functions encode and decode to convert between the two. For example write encode(sha256('abc'), 'hex') to get a hex-encoded text representation, or decode(md5('abc'), 'hex') to get a bytea value.

Functions for converting strings between different character sets (encodings), and for representing arbitrary binary data in textual form, are shown in Table 9.13. For these functions, an argument or result of type text is expressed in the database's default encoding, while arguments or results of type bytea are in an encoding named by another argument.

Table 9.13. Text/Binary String Conversion Functions

Function

Description

Example(s)

convert ( bytes bytea, src_encoding name, dest_encoding name ) → bytea

Converts a binary string representing text in encoding src_encoding to a binary string in encoding dest_encoding (see Section 24.3.4 for available conversions).

convert('text_in_utf8', 'UTF8', 'LATIN1') → \x746578745f696e5f75746638

convert_from ( bytes bytea, src_encoding name ) → text

Converts a binary string representing text in encoding src_encoding to text in the database encoding (see Section 24.3.4 for available conversions).

convert_from('text_in_utf8', 'UTF8') → text_in_utf8

convert_to ( string text, dest_encoding name ) → bytea

Converts a text string (in the database encoding) to a binary string encoded in encoding dest_encoding (see Section 24.3.4 for available conversions).

convert_to('some_text', 'UTF8') → \x736f6d655f74657874

encode ( bytes bytea, format text ) → text

Encodes binary data into a textual representation; supported format values are: base64, escape, hex.

encode('123\000\001', 'base64') → MTIzAAE=

decode ( string text, format text ) → bytea

Decodes binary data from a textual representation; supported format values are the same as for encode.

decode('MTIzAAE=', 'base64') → \x3132330001

The encode and decode functions support the following textual formats:

base64

The base64 format is that of RFC 2045 Section 6.8. As per the RFC, encoded lines are broken at 76 characters. However instead of the MIME CRLF end-of-line marker, only a newline is used for end-of-line. The decode function ignores carriage-return, newline, space, and tab characters. Otherwise, an error is raised when decode is supplied invalid base64 data — including when trailing padding is incorrect.

escape

The escape format converts zero bytes and bytes with the high bit set into octal escape sequences (\nnn), and it doubles backslashes. Other byte values are represented literally. The decode function will raise an error if a backslash is not followed by either a second backslash or three octal digits; it accepts other byte values unchanged.

hex

The hex format represents each 4 bits of data as one hexadecimal digit, 0 through f, writing the higher-order digit of each byte first. The encode function outputs the a-f hex digits in lower case. Because the smallest unit of data is 8 bits, there are always an even number of characters returned by encode. The decode function accepts the a-f characters in either upper or lower case. An error is raised when decode is given invalid hex data — including when given an odd number of characters.

See also the aggregate function string_agg in Section 9.21 and the large object functions in Section 35.4.

Prev | Up | Next
9.4. String Functions and Operators | Home | 9.6. Bit String Functions and Operators


9.6. Bit String Functions and Operators
Prev Up Chapter 9. Functions and Operators Home Next

9.6. Bit String Functions and Operators

This section describes functions and operators for examining and manipulating bit strings, that is values of the types bit and bit varying. (While only type bit is mentioned in these tables, values of type bit varying can be used interchangeably.) Bit strings support the usual comparison operators shown in Table 9.1, as well as the operators shown in Table 9.14.

Table 9.14. Bit String Operators

Operator

Description

Example(s)

bit || bit → bit

Concatenation

B'10001' || B'011' → 10001011

bit & bit → bit

Bitwise AND (inputs must be of equal length)

B'10001' & B'01101' → 00001

bit | bit → bit

Bitwise OR (inputs must be of equal length)

B'10001' | B'01101' → 11101

bit # bit → bit

Bitwise exclusive OR (inputs must be of equal length)

B'10001' # B'01101' → 11100

~ bit → bit

Bitwise NOT

~ B'10001' → 01110

bit << integer → bit

Bitwise shift left (string length is preserved)

B'10001' << 3 → 01000

bit >> integer → bit

Bitwise shift right (string length is preserved)

B'10001' >> 2 → 00100

Some of the functions available for binary strings are also available for bit strings, as shown in Table 9.15.

Table 9.15. Bit String Functions

Function

Description

Example(s)

bit_count ( bit ) → bigint

Returns the number of bits set in the bit string (also known as “popcount”).

bit_count(B'10111') → 4

bit_length ( bit ) → integer

Returns number of bits in the bit string.

bit_length(B'10111') → 5

length ( bit ) → integer

Returns number of bits in the bit string.

length(B'10111') → 5

octet_length ( bit ) → integer

Returns number of bytes in the bit string.

octet_length(B'1011111011') → 2

overlay ( bits bit PLACING newsubstring bit FROM start integer [ FOR count integer ] ) → bit

Replaces the substring of bits that starts at the start'th bit and extends for count bits with newsubstring. If count is omitted, it defaults to the length of newsubstring.

overlay(B'01010101010101010' placing B'11111' from 2 for 3) → 0111110101010101010

position ( substring bit IN bits bit ) → integer

Returns first starting index of the specified substring within bits, or zero if it's not present.

position(B'010' in B'000001101011') → 8

substring ( bits bit [ FROM start integer ] [ FOR count integer ] ) → bit

Extracts the substring of bits starting at the start'th bit if that is specified, and stopping after count bits if that is specified. Provide at least one of start and count.

substring(B'110010111111' from 3 for 2) → 00

get_bit ( bits bit, n integer ) → integer

Extracts n'th bit from bit string; the first (leftmost) bit is bit 0.

get_bit(B'101010101010101010', 6) → 1

set_bit ( bits bit, n integer, newvalue integer ) → bit

Sets n'th bit in bit string to newvalue; the first (leftmost) bit is bit 0.

set_bit(B'101010101010101010', 6, 0) → 101010001010101010

In addition, it is possible to cast integral values to and from type bit. Casting an integer to bit(n) copies the rightmost n bits. Casting an integer to a bit string width wider than the integer itself will sign-extend on the left. Some examples:

44::bit(10)                    0000101100
44::bit(3)                     100
cast(-44 as bit(12))           111111010100
'1110'::bit(4)::integer        14

Note that casting to just “bit” means casting to bit(1), and so will deliver only the least significant bit of the integer.

Prev | Up | Next
9.5. Binary String Functions and Operators | Home | 9.7. Pattern Matching


9.7. Pattern Matching
Prev Up Chapter 9. Functions and Operators Home Next

9.7. Pattern Matching

9.7.1. LIKE
9.7.2. SIMILAR TO Regular Expressions
9.7.3. POSIX Regular Expressions

There are three separate approaches to pattern matching provided by PostgreSQL: the traditional SQL LIKE operator, the more recent SIMILAR TO operator (added in SQL:1999), and POSIX-style regular expressions. Aside from the basic “does this string match this pattern?” operators, functions are available to extract or replace matching substrings and to split a string at matching locations.

Tip

If you have pattern matching needs that go beyond this, consider writing a user-defined function in Perl or Tcl.

Caution

While most regular-expression searches can be executed very quickly, regular expressions can be contrived that take arbitrary amounts of time and memory to process. Be wary of accepting regular-expression search patterns from hostile sources. If you must do so, it is advisable to impose a statement timeout.

Searches using SIMILAR TO patterns have the same security hazards, since SIMILAR TO provides many of the same capabilities as POSIX-style regular expressions.

LIKE searches, being much simpler than the other two options, are safer to use with possibly-hostile pattern sources.

The pattern matching operators of all three kinds do not support nondeterministic collations. If required, apply a different collation to the expression to work around this limitation.

9.7.1. LIKE

string LIKE pattern [ESCAPE escape-character]
string NOT LIKE pattern [ESCAPE escape-character]

The LIKE expression returns true if the string matches the supplied pattern. (As expected, the NOT LIKE expression returns false if LIKE returns true, and vice versa. An equivalent expression is NOT (string LIKE pattern).)

If pattern does not contain percent signs or underscores, then the pattern only represents the string itself; in that case LIKE acts like the equals operator. An underscore (_) in pattern stands for (matches) any single character; a percent sign (%) matches any sequence of zero or more characters.

Some examples:

'abc' LIKE 'abc'    true
'abc' LIKE 'a%'     true
'abc' LIKE '_b_'    true
'abc' LIKE 'c'      false

LIKE pattern matching always covers the entire string. Therefore, if it's desired to match a sequence anywhere within a string, the pattern must start and end with a percent sign.

To match a literal underscore or percent sign without matching other characters, the respective character in pattern must be preceded by the escape character. The default escape character is the backslash but a different one can be selected by using the ESCAPE clause. To match the escape character itself, write two escape characters.

Note

If you have standard_conforming_strings turned off, any backslashes you write in literal string constants will need to be doubled. See Section 4.1.2.1 for more information.

It's also possible to select no escape character by writing ESCAPE ''. This effectively disables the escape mechanism, which makes it impossible to turn off the special meaning of underscore and percent signs in the pattern.

According to the SQL standard, omitting ESCAPE means there is no escape character (rather than defaulting to a backslash), and a zero-length ESCAPE value is disallowed. PostgreSQL's behavior in this regard is therefore slightly nonstandard.

The key word ILIKE can be used instead of LIKE to make the match case-insensitive according to the active locale. This is not in the SQL standard but is a PostgreSQL extension.

The operator ~~ is equivalent to LIKE, and ~~* corresponds to ILIKE. There are also !~~ and !~~* operators that represent NOT LIKE and NOT ILIKE, respectively. All of these operators are PostgreSQL-specific. You may see these operator names in EXPLAIN output and similar places, since the parser actually translates LIKE et al. to these operators.

The phrases LIKE, ILIKE, NOT LIKE, and NOT ILIKE are generally treated as operators in PostgreSQL syntax; for example they can be used in expression operator ANY (subquery) constructs, although an ESCAPE clause cannot be included there. In some obscure cases it may be necessary to use the underlying operator names instead.

Also see the prefix operator ^@ and corresponding starts_with function, which are useful in cases where simply matching the beginning of a string is needed.

9.7.2. SIMILAR TO Regular Expressions

string SIMILAR TO pattern [ESCAPE escape-character]
string NOT SIMILAR TO pattern [ESCAPE escape-character]

The SIMILAR TO operator returns true or false depending on whether its pattern matches the given string. It is similar to LIKE, except that it interprets the pattern using the SQL standard's definition of a regular expression. SQL regular expressions are a curious cross between LIKE notation and common (POSIX) regular expression notation.

Like LIKE, the SIMILAR TO operator succeeds only if its pattern matches the entire string; this is unlike common regular expression behavior where the pattern can match any part of the string. Also like LIKE, SIMILAR TO uses _ and % as wildcard characters denoting any single character and any string, respectively (these are comparable to . and .* in POSIX regular expressions).

In addition to these facilities borrowed from LIKE, SIMILAR TO supports these pattern-matching metacharacters borrowed from POSIX regular expressions:

  • | denotes alternation (either of two alternatives).

  • * denotes repetition of the previous item zero or more times.

  • + denotes repetition of the previous item one or more times.

  • ? denotes repetition of the previous item zero or one time.

  • {m} denotes repetition of the previous item exactly m times.

  • {m,} denotes repetition of the previous item m or more times.

  • {m,n} denotes repetition of the previous item at least m and not more than n times.

  • Parentheses () can be used to group items into a single logical item.

  • A bracket expression [...] specifies a character class, just as in POSIX regular expressions.

Notice that the period (.) is not a metacharacter for SIMILAR TO.

As with LIKE, a backslash disables the special meaning of any of these metacharacters. A different escape character can be specified with ESCAPE, or the escape capability can be disabled by writing ESCAPE ''.

According to the SQL standard, omitting ESCAPE means there is no escape character (rather than defaulting to a backslash), and a zero-length ESCAPE value is disallowed. PostgreSQL's behavior in this regard is therefore slightly nonstandard.

Another nonstandard extension is that following the escape character with a letter or digit provides access to the escape sequences defined for POSIX regular expressions; see Table 9.20, Table 9.21, and Table 9.22 below.

Some examples:

'abc' SIMILAR TO 'abc'          true
'abc' SIMILAR TO 'a'            false
'abc' SIMILAR TO '%(b|d)%'      true
'abc' SIMILAR TO '(b|c)%'       false
'-abc-' SIMILAR TO '%\mabc\M%'  true
'xabcy' SIMILAR TO '%\mabc\M%'  false

The substring function with three parameters provides extraction of a substring that matches an SQL regular expression pattern. The function can be written according to standard SQL syntax:

substring(string similar pattern escape escape-character)

or using the now obsolete SQL:1999 syntax:

substring(string from pattern for escape-character)

or as a plain three-argument function:

substring(string, pattern, escape-character)

As with SIMILAR TO, the specified pattern must match the entire data string, or else the function fails and returns null. To indicate the part of the pattern for which the matching data sub-string is of interest, the pattern should contain two occurrences of the escape character followed by a double quote ("). The text matching the portion of the pattern between these separators is returned when the match is successful.

The escape-double-quote separators actually divide substring's pattern into three independent regular expressions; for example, a vertical bar (|) in any of the three sections affects only that section. Also, the first and third of these regular expressions are defined to match the smallest possible amount of text, not the largest, when there is any ambiguity about how much of the data string matches which pattern. (In POSIX parlance, the first and third regular expressions are forced to be non-greedy.)

As an extension to the SQL standard, PostgreSQL allows there to be just one escape-double-quote separator, in which case the third regular expression is taken as empty; or no separators, in which case the first and third regular expressions are taken as empty.

Some examples, with #" delimiting the return string:

substring('foobar' similar '%#"o_b#"%' escape '#')   oob
substring('foobar' similar '#"o_b#"%' escape '#')    NULL

9.7.3. POSIX Regular Expressions

Table 9.16 lists the available operators for pattern matching using POSIX regular expressions.

Table 9.16. Regular Expression Match Operators

Operator

Description

Example(s)

text ~ text → boolean

String matches regular expression, case sensitively

'thomas' ~ 't.*ma' → t

text ~* text → boolean

String matches regular expression, case insensitively

'thomas' ~* 'T.*ma' → t

text !~ text → boolean

String does not match regular expression, case sensitively

'thomas' !~ 't.*max' → t

text !~* text → boolean

String does not match regular expression, case insensitively

'thomas' !~* 'T.*ma' → f

POSIX regular expressions provide a more powerful means for pattern matching than the LIKE and SIMILAR TO operators. Many Unix tools such as egrep, sed, or awk use a pattern matching language that is similar to the one described here.

A regular expression is a character sequence that is an abbreviated definition of a set of strings (a regular set). A string is said to match a regular expression if it is a member of the regular set described by the regular expression. As with LIKE, pattern characters match string characters exactly unless they are special characters in the regular expression language — but regular expressions use different special characters than LIKE does. Unlike LIKE patterns, a regular expression is allowed to match anywhere within a string, unless the regular expression is explicitly anchored to the beginning or end of the string.

Some examples:

'abcd' ~ 'bc'     true
'abcd' ~ 'a.c'    true — dot matches any character
'abcd' ~ 'a.*d'   true — * repeats the preceding pattern item
'abcd' ~ '(b|x)'  true — | means OR, parentheses group
'abcd' ~ '^a'     true — ^ anchors to start of string
'abcd' ~ '^(b|c)' false — would match except for anchoring

The POSIX pattern language is described in much greater detail below.

The substring function with two parameters, substring(string from pattern), provides extraction of a substring that matches a POSIX regular expression pattern. It returns null if there is no match, otherwise the first portion of the text that matched the pattern. But if the pattern contains any parentheses, the portion of the text that matched the first parenthesized subexpression (the one whose left parenthesis comes first) is returned. You can put parentheses around the whole expression if you want to use parentheses within it without triggering this exception. If you need parentheses in the pattern before the subexpression you want to extract, see the non-capturing parentheses described below.

Some examples:

substring('foobar' from 'o.b')     oob
substring('foobar' from 'o(.)b')   o

The regexp_replace function provides substitution of new text for substrings that match POSIX regular expression patterns. It has the syntax regexp_replace(source, pattern, replacement [, flags ]). The source string is returned unchanged if there is no match to the pattern. If there is a match, the source string is returned with the replacement string substituted for the matching substring. The replacement string can contain \n, where n is 1 through 9, to indicate that the source substring matching the n'th parenthesized subexpression of the pattern should be inserted, and it can contain \& to indicate that the substring matching the entire pattern should be inserted. Write \\ if you need to put a literal backslash in the replacement text. The flags parameter is an optional text string containing zero or more single-letter flags that change the function's behavior. Flag i specifies case-insensitive matching, while flag g specifies replacement of each matching substring rather than only the first one. Supported flags (though not g) are described in Table 9.24.

Some examples:

regexp_replace('foobarbaz', 'b..', 'X')
                                   fooXbaz
regexp_replace('foobarbaz', 'b..', 'X', 'g')
                                   fooXX
regexp_replace('foobarbaz', 'b(..)', 'X\1Y', 'g')
                                   fooXarYXazY

The regexp_match function returns a text array of captured substring(s) resulting from the first match of a POSIX regular expression pattern to a string. It has the syntax regexp_match(string, pattern [, flags ]). If there is no match, the result is NULL. If a match is found, and the pattern contains no parenthesized subexpressions, then the result is a single-element text array containing the substring matching the whole pattern. If a match is found, and the pattern contains parenthesized subexpressions, then the result is a text array whose n'th element is the substring matching the n'th parenthesized subexpression of the pattern (not counting “non-capturing” parentheses; see below for details). The flags parameter is an optional text string containing zero or more single-letter flags that change the function's behavior. Supported flags are described in Table 9.24.

Some examples:

SELECT regexp_match('foobarbequebaz', 'bar.*que');
 regexp_match
--------------
 {barbeque}
(1 row)

SELECT regexp_match('foobarbequebaz', '(bar)(beque)');
 regexp_match
--------------
 {bar,beque}
(1 row)

In the common case where you just want the whole matching substring or NULL for no match, write something like

SELECT (regexp_match('foobarbequebaz', 'bar.*que'))[1];
 regexp_match
--------------
 barbeque
(1 row)

The regexp_matches function returns a set of text arrays of captured substring(s) resulting from matching a POSIX regular expression pattern to a string. It has the same syntax as regexp_match. This function returns no rows if there is no match, one row if there is a match and the g flag is not given, or N rows if there are N matches and the g flag is given. Each returned row is a text array containing the whole matched substring or the substrings matching parenthesized subexpressions of the pattern, just as described above for regexp_match. regexp_matches accepts all the flags shown in Table 9.24, plus the g flag which commands it to return all matches, not just the first one.

Some examples:

SELECT regexp_matches('foo', 'not there');
 regexp_matches
----------------
(0 rows)

SELECT regexp_matches('foobarbequebazilbarfbonk', '(b[^b]+)(b[^b]+)', 'g');
 regexp_matches
----------------
 {bar,beque}
 {bazil,barf}
(2 rows)

Tip

In most cases regexp_matches() should be used with the g flag, since if you only want the first match, it's easier and more efficient to use regexp_match(). However, regexp_match() only exists in PostgreSQL version 10 and up. When working in older versions, a common trick is to place a regexp_matches() call in a sub-select, for example:

SELECT col1, (SELECT regexp_matches(col2, '(bar)(beque)')) FROM tab;

This produces a text array if there's a match, or NULL if not, the same as regexp_match() would do. Without the sub-select, this query would produce no output at all for table rows without a match, which is typically not the desired behavior.

The regexp_split_to_table function splits a string using a POSIX regular expression pattern as a delimiter. It has the syntax regexp_split_to_table(string, pattern [, flags ]). If there is no match to the pattern, the function returns the string. If there is at least one match, for each match it returns the text from the end of the last match (or the beginning of the string) to the beginning of the match. When there are no more matches, it returns the text from the end of the last match to the end of the string. The flags parameter is an optional text string containing zero or more single-letter flags that change the function's behavior. regexp_split_to_table supports the flags described in Table 9.24.

The regexp_split_to_array function behaves the same as regexp_split_to_table, except that regexp_split_to_array returns its result as an array of text. It has the syntax regexp_split_to_array(string, pattern [, flags ]). The parameters are the same as for regexp_split_to_table.

Some examples:

SELECT foo FROM regexp_split_to_table('the quick brown fox jumps over the lazy dog', '\s+') AS foo;
  foo
-------
 the
 quick
 brown
 fox
 jumps
 over
 the
 lazy
 dog
(9 rows)

SELECT regexp_split_to_array('the quick brown fox jumps over the lazy dog', '\s+');
              regexp_split_to_array
-----------------------------------------------
 {the,quick,brown,fox,jumps,over,the,lazy,dog}
(1 row)

SELECT foo FROM regexp_split_to_table('the quick brown fox', '\s*') AS foo;
 foo
-----
 t
 h
 e
 q
 u
 i
 c
 k
 b
 r
 o
 w
 n
 f
 o
 x
(16 rows)

As the last example demonstrates, the regexp split functions ignore zero-length matches that occur at the start or end of the string or immediately after a previous match. This is contrary to the strict definition of regexp matching that is implemented by regexp_match and regexp_matches, but is usually the most convenient behavior in practice. Other software systems such as Perl use similar definitions.

9.7.3.1. Regular Expression Details

PostgreSQL's regular expressions are implemented using a software package written by Henry Spencer. Much of the description of regular expressions below is copied verbatim from his manual.

Regular expressions (REs), as defined in POSIX 1003.2, come in two forms: extended REs or EREs (roughly those of egrep), and basic REs or BREs (roughly those of ed). PostgreSQL supports both forms, and also implements some extensions that are not in the POSIX standard, but have become widely used due to their availability in programming languages such as Perl and Tcl. REs using these non-POSIX extensions are called advanced REs or AREs in this documentation. AREs are almost an exact superset of EREs, but BREs have several notational incompatibilities (as well as being much more limited). We first describe the ARE and ERE forms, noting features that apply only to AREs, and then describe how BREs differ.

Note

PostgreSQL always initially presumes that a regular expression follows the ARE rules. However, the more limited ERE or BRE rules can be chosen by prepending an embedded option to the RE pattern, as described in Section 9.7.3.4. This can be useful for compatibility with applications that expect exactly the POSIX 1003.2 rules.

A regular expression is defined as one or more branches, separated by |. It matches anything that matches one of the branches.

A branch is zero or more quantified atoms or constraints, concatenated. It matches a match for the first, followed by a match for the second, etc; an empty branch matches the empty string.

A quantified atom is an atom possibly followed by a single quantifier. Without a quantifier, it matches a match for the atom. With a quantifier, it can match some number of matches of the atom. An atom can be any of the possibilities shown in Table 9.17. The possible quantifiers and their meanings are shown in Table 9.18.

A constraint matches an empty string, but matches only when specific conditions are met. A constraint can be used where an atom could be used, except it cannot be followed by a quantifier. The simple constraints are shown in Table 9.19; some more constraints are described later.

Table 9.17. Regular Expression Atoms

Atom | Description
(re) | (where re is any regular expression) matches a match for re, with the match noted for possible reporting
(?:re) | as above, but the match is not noted for reporting (a “non-capturing” set of parentheses) (AREs only)
. | matches any single character
[chars] | a bracket expression, matching any one of the chars (see Section 9.7.3.2 for more detail)
\k | (where k is a non-alphanumeric character) matches that character taken as an ordinary character, e.g., \\ matches a backslash character
\c | where c is alphanumeric (possibly followed by other characters) is an escape, see Section 9.7.3.3 (AREs only; in EREs and BREs, this matches c)
{ | when followed by a character other than a digit, matches the left-brace character {; when followed by a digit, it is the beginning of a bound (see below)
x | where x is a single character with no other significance, matches that character

An RE cannot end with a backslash (\).

Note

If you have standard_conforming_strings turned off, any backslashes you write in literal string constants will need to be doubled. See Section 4.1.2.1 for more information.

Table 9.18. Regular Expression Quantifiers

Quantifier | Matches
* | a sequence of 0 or more matches of the atom
+ | a sequence of 1 or more matches of the atom
? | a sequence of 0 or 1 matches of the atom
{m} | a sequence of exactly m matches of the atom
{m,} | a sequence of m or more matches of the atom
{m,n} | a sequence of m through n (inclusive) matches of the atom; m cannot exceed n
*? | non-greedy version of *
+? | non-greedy version of +
?? | non-greedy version of ?
{m}? | non-greedy version of {m}
{m,}? | non-greedy version of {m,}
{m,n}? | non-greedy version of {m,n}

The forms using {...} are known as bounds. The numbers m and n within a bound are unsigned decimal integers with permissible values from 0 to 255 inclusive.

Non-greedy quantifiers (available in AREs only) match the same possibilities as their corresponding normal (greedy) counterparts, but prefer the smallest number rather than the largest number of matches. See Section 9.7.3.5 for more detail.

Note

A quantifier cannot immediately follow another quantifier, e.g., ** is invalid. A quantifier cannot begin an expression or subexpression or follow ^ or |.

Table 9.19. Regular Expression Constraints

Constraint | Description
^ | matches at the beginning of the string
$ | matches at the end of the string
(?=re) | positive lookahead matches at any point where a substring matching re begins (AREs only)
(?!re) | negative lookahead matches at any point where no substring matching re begins (AREs only)
(?<=re) | positive lookbehind matches at any point where a substring matching re ends (AREs only)
(?<!re) | negative lookbehind matches at any point where no substring matching re ends (AREs only)

Lookahead and lookbehind constraints cannot contain back references (see Section 9.7.3.3), and all parentheses within them are considered non-capturing.

9.7.3.2. Bracket Expressions

A bracket expression is a list of characters enclosed in []. It normally matches any single character from the list (but see below). If the list begins with ^, it matches any single character not from the rest of the list. If two characters in the list are separated by -, this is shorthand for the full range of characters between those two (inclusive) in the collating sequence, e.g., [0-9] in ASCII matches any decimal digit. It is illegal for two ranges to share an endpoint, e.g., a-c-e. Ranges are very collating-sequence-dependent, so portable programs should avoid relying on them.

To include a literal ] in the list, make it the first character (after ^, if that is used). To include a literal -, make it the first or last character, or the second endpoint of a range. To use a literal - as the first endpoint of a range, enclose it in [. and .] to make it a collating element (see below). With the exception of these characters, some combinations using [ (see next paragraphs), and escapes (AREs only), all other special characters lose their special significance within a bracket expression. In particular, \ is not special when following ERE or BRE rules, though it is special (as introducing an escape) in AREs.

Within a bracket expression, a collating element (a character, a multiple-character sequence that collates as if it were a single character, or a collating-sequence name for either) enclosed in [. and .] stands for the sequence of characters of that collating element. The sequence is treated as a single element of the bracket expression's list. This allows a bracket expression containing a multiple-character collating element to match more than one character, e.g., if the collating sequence includes a ch collating element, then the RE [[.ch.]]*c matches the first five characters of chchcc.

Note

PostgreSQL currently does not support multi-character collating elements. This information describes possible future behavior.

Within a bracket expression, a collating element enclosed in [= and =] is an equivalence class, standing for the sequences of characters of all collating elements equivalent to that one, including itself. (If there are no other equivalent collating elements, the treatment is as if the enclosing delimiters were [. and .].) For example, if o and ^ are the members of an equivalence class, then [[=o=]], [[=^=]], and [o^] are all synonymous. An equivalence class cannot be an endpoint of a range.

Within a bracket expression, the name of a character class enclosed in [: and :] stands for the list of all characters belonging to that class. A character class cannot be used as an endpoint of a range. The POSIX standard defines these character class names: alnum (letters and numeric digits), alpha (letters), blank (space and tab), cntrl (control characters), digit (numeric digits), graph (printable characters except space), lower (lower-case letters), print (printable characters including space), punct (punctuation), space (any white space), upper (upper-case letters), and xdigit (hexadecimal digits). The behavior of these standard character classes is generally consistent across platforms for characters in the 7-bit ASCII set. Whether a given non-ASCII character is considered to belong to one of these classes depends on the collation that is used for the regular-expression function or operator (see Section 24.2), or by default on the database's LC_CTYPE locale setting (see Section 24.1). The classification of non-ASCII characters can vary across platforms even in similarly-named locales. (But the C locale never considers any non-ASCII characters to belong to any of these classes.) In addition to these standard character classes, PostgreSQL defines the word character class, which is the same as alnum plus the underscore (_) character, and the ascii character class, which contains exactly the 7-bit ASCII set.

There are two special cases of bracket expressions: the bracket expressions [[:<:]] and [[:>:]] are constraints, matching empty strings at the beginning and end of a word respectively. A word is defined as a sequence of word characters that is neither preceded nor followed by word characters. A word character is any character belonging to the word character class, that is, any letter, digit, or underscore. This is an extension, compatible with but not specified by POSIX 1003.2, and should be used with caution in software intended to be portable to other systems. The constraint escapes described below are usually preferable; they are no more standard, but are easier to type.

9.7.3.3. Regular Expression Escapes

Escapes are special sequences beginning with \ followed by an alphanumeric character. Escapes come in several varieties: character entry, class shorthands, constraint escapes, and back references. A \ followed by an alphanumeric character but not constituting a valid escape is illegal in AREs. In EREs, there are no escapes: outside a bracket expression, a \ followed by an alphanumeric character merely stands for that character as an ordinary character, and inside a bracket expression, \ is an ordinary character. (The latter is the one actual incompatibility between EREs and AREs.)

Character-entry escapes exist to make it easier to specify non-printing and other inconvenient characters in REs. They are shown in Table 9.20.

Class-shorthand escapes provide shorthands for certain commonly-used character classes. They are shown in Table 9.21.

A constraint escape is a constraint, matching the empty string if specific conditions are met, written as an escape. They are shown in Table 9.22.

A back reference (\n) matches the same string matched by the previous parenthesized subexpression specified by the number n (see Table 9.23). For example, ([bc])\1 matches bb or cc but not bc or cb. The subexpression must entirely precede the back reference in the RE. Subexpressions are numbered in the order of their leading parentheses. Non-capturing parentheses do not define subexpressions. The back reference considers only the string characters matched by the referenced subexpression, not any constraints contained in it. For example, (^\d)\1 will match 22.

Table 9.20. Regular Expression Character-Entry Escapes

Escape | Description
\a | alert (bell) character, as in C
\b | backspace, as in C
\B | synonym for backslash (\) to help reduce the need for backslash doubling
\c X | (where X is any character) the character whose low-order 5 bits are the same as those of X, and whose other bits are all zero
\e | the character whose collating-sequence name is ESC, or failing that, the character with octal value 033
\f | form feed, as in C
\n | newline, as in C
\r | carriage return, as in C
\t | horizontal tab, as in C
\u wxyz | (where wxyz is exactly four hexadecimal digits) the character whose hexadecimal value is 0x wxyz
\U stuvwxyz | (where stuvwxyz is exactly eight hexadecimal digits) the character whose hexadecimal value is 0x stuvwxyz
\v | vertical tab, as in C
\x hhh | (where hhh is any sequence of hexadecimal digits) the character whose hexadecimal value is 0x hhh (a single character no matter how many hexadecimal digits are used)
\0 | the character whose value is 0 (the null byte)
\xy | (where xy is exactly two octal digits, and is not a back reference) the character whose octal value is 0 xy
\xyz | (where xyz is exactly three octal digits, and is not a back reference) the character whose octal value is 0 xyz

Hexadecimal digits are 0-9, a-f, and A-F. Octal digits are 0-7.

Numeric character-entry escapes specifying values outside the ASCII range (0–127) have meanings dependent on the database encoding. When the encoding is UTF-8, escape values are equivalent to Unicode code points, for example \u1234 means the character U+1234. For other multibyte encodings, character-entry escapes usually just specify the concatenation of the byte values for the character. If the escape value does not correspond to any legal character in the database encoding, no error will be raised, but it will never match any data.

The character-entry escapes are always taken as ordinary characters. For example, \135 is ] in ASCII, but \135 does not terminate a bracket expression.

Table 9.21. Regular Expression Class-Shorthand Escapes

Escape | Description
\d | matches any digit, like [[:digit:]]
\s | matches any whitespace character, like [[:space:]]
\w | matches any word character, like [[:word:]]
\D | matches any non-digit, like [^[:digit:]]
\S | matches any non-whitespace character, like [^[:space:]]
\W | matches any non-word character, like [^[:word:]]

The class-shorthand escapes also work within bracket expressions, although the definitions shown above are not quite syntactically valid in that context. For example, [a-c\d] is equivalent to [a-c[:digit:]].

Table 9.22. Regular Expression Constraint Escapes

Escape | Description
\A | matches only at the beginning of the string (see Section 9.7.3.5 for how this differs from ^)
\m | matches only at the beginning of a word
\M | matches only at the end of a word
\y | matches only at the beginning or end of a word
\Y | matches only at a point that is not the beginning or end of a word
\Z | matches only at the end of the string (see Section 9.7.3.5 for how this differs from $)

A word is defined as in the specification of [[:<:]] and [[:>:]] above. Constraint escapes are illegal within bracket expressions.

Table 9.23. Regular Expression Back References

Escape | Description
\m | (where m is a nonzero digit) a back reference to the m'th subexpression
\mnn | (where m is a nonzero digit, and nn is some more digits, and the decimal value mnn is not greater than the number of closing capturing parentheses seen so far) a back reference to the mnn'th subexpression

Note

There is an inherent ambiguity between octal character-entry escapes and back references, which is resolved by the following heuristics, as hinted at above. A leading zero always indicates an octal escape. A single non-zero digit, not followed by another digit, is always taken as a back reference. A multi-digit sequence not starting with a zero is taken as a back reference if it comes after a suitable subexpression (i.e., the number is in the legal range for a back reference), and otherwise is taken as octal.

9.7.3.4. Regular Expression Metasyntax

In addition to the main syntax described above, there are some special forms and miscellaneous syntactic facilities available.

An RE can begin with one of two special director prefixes. If an RE begins with ***:, the rest of the RE is taken as an ARE. (This normally has no effect in PostgreSQL, since REs are assumed to be AREs; but it does have an effect if ERE or BRE mode had been specified by the flags parameter to a regex function.) If an RE begins with ***=, the rest of the RE is taken to be a literal string, with all characters considered ordinary characters.

An ARE can begin with embedded options: a sequence (?xyz) (where xyz is one or more alphabetic characters) specifies options affecting the rest of the RE. These options override any previously determined options — in particular, they can override the case-sensitivity behavior implied by a regex operator, or the flags parameter to a regex function. The available option letters are shown in Table 9.24. Note that these same option letters are used in the flags parameters of regex functions.

Table 9.24. ARE Embedded-Option Letters

Option | Description
b | rest of RE is a BRE
c | case-sensitive matching (overrides operator type)
e | rest of RE is an ERE
i | case-insensitive matching (see Section 9.7.3.5) (overrides operator type)
m | historical synonym for n
n | newline-sensitive matching (see Section 9.7.3.5)
p | partial newline-sensitive matching (see Section 9.7.3.5)
q | rest of RE is a literal (“quoted”) string, all ordinary characters
s | non-newline-sensitive matching (default)
t | tight syntax (default; see below)
w | inverse partial newline-sensitive (“weird”) matching (see Section 9.7.3.5)
x | expanded syntax (see below)

Embedded options take effect at the ) terminating the sequence. They can appear only at the start of an ARE (after the ***: director if any).

In addition to the usual (tight) RE syntax, in which all characters are significant, there is an expanded syntax, available by specifying the embedded x option. In the expanded syntax, white-space characters in the RE are ignored, as are all characters between a # and the following newline (or the end of the RE). This permits paragraphing and commenting a complex RE. There are three exceptions to that basic rule:

  • a white-space character or # preceded by \ is retained

  • white space or # within a bracket expression is retained

  • white space and comments cannot appear within multi-character symbols, such as (?:

For this purpose, white-space characters are blank, tab, newline, and any character that belongs to the space character class.

Finally, in an ARE, outside bracket expressions, the sequence (?#ttt) (where ttt is any text not containing a )) is a comment, completely ignored. Again, this is not allowed between the characters of multi-character symbols, like (?:. Such comments are more a historical artifact than a useful facility, and their use is deprecated; use the expanded syntax instead.

None of these metasyntax extensions is available if an initial ***= director has specified that the user's input be treated as a literal string rather than as an RE.

9.7.3.5. Regular Expression Matching Rules

In the event that an RE could match more than one substring of a given string, the RE matches the one starting earliest in the string. If the RE could match more than one substring starting at that point, either the longest possible match or the shortest possible match will be taken, depending on whether the RE is greedy or non-greedy.

Whether an RE is greedy or not is determined by the following rules:

  • Most atoms, and all constraints, have no greediness attribute (because they cannot match variable amounts of text anyway).

  • Adding parentheses around an RE does not change its greediness.

  • A quantified atom with a fixed-repetition quantifier ({m} or {m}?) has the same greediness (possibly none) as the atom itself.

  • A quantified atom with other normal quantifiers (including {m,n} with m equal to n) is greedy (prefers longest match).

  • A quantified atom with a non-greedy quantifier (including {m,n}? with m equal to n) is non-greedy (prefers shortest match).

  • A branch — that is, an RE that has no top-level | operator — has the same greediness as the first quantified atom in it that has a greediness attribute.

  • An RE consisting of two or more branches connected by the | operator is always greedy.

The above rules associate greediness attributes not only with individual quantified atoms, but with branches and entire REs that contain quantified atoms. What that means is that the matching is done in such a way that the branch, or whole RE, matches the longest or shortest possible substring as a whole. Once the length of the entire match is determined, the part of it that matches any particular subexpression is determined on the basis of the greediness attribute of that subexpression, with subexpressions starting earlier in the RE taking priority over ones starting later.

An example of what this means:

SELECT SUBSTRING('XY1234Z', 'Y*([0-9]{1,3})');
Result: 123
SELECT SUBSTRING('XY1234Z', 'Y*?([0-9]{1,3})');
Result: 1

In the first case, the RE as a whole is greedy because Y* is greedy. It can match beginning at the Y, and it matches the longest possible string starting there, i.e., Y123. The output is the parenthesized part of that, or 123. In the second case, the RE as a whole is non-greedy because Y*? is non-greedy. It can match beginning at the Y, and it matches the shortest possible string starting there, i.e., Y1. The subexpression [0-9]{1,3} is greedy but it cannot change the decision as to the overall match length; so it is forced to match just 1.

In short, when an RE contains both greedy and non-greedy subexpressions, the total match length is either as long as possible or as short as possible, according to the attribute assigned to the whole RE. The attributes assigned to the subexpressions only affect how much of that match they are allowed to “eat” relative to each other.

The quantifiers {1,1} and {1,1}? can be used to force greediness or non-greediness, respectively, on a subexpression or a whole RE. This is useful when you need the whole RE to have a greediness attribute different from what's deduced from its elements. As an example, suppose that we are trying to separate a string containing some digits into the digits and the parts before and after them. We might try to do that like this:

SELECT regexp_match('abc01234xyz', '(.*)(\d+)(.*)');
Result: {abc0123,4,xyz}

That didn't work: the first .* is greedy so it “eats” as much as it can, leaving the \d+ to match at the last possible place, the last digit. We might try to fix that by making it non-greedy:

SELECT regexp_match('abc01234xyz', '(.*?)(\d+)(.*)');
Result: {abc,0,""}

That didn't work either, because now the RE as a whole is non-greedy and so it ends the overall match as soon as possible. We can get what we want by forcing the RE as a whole to be greedy:

SELECT regexp_match('abc01234xyz', '(?:(.*?)(\d+)(.*)){1,1}');
Result: {abc,01234,xyz}

Controlling the RE's overall greediness separately from its components' greediness allows great flexibility in handling variable-length patterns.

When deciding what is a longer or shorter match, match lengths are measured in characters, not collating elements. An empty string is considered longer than no match at all. For example: bb* matches the three middle characters of abbbc; (week|wee)(night|knights) matches all ten characters of weeknights; when (.*).* is matched against abc the parenthesized subexpression matches all three characters; and when (a*)* is matched against bc both the whole RE and the parenthesized subexpression match an empty string.

If case-independent matching is specified, the effect is much as if all case distinctions had vanished from the alphabet. When an alphabetic that exists in multiple cases appears as an ordinary character outside a bracket expression, it is effectively transformed into a bracket expression containing both cases, e.g., x becomes [xX]. When it appears inside a bracket expression, all case counterparts of it are added to the bracket expression, e.g., [x] becomes [xX] and [^x] becomes [^xX].

If newline-sensitive matching is specified, . and bracket expressions using ^ will never match the newline character (so that matches will not cross lines unless the RE explicitly includes a newline) and ^ and $ will match the empty string after and before a newline respectively, in addition to matching at beginning and end of string respectively. But the ARE escapes \A and \Z continue to match beginning or end of string only. Also, the character class shorthands \D and \W will match a newline regardless of this mode. (Before PostgreSQL 14, they did not match newlines when in newline-sensitive mode. Write [^[:digit:]] or [^[:word:]] to get the old behavior.)

If partial newline-sensitive matching is specified, this affects . and bracket expressions as with newline-sensitive matching, but not ^ and $.

If inverse partial newline-sensitive matching is specified, this affects ^ and $ as with newline-sensitive matching, but not . and bracket expressions. This isn't very useful but is provided for symmetry.

9.7.3.6. Limits and Compatibility

No particular limit is imposed on the length of REs in this implementation. However, programs intended to be highly portable should not employ REs longer than 256 bytes, as a POSIX-compliant implementation can refuse to accept such REs.

The only feature of AREs that is actually incompatible with POSIX EREs is that \ does not lose its special significance inside bracket expressions. All other ARE features use syntax which is illegal or has undefined or unspecified effects in POSIX EREs; the *** syntax of directors likewise is outside the POSIX syntax for both BREs and EREs.

Many of the ARE extensions are borrowed from Perl, but some have been changed to clean them up, and a few Perl extensions are not present. Incompatibilities of note include \b, \B, the lack of special treatment for a trailing newline, the addition of complemented bracket expressions to the things affected by newline-sensitive matching, the restrictions on parentheses and back references in lookahead/lookbehind constraints, and the longest/shortest-match (rather than first-match) matching semantics.

9.7.3.7. Basic Regular Expressions

BREs differ from EREs in several respects. In BREs, |, +, and ? are ordinary characters and there is no equivalent for their functionality. The delimiters for bounds are \{ and \}, with { and } by themselves ordinary characters. The parentheses for nested subexpressions are \( and \), with ( and ) by themselves ordinary characters. ^ is an ordinary character except at the beginning of the RE or the beginning of a parenthesized subexpression, $ is an ordinary character except at the end of the RE or the end of a parenthesized subexpression, and * is an ordinary character if it appears at the beginning of the RE or the beginning of a parenthesized subexpression (after a possible leading ^). Finally, single-digit back references are available, and \< and \> are synonyms for [[:<:]] and [[:>:]] respectively; no other escapes are available in BREs.

9.7.3.8. Differences from XQuery (LIKE_REGEX)

Since SQL:2008, the SQL standard includes a LIKE_REGEX operator that performs pattern matching according to the XQuery regular expression standard. PostgreSQL does not yet implement this operator, but you can get very similar behavior using the regexp_match() function, since XQuery regular expressions are quite close to the ARE syntax described above.

Notable differences between the existing POSIX-based regular-expression feature and XQuery regular expressions include:

  • XQuery character class subtraction is not supported. An example of this feature is using the following to match only English consonants: [a-z-[aeiou]].

  • XQuery character class shorthands \c, \C, \i, and \I are not supported.

  • XQuery character class elements using \p{UnicodeProperty} or the inverse \P{UnicodeProperty} are not supported.

  • POSIX interprets character classes such as \w (see Table 9.21) according to the prevailing locale (which you can control by attaching a COLLATE clause to the operator or function). XQuery specifies these classes by reference to Unicode character properties, so equivalent behavior is obtained only with a locale that follows the Unicode rules.

  • The SQL standard (not XQuery itself) attempts to cater for more variants of “newline” than POSIX does. The newline-sensitive matching options described above consider only ASCII NL (\n) to be a newline, but SQL would have us treat CR (\r), CRLF (\r\n) (a Windows-style newline), and some Unicode-only characters like LINE SEPARATOR (U+2028) as newlines as well. Notably, . and \s should count \r\n as one character not two according to SQL.

  • Of the character-entry escapes described in Table 9.20, XQuery supports only \n, \r, and \t.

  • XQuery does not support the [:name:] syntax for character classes within bracket expressions.

  • XQuery does not have lookahead or lookbehind constraints, nor any of the constraint escapes described in Table 9.22.

  • The metasyntax forms described in Section 9.7.3.4 do not exist in XQuery.

  • The regular expression flag letters defined by XQuery are related to but not the same as the option letters for POSIX (Table 9.24). While the i and q options behave the same, others do not:

    • XQuery's s (allow dot to match newline) and m (allow ^ and $ to match at newlines) flags provide access to the same behaviors as POSIX's n, p and w flags, but they do not match the behavior of POSIX's s and m flags. Note in particular that dot-matches-newline is the default behavior in POSIX but not XQuery.

    • XQuery's x (ignore whitespace in pattern) flag is noticeably different from POSIX's expanded-mode flag. POSIX's x flag also allows # to begin a comment in the pattern, and POSIX will not ignore a whitespace character after a backslash.

Prev | Up | Next
9.6. Bit String Functions and Operators | Home | 9.8. Data Type Formatting Functions


9.8. Data Type Formatting Functions
Prev Up Chapter 9. Functions and Operators Home Next

9.8. Data Type Formatting Functions

The PostgreSQL formatting functions provide a powerful set of tools for converting various data types (date/time, integer, floating point, numeric) to formatted strings and for converting from formatted strings to specific data types. Table 9.25 lists them. These functions all follow a common calling convention: the first argument is the value to be formatted and the second argument is a template that defines the output or input format.

Table 9.25. Formatting Functions

Function

Description

Example(s)

to_char ( timestamp, text ) → text

to_char ( timestamp with time zone, text ) → text

Converts time stamp to string according to the given format.

to_char(timestamp '2002-04-20 17:31:12.66', 'HH12:MI:SS') → 05:31:12

to_char ( interval, text ) → text

Converts interval to string according to the given format.

to_char(interval '15h 2m 12s', 'HH24:MI:SS') → 15:02:12

to_char ( numeric_type, text ) → text

Converts number to string according to the given format; available for integer, bigint, numeric, real, double precision.

to_char(125, '999') → 125

to_char(125.8::real, '999D9') → 125.8

to_char(-125.8, '999D99S') → 125.80-

to_date ( text, text ) → date

Converts string to date according to the given format.

to_date('05 Dec 2000', 'DD Mon YYYY') → 2000-12-05

to_number ( text, text ) → numeric

Converts string to numeric according to the given format.

to_number('12,454.8-', '99G999D9S') → -12454.8

to_timestamp ( text, text ) → timestamp with time zone

Converts string to time stamp according to the given format. (See also to_timestamp(double precision) in Table 9.32.)

to_timestamp('05 Dec 2000', 'DD Mon YYYY') → 2000-12-05 00:00:00-05

Tip

to_timestamp and to_date exist to handle input formats that cannot be converted by simple casting. For most standard date/time formats, simply casting the source string to the required data type works, and is much easier. Similarly, to_number is unnecessary for standard numeric representations.

In a to_char output template string, there are certain patterns that are recognized and replaced with appropriately-formatted data based on the given value. Any text that is not a template pattern is simply copied verbatim. Similarly, in an input template string (for the other functions), template patterns identify the values to be supplied by the input data string. If there are characters in the template string that are not template patterns, the corresponding characters in the input data string are simply skipped over (whether or not they are equal to the template string characters).

Table 9.26 shows the template patterns available for formatting date and time values.

Table 9.26. Template Patterns for Date/Time Formatting

Pattern | Description
HH | hour of day (01–12)
HH12 | hour of day (01–12)
HH24 | hour of day (00–23)
MI | minute (00–59)
SS | second (00–59)
MS | millisecond (000–999)
US | microsecond (000000–999999)
FF1 | tenth of second (0–9)
FF2 | hundredth of second (00–99)
FF3 | millisecond (000–999)
FF4 | tenth of a millisecond (0000–9999)
FF5 | hundredth of a millisecond (00000–99999)
FF6 | microsecond (000000–999999)
SSSS, SSSSS | seconds past midnight (0–86399)
AM, am, PM or pm | meridiem indicator (without periods)
A.M., a.m., P.M. or p.m. | meridiem indicator (with periods)
Y,YYY | year (4 or more digits) with comma
YYYY | year (4 or more digits)
YYY | last 3 digits of year
YY | last 2 digits of year
Y | last digit of year
IYYY | ISO 8601 week-numbering year (4 or more digits)
IYY | last 3 digits of ISO 8601 week-numbering year
IY | last 2 digits of ISO 8601 week-numbering year
I | last digit of ISO 8601 week-numbering year
BC, bc, AD or ad | era indicator (without periods)
B.C., b.c., A.D. or a.d. | era indicator (with periods)
MONTH | full upper case month name (blank-padded to 9 chars)
Month | full capitalized month name (blank-padded to 9 chars)
month | full lower case month name (blank-padded to 9 chars)
MON | abbreviated upper case month name (3 chars in English, localized lengths vary)
Mon | abbreviated capitalized month name (3 chars in English, localized lengths vary)
mon | abbreviated lower case month name (3 chars in English, localized lengths vary)
MM | month number (01–12)
DAY | full upper case day name (blank-padded to 9 chars)
Day | full capitalized day name (blank-padded to 9 chars)
day | full lower case day name (blank-padded to 9 chars)
DY | abbreviated upper case day name (3 chars in English, localized lengths vary)
Dy | abbreviated capitalized day name (3 chars in English, localized lengths vary)
dy | abbreviated lower case day name (3 chars in English, localized lengths vary)
DDD | day of year (001–366)
IDDD | day of ISO 8601 week-numbering year (001–371; day 1 of the year is Monday of the first ISO week)
DD | day of month (01–31)
D | day of the week, Sunday (1) to Saturday (7)
ID | ISO 8601 day of the week, Monday (1) to Sunday (7)
W | week of month (1–5) (the first week starts on the first day of the month)
WW | week number of year (1–53) (the first week starts on the first day of the year)
IW | week number of ISO 8601 week-numbering year (01–53; the first Thursday of the year is in week 1)
CC | century (2 digits) (the twenty-first century starts on 2001-01-01)
J | Julian Date (integer days since November 24, 4714 BC at local midnight; see Section B.7)
Q | quarter
RM | month in upper case Roman numerals (I–XII; I=January)
rm | month in lower case Roman numerals (i–xii; i=January)
TZ | upper case time-zone abbreviation (only supported in to_char)
tz | lower case time-zone abbreviation (only supported in to_char)
TZH | time-zone hours
TZM | time-zone minutes
OF | time-zone offset from UTC (only supported in to_char)

Modifiers can be applied to any template pattern to alter its behavior. For example, FMMonth is the Month pattern with the FM modifier. Table 9.27 shows the modifier patterns for date/time formatting.

Table 9.27. Template Pattern Modifiers for Date/Time Formatting

Modifier | Description | Example
FM prefix | fill mode (suppress leading zeroes and padding blanks) | FMMonth
TH suffix | upper case ordinal number suffix | DDTH, e.g., 12TH
th suffix | lower case ordinal number suffix | DDth, e.g., 12th
FX prefix | fixed format global option (see usage notes) | FX Month DD Day
TM prefix | translation mode (use localized day and month names based on lc_time) | TMMonth
SP suffix | spell mode (not implemented) | DDSP

Usage notes for date/time formatting:

  • FM suppresses leading zeroes and trailing blanks that would otherwise be added to make the output of a pattern be fixed-width. In PostgreSQL, FM modifies only the next specification, while in Oracle FM affects all subsequent specifications, and repeated FM modifiers toggle fill mode on and off.

  • TM suppresses trailing blanks whether or not FM is specified.

  • to_timestamp and to_date ignore letter case in the input; so for example MON, Mon, and mon all accept the same strings. When using the TM modifier, case-folding is done according to the rules of the function's input collation (see Section 24.2).

  • to_timestamp and to_date skip multiple blank spaces at the beginning of the input string and around date and time values unless the FX option is used. For example, to_timestamp(' 2000    JUN', 'YYYY MON') and to_timestamp('2000 - JUN', 'YYYY-MON') work, but to_timestamp('2000    JUN', 'FXYYYY MON') returns an error because to_timestamp expects only a single space. FX must be specified as the first item in the template.

  • A separator (a space or non-letter/non-digit character) in the template string of to_timestamp and to_date matches any single separator in the input string or is skipped, unless the FX option is used. For example, to_timestamp('2000JUN', 'YYYY///MON') and to_timestamp('2000/JUN', 'YYYY MON') work, but to_timestamp('2000//JUN', 'YYYY/MON') returns an error because the number of separators in the input string exceeds the number of separators in the template.

    If FX is specified, a separator in the template string matches exactly one character in the input string. But note that the input string character is not required to be the same as the separator from the template string. For example, to_timestamp('2000/JUN', 'FXYYYY MON') works, but to_timestamp('2000/JUN', 'FXYYYY  MON') returns an error because the second space in the template string consumes the letter J from the input string.

  • A TZH template pattern can match a signed number. Without the FX option, minus signs may be ambiguous, and could be interpreted as a separator. This ambiguity is resolved as follows: If the number of separators before TZH in the template string is less than the number of separators before the minus sign in the input string, the minus sign is interpreted as part of TZH. Otherwise, the minus sign is considered to be a separator between values. For example, to_timestamp('2000 -10', 'YYYY TZH') matches -10 to TZH, but to_timestamp('2000 -10', 'YYYY  TZH') matches 10 to TZH.

  • Ordinary text is allowed in to_char templates and will be output literally. You can put a substring in double quotes to force it to be interpreted as literal text even if it contains template patterns. For example, in '"Hello Year "YYYY', the YYYY will be replaced by the year data, but the single Y in Year will not be. In to_date, to_number, and to_timestamp, literal text and double-quoted strings result in skipping the number of characters contained in the string; for example "XX" skips two input characters (whether or not they are XX).

    Tip

    Prior to PostgreSQL 12, it was possible to skip arbitrary text in the input string using non-letter or non-digit characters. For example, to_timestamp('2000y6m1d', 'yyyy-MM-DD') used to work. Now you can only use letter characters for this purpose. For example, to_timestamp('2000y6m1d', 'yyyytMMtDDt') and to_timestamp('2000y6m1d', 'yyyy"y"MM"m"DD"d"') skip y, m, and d.

  • If you want to have a double quote in the output you must precede it with a backslash, for example '\"YYYY Month\"'. Backslashes are not otherwise special outside of double-quoted strings. Within a double-quoted string, a backslash causes the next character to be taken literally, whatever it is (but this has no special effect unless the next character is a double quote or another backslash).

  • In to_timestamp and to_date, if the year format specification is less than four digits, e.g., YYY, and the supplied year is less than four digits, the year will be adjusted to be nearest to the year 2020, e.g., 95 becomes 1995.

  • In to_timestamp and to_date, negative years are treated as signifying BC. If you write both a negative year and an explicit BC field, you get AD again. An input of year zero is treated as 1 BC.

  • In to_timestamp and to_date, the YYYY conversion has a restriction when processing years with more than 4 digits. You must use some non-digit character or template after YYYY, otherwise the year is always interpreted as 4 digits. For example (with the year 20000): to_date('200001131', 'YYYYMMDD') will be interpreted as a 4-digit year; instead use a non-digit separator after the year, like to_date('20000-1131', 'YYYY-MMDD') or to_date('20000Nov31', 'YYYYMonDD').

  • In to_timestamp and to_date, the CC (century) field is accepted but ignored if there is a YYY, YYYY or Y,YYY field. If CC is used with YY or Y then the result is computed as that year in the specified century. If the century is specified but the year is not, the first year of the century is assumed.

  • In to_timestamp and to_date, weekday names or numbers (DAY, D, and related field types) are accepted but are ignored for purposes of computing the result. The same is true for quarter (Q) fields.

  • In to_timestamp and to_date, an ISO 8601 week-numbering date (as distinct from a Gregorian date) can be specified in one of two ways:

    • Year, week number, and weekday: for example to_date('2006-42-4', 'IYYY-IW-ID') returns the date 2006-10-19. If you omit the weekday it is assumed to be 1 (Monday).

    • Year and day of year: for example to_date('2006-291', 'IYYY-IDDD') also returns 2006-10-19.

    Attempting to enter a date using a mixture of ISO 8601 week-numbering fields and Gregorian date fields is nonsensical, and will cause an error. In the context of an ISO 8601 week-numbering year, the concept of a “month” or “day of month” has no meaning. In the context of a Gregorian year, the ISO week has no meaning.

    Caution

    While to_date will reject a mixture of Gregorian and ISO week-numbering date fields, to_char will not, since output format specifications like YYYY-MM-DD (IYYY-IDDD) can be useful. But avoid writing something like IYYY-MM-DD; that would yield surprising results near the start of the year. (See Section 9.9.1 for more information.)

  • In to_timestamp, millisecond (MS) or microsecond (US) fields are used as the seconds digits after the decimal point. For example to_timestamp('12.3', 'SS.MS') is not 3 milliseconds, but 300, because the conversion treats it as 12 + 0.3 seconds. So, for the format SS.MS, the input values 12.3, 12.30, and 12.300 specify the same number of milliseconds. To get three milliseconds, one must write 12.003, which the conversion treats as 12 + 0.003 = 12.003 seconds.

    Here is a more complex example: to_timestamp('15:12:02.020.001230', 'HH24:MI:SS.MS.US') is 15 hours, 12 minutes, and 2 seconds + 20 milliseconds + 1230 microseconds = 2.021230 seconds.

  • to_char(..., 'ID')'s day of the week numbering matches the extract(isodow from ...) function, but to_char(..., 'D')'s does not match extract(dow from ...)'s day numbering.

  • to_char(interval) formats HH and HH12 as shown on a 12-hour clock, for example zero hours and 36 hours both output as 12, while HH24 outputs the full hour value, which can exceed 23 in an interval value.

Table 9.28 shows the template patterns available for formatting numeric values.

Table 9.28. Template Patterns for Numeric Formatting

Pattern | Description
9 | digit position (can be dropped if insignificant)
0 | digit position (will not be dropped, even if insignificant)
. (period) | decimal point
, (comma) | group (thousands) separator
PR | negative value in angle brackets
S | sign anchored to number (uses locale)
L | currency symbol (uses locale)
D | decimal point (uses locale)
G | group separator (uses locale)
MI | minus sign in specified position (if number < 0)
PL | plus sign in specified position (if number > 0)
SG | plus/minus sign in specified position
RN | Roman numeral (input between 1 and 3999)
TH or th | ordinal number suffix
V | shift specified number of digits (see notes)
EEEE | exponent for scientific notation

Usage notes for numeric formatting:

  • 0 specifies a digit position that will always be printed, even if it contains a leading/trailing zero. 9 also specifies a digit position, but if it is a leading zero then it will be replaced by a space, while if it is a trailing zero and fill mode is specified then it will be deleted. (For to_number(), these two pattern characters are equivalent.)

  • The pattern characters S, L, D, and G represent the sign, currency symbol, decimal point, and thousands separator characters defined by the current locale (see lc_monetary and lc_numeric). The pattern characters period and comma represent those exact characters, with the meanings of decimal point and thousands separator, regardless of locale.

  • If no explicit provision is made for a sign in to_char()'s pattern, one column will be reserved for the sign, and it will be anchored to (appear just left of) the number. If S appears just left of some 9's, it will likewise be anchored to the number.

  • A sign formatted using SG, PL, or MI is not anchored to the number; for example, to_char(-12, 'MI9999') produces '-  12' but to_char(-12, 'S9999') produces '  -12'. (The Oracle implementation does not allow the use of MI before 9, but rather requires that 9 precede MI.)

  • TH does not convert values less than zero and does not convert fractional numbers.

  • PL, SG, and TH are PostgreSQL extensions.

  • In to_number, if non-data template patterns such as L or TH are used, the corresponding number of input characters are skipped, whether or not they match the template pattern, unless they are data characters (that is, digits, sign, decimal point, or comma). For example, TH would skip two non-data characters.

  • V with to_char multiplies the input values by 10^n, where n is the number of digits following V. V with to_number divides in a similar manner. to_char and to_number do not support the use of V combined with a decimal point (e.g., 99.9V99 is not allowed).

  • EEEE (scientific notation) cannot be used in combination with any of the other formatting patterns or modifiers other than digit and decimal point patterns, and must be at the end of the format string (e.g., 9.99EEEE is a valid pattern).

Certain modifiers can be applied to any template pattern to alter its behavior. For example, FM99.99 is the 99.99 pattern with the FM modifier. Table 9.29 shows the modifier patterns for numeric formatting.

Table 9.29. Template Pattern Modifiers for Numeric Formatting

Modifier | Description | Example
FM prefix | fill mode (suppress trailing zeroes and padding blanks) | FM99.99
TH suffix | upper case ordinal number suffix | 999TH
th suffix | lower case ordinal number suffix | 999th

Table 9.30 shows some examples of the use of the to_char function.

Table 9.30. to_char Examples

Expression | Result
to_char(current_timestamp, 'Day, DD HH12:MI:SS') | 'Tuesday , 06 05:39:18'
to_char(current_timestamp, 'FMDay, FMDD HH12:MI:SS') | 'Tuesday, 6 05:39:18'
to_char(-0.1, '99.99') | ' -.10'
to_char(-0.1, 'FM9.99') | '-.1'
to_char(-0.1, 'FM90.99') | '-0.1'
to_char(0.1, '0.9') | ' 0.1'
to_char(12, '9990999.9') | ' 0012.0'
to_char(12, 'FM9990999.9') | '0012.'
to_char(485, '999') | ' 485'
to_char(-485, '999') | '-485'
to_char(485, '9 9 9') | ' 4 8 5'
to_char(1485, '9,999') | ' 1,485'
to_char(1485, '9G999') | ' 1 485'
to_char(148.5, '999.999') | ' 148.500'
to_char(148.5, 'FM999.999') | '148.5'
to_char(148.5, 'FM999.990') | '148.500'
to_char(148.5, '999D999') | ' 148,500'
to_char(3148.5, '9G999D999') | ' 3 148,500'
to_char(-485, '999S') | '485-'
to_char(-485, '999MI') | '485-'
to_char(485, '999MI') | '485 '
to_char(485, 'FM999MI') | '485'
to_char(485, 'PL999') | '+485'
to_char(485, 'SG999') | '+485'
to_char(-485, 'SG999') | '-485'
to_char(-485, '9SG99') | '4-85'
to_char(-485, '999PR') | '<485>'
to_char(485, 'L999') | 'DM 485'
to_char(485, 'RN') | ' CDLXXXV'
to_char(485, 'FMRN') | 'CDLXXXV'
to_char(5.2, 'FMRN') | 'V'
to_char(482, '999th') | ' 482nd'
to_char(485, '"Good number:"999') | 'Good number: 485'
to_char(485.8, '"Pre:"999" Post:" .999') | 'Pre: 485 Post: .800'
to_char(12, '99V999') | ' 12000'
to_char(12.4, '99V999') | ' 12400'
to_char(12.45, '99V9') | ' 125'
to_char(0.0004859, '9.99EEEE') | ' 4.86e-04'

Prev | Up | Next
9.7. Pattern Matching | Home | 9.9. Date/Time Functions and Operators


9.9. Date/Time Functions and Operators
Prev Up Chapter 9. Functions and Operators Home Next

9.9. Date/Time Functions and Operators

9.9.1. EXTRACT, date_part
9.9.2. date_trunc
9.9.3. date_bin
9.9.4. AT TIME ZONE
9.9.5. Current Date/Time
9.9.6. Delaying Execution

Table 9.32 shows the available functions for date/time value processing, with details appearing in the following subsections. Table 9.31 illustrates the behaviors of the basic arithmetic operators (+, *, etc.). For formatting functions, refer to Section 9.8. You should be familiar with the background information on date/time data types from Section 8.5.

In addition, the usual comparison operators shown in Table 9.1 are available for the date/time types. Dates and timestamps (with or without time zone) are all comparable, while times (with or without time zone) and intervals can only be compared to other values of the same data type. When comparing a timestamp without time zone to a timestamp with time zone, the former value is assumed to be given in the time zone specified by the TimeZone configuration parameter, and is rotated to UTC for comparison to the latter value (which is already in UTC internally). Similarly, a date value is assumed to represent midnight in the TimeZone zone when comparing it to a timestamp.

All the functions and operators described below that take time or timestamp inputs actually come in two variants: one that takes time with time zone or timestamp with time zone, and one that takes time without time zone or timestamp without time zone. For brevity, these variants are not shown separately. Also, the + and * operators come in commutative pairs (for example both date + integer and integer + date); we show only one of each such pair.

Table 9.31. Date/Time Operators

Operator

Description

Example(s)

date + integer → date

Add a number of days to a date

date '2001-09-28' + 7 → 2001-10-05

date + interval → timestamp

Add an interval to a date

date '2001-09-28' + interval '1 hour' → 2001-09-28 01:00:00

date + time → timestamp

Add a time-of-day to a date

date '2001-09-28' + time '03:00' → 2001-09-28 03:00:00

interval + interval → interval

Add intervals

interval '1 day' + interval '1 hour' → 1 day 01:00:00

timestamp + interval → timestamp

Add an interval to a timestamp

timestamp '2001-09-28 01:00' + interval '23 hours' → 2001-09-29 00:00:00

time + interval → time

Add an interval to a time

time '01:00' + interval '3 hours' → 04:00:00

- interval → interval

Negate an interval

- interval '23 hours' → -23:00:00

date - date → integer

Subtract dates, producing the number of days elapsed

date '2001-10-01' - date '2001-09-28' → 3

date - integer → date

Subtract a number of days from a date

date '2001-10-01' - 7 → 2001-09-24

date - interval → timestamp

Subtract an interval from a date

date '2001-09-28' - interval '1 hour' → 2001-09-27 23:00:00

time - time → interval

Subtract times

time '05:00' - time '03:00' → 02:00:00

time - interval → time

Subtract an interval from a time

time '05:00' - interval '2 hours' → 03:00:00

timestamp - interval → timestamp

Subtract an interval from a timestamp

timestamp '2001-09-28 23:00' - interval '23 hours' → 2001-09-28 00:00:00

interval - interval → interval

Subtract intervals

interval '1 day' - interval '1 hour' → 1 day -01:00:00

timestamp - timestamp → interval

Subtract timestamps (converting 24-hour intervals into days, similarly to justify_hours())

timestamp '2001-09-29 03:00' - timestamp '2001-07-27 12:00' → 63 days 15:00:00

interval * double precision → interval

Multiply an interval by a scalar

interval '1 second' * 900 → 00:15:00

interval '1 day' * 21 → 21 days

interval '1 hour' * 3.5 → 03:30:00

interval / double precision → interval

Divide an interval by a scalar

interval '1 hour' / 1.5 → 00:40:00

Table 9.32. Date/Time Functions

Function

Description

Example(s)

age ( timestamp, timestamp ) → interval

Subtract arguments, producing a “symbolic” result that uses years and months, rather than just days

age(timestamp '2001-04-10', timestamp '1957-06-13') → 43 years 9 mons 27 days

age ( timestamp ) → interval

Subtract argument from current_date (at midnight)

age(timestamp '1957-06-13') → 62 years 6 mons 10 days

clock_timestamp ( ) → timestamp with time zone

Current date and time (changes during statement execution); see Section 9.9.5

clock_timestamp() → 2019-12-23 14:39:53.662522-05

current_date → date

Current date; see Section 9.9.5

current_date → 2019-12-23

current_time → time with time zone

Current time of day; see Section 9.9.5

current_time → 14:39:53.662522-05

current_time ( integer ) → time with time zone

Current time of day, with limited precision; see Section 9.9.5

current_time(2) → 14:39:53.66-05

current_timestamp → timestamp with time zone

Current date and time (start of current transaction); see Section 9.9.5

current_timestamp → 2019-12-23 14:39:53.662522-05

current_timestamp ( integer ) → timestamp with time zone

Current date and time (start of current transaction), with limited precision; see Section 9.9.5

current_timestamp(0) → 2019-12-23 14:39:53-05

date_bin ( interval, timestamp, timestamp ) → timestamp

Bin input into specified interval aligned with specified origin; see Section 9.9.3

date_bin('15 minutes', timestamp '2001-02-16 20:38:40', timestamp '2001-02-16 20:05:00') → 2001-02-16 20:35:00

date_part ( text, timestamp ) → double precision

Get timestamp subfield (equivalent to extract); see Section 9.9.1

date_part('hour', timestamp '2001-02-16 20:38:40') → 20

date_part ( text, interval ) → double precision

Get interval subfield (equivalent to extract); see Section 9.9.1

date_part('month', interval '2 years 3 months') → 3

date_trunc ( text, timestamp ) → timestamp

Truncate to specified precision; see Section 9.9.2

date_trunc('hour', timestamp '2001-02-16 20:38:40') → 2001-02-16 20:00:00

date_trunc ( text, timestamp with time zone, text ) → timestamp with time zone

Truncate to specified precision in the specified time zone; see Section 9.9.2

date_trunc('day', timestamptz '2001-02-16 20:38:40+00', 'Australia/Sydney') → 2001-02-16 13:00:00+00

date_trunc ( text, interval ) → interval

Truncate to specified precision; see Section 9.9.2

date_trunc('hour', interval '2 days 3 hours 40 minutes') → 2 days 03:00:00

extract ( field from timestamp ) → numeric

Get timestamp subfield; see Section 9.9.1

extract(hour from timestamp '2001-02-16 20:38:40') → 20

extract ( field from interval ) → numeric

Get interval subfield; see Section 9.9.1

extract(month from interval '2 years 3 months') → 3

isfinite ( date ) → boolean

Test for finite date (not +/-infinity)

isfinite(date '2001-02-16') → true

isfinite ( timestamp ) → boolean

Test for finite timestamp (not +/-infinity)

isfinite(timestamp 'infinity') → false

isfinite ( interval ) → boolean

Test for finite interval (currently always true)

isfinite(interval '4 hours') → true

justify_days ( interval ) → interval

Adjust interval so 30-day time periods are represented as months

justify_days(interval '35 days') → 1 mon 5 days

justify_hours ( interval ) → interval

Adjust interval so 24-hour time periods are represented as days

justify_hours(interval '27 hours') → 1 day 03:00:00

justify_interval ( interval ) → interval

Adjust interval using justify_days and justify_hours, with additional sign adjustments

justify_interval(interval '1 mon -1 hour') → 29 days 23:00:00

localtime → time

Current time of day; see Section 9.9.5

localtime → 14:39:53.662522

localtime ( integer ) → time

Current time of day, with limited precision; see Section 9.9.5

localtime(0) → 14:39:53

localtimestamp → timestamp

Current date and time (start of current transaction); see Section 9.9.5

localtimestamp → 2019-12-23 14:39:53.662522

localtimestamp ( integer ) → timestamp

Current date and time (start of current transaction), with limited precision; see Section 9.9.5

localtimestamp(2) → 2019-12-23 14:39:53.66

make_date ( year int, month int, day int ) → date

Create date from year, month and day fields (negative years signify BC)

make_date(2013, 7, 15) → 2013-07-15

make_interval ( [ years int [, months int [, weeks int [, days int [, hours int [, mins int [, secs double precision ]]]]]]] ) → interval

Create interval from years, months, weeks, days, hours, minutes and seconds fields, each of which can default to zero

make_interval(days => 10) → 10 days

make_time ( hour int, min int, sec double precision ) → time

Create time from hour, minute and seconds fields

make_time(8, 15, 23.5) → 08:15:23.5

make_timestamp ( year int, month int, day int, hour int, min int, sec double precision ) → timestamp

Create timestamp from year, month, day, hour, minute and seconds fields (negative years signify BC)

make_timestamp(2013, 7, 15, 8, 15, 23.5) → 2013-07-15 08:15:23.5

make_timestamptz ( year int, month int, day int, hour int, min int, sec double precision [, timezone text ] ) → timestamp with time zone

Create timestamp with time zone from year, month, day, hour, minute and seconds fields (negative years signify BC). If timezone is not specified, the current time zone is used; the examples assume the session time zone is Europe/London

make_timestamptz(2013, 7, 15, 8, 15, 23.5) → 2013-07-15 08:15:23.5+01

make_timestamptz(2013, 7, 15, 8, 15, 23.5, 'America/New_York') → 2013-07-15 13:15:23.5+01

now ( ) → timestamp with time zone

Current date and time (start of current transaction); see Section 9.9.5

now() → 2019-12-23 14:39:53.662522-05

statement_timestamp ( ) → timestamp with time zone

Current date and time (start of current statement); see Section 9.9.5

statement_timestamp() → 2019-12-23 14:39:53.662522-05

timeofday ( ) → text

Current date and time (like clock_timestamp, but as a text string); see Section 9.9.5

timeofday() → Mon Dec 23 14:39:53.662522 2019 EST

transaction_timestamp ( ) → timestamp with time zone

Current date and time (start of current transaction); see Section 9.9.5

transaction_timestamp() → 2019-12-23 14:39:53.662522-05

to_timestamp ( double precision ) → timestamp with time zone

Convert Unix epoch (seconds since 1970-01-01 00:00:00+00) to timestamp with time zone

to_timestamp(1284352323) → 2010-09-13 04:32:03+00

In addition to these functions, the SQL OVERLAPS operator is supported:

(start1, end1) OVERLAPS (start2, end2)
(start1, length1) OVERLAPS (start2, length2)

This expression yields true when two time periods (defined by their endpoints) overlap, false when they do not overlap. The endpoints can be specified as pairs of dates, times, or time stamps; or as a date, time, or time stamp followed by an interval. When a pair of values is provided, either the start or the end can be written first; OVERLAPS automatically takes the earlier value of the pair as the start. Each time period is considered to represent the half-open interval start <= time < end, unless start and end are equal in which case it represents that single time instant. This means for instance that two time periods with only an endpoint in common do not overlap.

SELECT (DATE '2001-02-16', DATE '2001-12-21') OVERLAPS
       (DATE '2001-10-30', DATE '2002-10-30');
Result: true
SELECT (DATE '2001-02-16', INTERVAL '100 days') OVERLAPS
       (DATE '2001-10-30', DATE '2002-10-30');
Result: false
SELECT (DATE '2001-10-29', DATE '2001-10-30') OVERLAPS
       (DATE '2001-10-30', DATE '2001-10-31');
Result: false
SELECT (DATE '2001-10-30', DATE '2001-10-30') OVERLAPS
       (DATE '2001-10-30', DATE '2001-10-31');
Result: true

When adding an interval value to (or subtracting an interval value from) a timestamp with time zone value, the days component advances or decrements the date of the timestamp with time zone by the indicated number of days, keeping the time of day the same. Across daylight saving time changes (when the session time zone is set to a time zone that recognizes DST), this means interval '1 day' does not necessarily equal interval '24 hours'. For example, with the session time zone set to America/Denver:

SELECT timestamp with time zone '2005-04-02 12:00:00-07' + interval '1 day';
Result: 2005-04-03 12:00:00-06
SELECT timestamp with time zone '2005-04-02 12:00:00-07' + interval '24 hours';
Result: 2005-04-03 13:00:00-06

This happens because an hour was skipped due to a change in daylight saving time at 2005-04-03 02:00:00 in time zone America/Denver.

Note there can be ambiguity in the months field returned by age because different months have different numbers of days. PostgreSQL's approach uses the month from the earlier of the two dates when calculating partial months. For example, age('2004-06-01', '2004-04-30') uses April to yield 1 mon 1 day, while using May would yield 1 mon 2 days because May has 31 days, while April has only 30.

Subtraction of dates and timestamps can also be complex. One conceptually simple way to perform subtraction is to convert each value to a number of seconds using EXTRACT(EPOCH FROM ...), then subtract the results; this produces the number of seconds between the two values. This will adjust for the number of days in each month, timezone changes, and daylight saving time adjustments. Subtraction of date or timestamp values with the “-” operator returns the number of days (24-hours) and hours/minutes/seconds between the values, making the same adjustments. The age function returns years, months, days, and hours/minutes/seconds, performing field-by-field subtraction and then adjusting for negative field values. The following queries illustrate the differences in these approaches. The sample results were produced with timezone = 'US/Eastern'; there is a daylight saving time change between the two dates used:

SELECT EXTRACT(EPOCH FROM timestamptz '2013-07-01 12:00:00') -
       EXTRACT(EPOCH FROM timestamptz '2013-03-01 12:00:00');
Result: 10537200
SELECT (EXTRACT(EPOCH FROM timestamptz '2013-07-01 12:00:00') -
        EXTRACT(EPOCH FROM timestamptz '2013-03-01 12:00:00'))
        / 60 / 60 / 24;
Result: 121.958333333333
SELECT timestamptz '2013-07-01 12:00:00' - timestamptz '2013-03-01 12:00:00';
Result: 121 days 23:00:00
SELECT age(timestamptz '2013-07-01 12:00:00', timestamptz '2013-03-01 12:00:00');
Result: 4 mons

9.9.1. EXTRACT, date_part

EXTRACT(field FROM source)

The extract function retrieves subfields such as year or hour from date/time values. source must be a value expression of type timestamp, time, or interval. (Expressions of type date are cast to timestamp and can therefore be used as well.) field is an identifier or string that selects what field to extract from the source value. The extract function returns values of type numeric. The following are valid field names:

century

The century

SELECT EXTRACT(CENTURY FROM TIMESTAMP '2000-12-16 12:21:13');
Result: 20
SELECT EXTRACT(CENTURY FROM TIMESTAMP '2001-02-16 20:38:40');
Result: 21

The first century starts at 0001-01-01 00:00:00 AD, although they did not know it at the time. This definition applies to all Gregorian calendar countries. There is no century number 0, you go from -1 century to 1 century. If you disagree with this, please write your complaint to: Pope, Cathedral Saint-Peter of Roma, Vatican.

day

For timestamp values, the day (of the month) field (1–31) ; for interval values, the number of days

SELECT EXTRACT(DAY FROM TIMESTAMP '2001-02-16 20:38:40');
Result: 16

SELECT EXTRACT(DAY FROM INTERVAL '40 days 1 minute');
Result: 40
decade

The year field divided by 10

SELECT EXTRACT(DECADE FROM TIMESTAMP '2001-02-16 20:38:40');
Result: 200
dow

The day of the week as Sunday (0) to Saturday (6)

SELECT EXTRACT(DOW FROM TIMESTAMP '2001-02-16 20:38:40');
Result: 5

Note that extract's day of the week numbering differs from that of the to_char(..., 'D') function.

doy

The day of the year (1–365/366)

SELECT EXTRACT(DOY FROM TIMESTAMP '2001-02-16 20:38:40');
Result: 47
epoch

For timestamp with time zone values, the number of seconds since 1970-01-01 00:00:00 UTC (negative for timestamps before that); for date and timestamp values, the nominal number of seconds since 1970-01-01 00:00:00, without regard to timezone or daylight-savings rules; for interval values, the total number of seconds in the interval

SELECT EXTRACT(EPOCH FROM TIMESTAMP WITH TIME ZONE '2001-02-16 20:38:40.12-08');
Result: 982384720.12

SELECT EXTRACT(EPOCH FROM TIMESTAMP '2001-02-16 20:38:40.12');
Result: 982355920.12

SELECT EXTRACT(EPOCH FROM INTERVAL '5 days 3 hours');
Result: 442800

You can convert an epoch value back to a timestamp with time zone with to_timestamp:

SELECT to_timestamp(982384720.12);
Result: 2001-02-17 04:38:40.12+00

Beware that applying to_timestamp to an epoch extracted from a date or timestamp value could produce a misleading result: the result will effectively assume that the original value had been given in UTC, which might not be the case.

hour

The hour field (0–23)

SELECT EXTRACT(HOUR FROM TIMESTAMP '2001-02-16 20:38:40');
Result: 20
isodow

The day of the week as Monday (1) to Sunday (7)

SELECT EXTRACT(ISODOW FROM TIMESTAMP '2001-02-18 20:38:40');
Result: 7

This is identical to dow except for Sunday. This matches the ISO 8601 day of the week numbering.

isoyear

The ISO 8601 week-numbering year that the date falls in (not applicable to intervals)

SELECT EXTRACT(ISOYEAR FROM DATE '2006-01-01');
Result: 2005
SELECT EXTRACT(ISOYEAR FROM DATE '2006-01-02');
Result: 2006

Each ISO 8601 week-numbering year begins with the Monday of the week containing the 4th of January, so in early January or late December the ISO year may be different from the Gregorian year. See the week field for more information.

This field is not available in PostgreSQL releases prior to 8.3.

julian

The Julian Date corresponding to the date or timestamp (not applicable to intervals). Timestamps that are not local midnight result in a fractional value. See Section B.7 for more information.

SELECT EXTRACT(JULIAN FROM DATE '2006-01-01');
Result: 2453737
SELECT EXTRACT(JULIAN FROM TIMESTAMP '2006-01-01 12:00');
Result: 2453737.50000000000000000000
microseconds

The seconds field, including fractional parts, multiplied by 1 000 000; note that this includes full seconds

SELECT EXTRACT(MICROSECONDS FROM TIME '17:12:28.5');
Result: 28500000
millennium

The millennium

SELECT EXTRACT(MILLENNIUM FROM TIMESTAMP '2001-02-16 20:38:40');
Result: 3

Years in the 1900s are in the second millennium. The third millennium started January 1, 2001.

milliseconds

The seconds field, including fractional parts, multiplied by 1000. Note that this includes full seconds.

SELECT EXTRACT(MILLISECONDS FROM TIME '17:12:28.5');
Result: 28500
minute

The minutes field (0–59)

SELECT EXTRACT(MINUTE FROM TIMESTAMP '2001-02-16 20:38:40');
Result: 38
month

For timestamp values, the number of the month within the year (1–12) ; for interval values, the number of months, modulo 12 (0–11)

SELECT EXTRACT(MONTH FROM TIMESTAMP '2001-02-16 20:38:40');
Result: 2

SELECT EXTRACT(MONTH FROM INTERVAL '2 years 3 months');
Result: 3

SELECT EXTRACT(MONTH FROM INTERVAL '2 years 13 months');
Result: 1
quarter

The quarter of the year (1–4) that the date is in

SELECT EXTRACT(QUARTER FROM TIMESTAMP '2001-02-16 20:38:40');
Result: 1
second

The seconds field, including any fractional seconds

SELECT EXTRACT(SECOND FROM TIMESTAMP '2001-02-16 20:38:40');
Result: 40

SELECT EXTRACT(SECOND FROM TIME '17:12:28.5');
Result: 28.5
timezone

The time zone offset from UTC, measured in seconds. Positive values correspond to time zones east of UTC, negative values to zones west of UTC. (Technically, PostgreSQL does not use UTC because leap seconds are not handled.)

timezone_hour

The hour component of the time zone offset

timezone_minute

The minute component of the time zone offset

week

The number of the ISO 8601 week-numbering week of the year. By definition, ISO weeks start on Mondays and the first week of a year contains January 4 of that year. In other words, the first Thursday of a year is in week 1 of that year.

In the ISO week-numbering system, it is possible for early-January dates to be part of the 52nd or 53rd week of the previous year, and for late-December dates to be part of the first week of the next year. For example, 2005-01-01 is part of the 53rd week of year 2004, and 2006-01-01 is part of the 52nd week of year 2005, while 2012-12-31 is part of the first week of 2013. It's recommended to use the isoyear field together with week to get consistent results.

SELECT EXTRACT(WEEK FROM TIMESTAMP '2001-02-16 20:38:40');
Result: 7
year

The year field. Keep in mind there is no 0 AD, so subtracting BC years from AD years should be done with care.

SELECT EXTRACT(YEAR FROM TIMESTAMP '2001-02-16 20:38:40');
Result: 2001

Note

When the input value is +/-Infinity, extract returns +/-Infinity for monotonically-increasing fields (epoch, julian, year, isoyear, decade, century, and millennium). For other fields, NULL is returned. PostgreSQL versions before 9.6 returned zero for all cases of infinite input.

The extract function is primarily intended for computational processing. For formatting date/time values for display, see Section 9.8.

The date_part function is modeled on the traditional Ingres equivalent to the SQL-standard function extract:

date_part('field', source)

Note that here the field parameter needs to be a string value, not a name. The valid field names for date_part are the same as for extract. For historical reasons, the date_part function returns values of type double precision. This can result in a loss of precision in certain uses. Using extract is recommended instead.

SELECT date_part('day', TIMESTAMP '2001-02-16 20:38:40');
Result: 16

SELECT date_part('hour', INTERVAL '4 hours 3 minutes');
Result: 4

9.9.2. date_trunc

The function date_trunc is conceptually similar to the trunc function for numbers.

date_trunc(field, source [, time_zone ])

source is a value expression of type timestamp, timestamp with time zone, or interval. (Values of type date and time are cast automatically to timestamp or interval, respectively.) field selects to which precision to truncate the input value. The return value is likewise of type timestamp, timestamp with time zone, or interval, and it has all fields that are less significant than the selected one set to zero (or one, for day and month).

Valid values for field are:

microseconds
milliseconds
second
minute
hour
day
week
month
quarter
year
decade
century
millennium

When the input value is of type timestamp with time zone, the truncation is performed with respect to a particular time zone; for example, truncation to day produces a value that is midnight in that zone. By default, truncation is done with respect to the current TimeZone setting, but the optional time_zone argument can be provided to specify a different time zone. The time zone name can be specified in any of the ways described in Section 8.5.3.

A time zone cannot be specified when processing timestamp without time zone or interval inputs. These are always taken at face value.

Examples (assuming the local time zone is America/New_York):

SELECT date_trunc('hour', TIMESTAMP '2001-02-16 20:38:40');
Result: 2001-02-16 20:00:00

SELECT date_trunc('year', TIMESTAMP '2001-02-16 20:38:40');
Result: 2001-01-01 00:00:00

SELECT date_trunc('day', TIMESTAMP WITH TIME ZONE '2001-02-16 20:38:40+00');
Result: 2001-02-16 00:00:00-05

SELECT date_trunc('day', TIMESTAMP WITH TIME ZONE '2001-02-16 20:38:40+00', 'Australia/Sydney');
Result: 2001-02-16 08:00:00-05

SELECT date_trunc('hour', INTERVAL '3 days 02:47:33');
Result: 3 days 02:00:00

9.9.3. date_bin

The function date_bin “bins” the input timestamp into the specified interval (the stride) aligned with a specified origin.

date_bin(stride, source, origin)

source is a value expression of type timestamp or timestamp with time zone. (Values of type date are cast automatically to timestamp.) stride is a value expression of type interval. The return value is likewise of type timestamp or timestamp with time zone, and it marks the beginning of the bin into which the source is placed.

Examples:

SELECT date_bin('15 minutes', TIMESTAMP '2020-02-11 15:44:17', TIMESTAMP '2001-01-01');
Result: 2020-02-11 15:30:00

SELECT date_bin('15 minutes', TIMESTAMP '2020-02-11 15:44:17', TIMESTAMP '2001-01-01 00:02:30');
Result: 2020-02-11 15:32:30

In the case of full units (1 minute, 1 hour, etc.), it gives the same result as the analogous date_trunc call, but the difference is that date_bin can truncate to an arbitrary interval.

The stride interval must be greater than zero and cannot contain units of month or larger.

9.9.4. AT TIME ZONE

The AT TIME ZONE operator converts time stamp without time zone to/from time stamp with time zone, and time with time zone values to different time zones. Table 9.33 shows its variants.

Table 9.33. AT TIME ZONE Variants

Operator

Description

Example(s)

timestamp without time zone AT TIME ZONE zone → timestamp with time zone

Converts given time stamp without time zone to time stamp with time zone, assuming the given value is in the named time zone.

timestamp '2001-02-16 20:38:40' at time zone 'America/Denver' → 2001-02-17 03:38:40+00

timestamp with time zone AT TIME ZONE zone → timestamp without time zone

Converts given time stamp with time zone to time stamp without time zone, as the time would appear in that zone.

timestamp with time zone '2001-02-16 20:38:40-05' at time zone 'America/Denver' → 2001-02-16 18:38:40

time with time zone AT TIME ZONE zone → time with time zone

Converts given time with time zone to a new time zone. Since no date is supplied, this uses the currently active UTC offset for the named destination zone.

time with time zone '05:34:17-05' at time zone 'UTC' → 10:34:17+00

In these expressions, the desired time zone zone can be specified either as a text value (e.g., 'America/Los_Angeles') or as an interval (e.g., INTERVAL '-08:00'). In the text case, a time zone name can be specified in any of the ways described in Section 8.5.3. The interval case is only useful for zones that have fixed offsets from UTC, so it is not very common in practice.

Examples (assuming the current TimeZone setting is America/Los_Angeles):

SELECT TIMESTAMP '2001-02-16 20:38:40' AT TIME ZONE 'America/Denver';
Result: 2001-02-16 19:38:40-08

SELECT TIMESTAMP WITH TIME ZONE '2001-02-16 20:38:40-05' AT TIME ZONE 'America/Denver';
Result: 2001-02-16 18:38:40

SELECT TIMESTAMP '2001-02-16 20:38:40' AT TIME ZONE 'Asia/Tokyo' AT TIME ZONE 'America/Chicago';
Result: 2001-02-16 05:38:40

The first example adds a time zone to a value that lacks it, and displays the value using the current TimeZone setting. The second example shifts the time stamp with time zone value to the specified time zone, and returns the value without a time zone. This allows storage and display of values different from the current TimeZone setting. The third example converts Tokyo time to Chicago time.

The function timezone(zone, timestamp) is equivalent to the SQL-conforming construct timestamp AT TIME ZONE zone.

9.9.5. Current Date/Time

PostgreSQL provides a number of functions that return values related to the current date and time. These SQL-standard functions all return values based on the start time of the current transaction:

CURRENT_DATE
CURRENT_TIME
CURRENT_TIMESTAMP
CURRENT_TIME(precision)
CURRENT_TIMESTAMP(precision)
LOCALTIME
LOCALTIMESTAMP
LOCALTIME(precision)
LOCALTIMESTAMP(precision)

CURRENT_TIME and CURRENT_TIMESTAMP deliver values with time zone; LOCALTIME and LOCALTIMESTAMP deliver values without time zone.

CURRENT_TIME, CURRENT_TIMESTAMP, LOCALTIME, and LOCALTIMESTAMP can optionally take a precision parameter, which causes the result to be rounded to that many fractional digits in the seconds field. Without a precision parameter, the result is given to the full available precision.

Some examples:

SELECT CURRENT_TIME;
Result: 14:39:53.662522-05

SELECT CURRENT_DATE;
Result: 2019-12-23

SELECT CURRENT_TIMESTAMP;
Result: 2019-12-23 14:39:53.662522-05

SELECT CURRENT_TIMESTAMP(2);
Result: 2019-12-23 14:39:53.66-05

SELECT LOCALTIMESTAMP;
Result: 2019-12-23 14:39:53.662522

Since these functions return the start time of the current transaction, their values do not change during the transaction. This is considered a feature: the intent is to allow a single transaction to have a consistent notion of the “current” time, so that multiple modifications within the same transaction bear the same time stamp.

Note

Other database systems might advance these values more frequently.

PostgreSQL also provides functions that return the start time of the current statement, as well as the actual current time at the instant the function is called. The complete list of non-SQL-standard time functions is:

transaction_timestamp()
statement_timestamp()
clock_timestamp()
timeofday()
now()

transaction_timestamp() is equivalent to CURRENT_TIMESTAMP, but is named to clearly reflect what it returns. statement_timestamp() returns the start time of the current statement (more specifically, the time of receipt of the latest command message from the client). statement_timestamp() and transaction_timestamp() return the same value during the first command of a transaction, but might differ during subsequent commands. clock_timestamp() returns the actual current time, and therefore its value changes even within a single SQL command. timeofday() is a historical PostgreSQL function. Like clock_timestamp(), it returns the actual current time, but as a formatted text string rather than a timestamp with time zone value. now() is a traditional PostgreSQL equivalent to transaction_timestamp().

All the date/time data types also accept the special literal value now to specify the current date and time (again, interpreted as the transaction start time). Thus, the following three all return the same result:

SELECT CURRENT_TIMESTAMP;
SELECT now();
SELECT TIMESTAMP 'now';  -- but see tip below

Tip

Do not use the third form when specifying a value to be evaluated later, for example in a DEFAULT clause for a table column. The system will convert now to a timestamp as soon as the constant is parsed, so that when the default value is needed, the time of the table creation would be used! The first two forms will not be evaluated until the default value is used, because they are function calls. Thus they will give the desired behavior of defaulting to the time of row insertion. (See also Section 8.5.1.4.)

9.9.6. Delaying Execution

The following functions are available to delay execution of the server process:

pg_sleep ( double precision )
pg_sleep_for ( interval )
pg_sleep_until ( timestamp with time zone )

pg_sleep makes the current session's process sleep until the given number of seconds have elapsed. Fractional-second delays can be specified. pg_sleep_for is a convenience function to allow the sleep time to be specified as an interval. pg_sleep_until is a convenience function for when a specific wake-up time is desired. For example:

SELECT pg_sleep(1.5);
SELECT pg_sleep_for('5 minutes');
SELECT pg_sleep_until('tomorrow 03:00');

Note

The effective resolution of the sleep interval is platform-specific; 0.01 seconds is a common value. The sleep delay will be at least as long as specified. It might be longer depending on factors such as server load. In particular, pg_sleep_until is not guaranteed to wake up exactly at the specified time, but it will not wake up any earlier.

Warning

Make sure that your session does not hold more locks than necessary when calling pg_sleep or its variants. Otherwise other sessions might have to wait for your sleeping process, slowing down the entire system.

Prev | Up | Next
9.8. Data Type Formatting Functions | Home | 9.10. Enum Support Functions


9.10. Enum Support Functions
Prev Up Chapter 9. Functions and Operators Home Next

9.10. Enum Support Functions

For enum types (described in Section 8.7), there are several functions that allow cleaner programming without hard-coding particular values of an enum type. These are listed in Table 9.34. The examples assume an enum type created as:

CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow', 'green', 'blue', 'purple');

Table 9.34. Enum Support Functions

Function

Description

Example(s)

enum_first ( anyenum ) → anyenum

Returns the first value of the input enum type.

enum_first(null::rainbow) → red

enum_last ( anyenum ) → anyenum

Returns the last value of the input enum type.

enum_last(null::rainbow) → purple

enum_range ( anyenum ) → anyarray

Returns all values of the input enum type in an ordered array.

enum_range(null::rainbow) → {red,orange,yellow,​green,blue,purple}

enum_range ( anyenum, anyenum ) → anyarray

Returns the range between the two given enum values, as an ordered array. The values must be from the same enum type. If the first parameter is null, the result will start with the first value of the enum type. If the second parameter is null, the result will end with the last value of the enum type.

enum_range('orange'::rainbow, 'green'::rainbow) → {orange,yellow,green}

enum_range(NULL, 'green'::rainbow) → {red,orange,​yellow,green}

enum_range('orange'::rainbow, NULL) → {orange,yellow,green,​blue,purple}

Notice that except for the two-argument form of enum_range, these functions disregard the specific value passed to them; they care only about its declared data type. Either null or a specific value of the type can be passed, with the same result. It is more common to apply these functions to a table column or function argument than to a hardwired type name as used in the examples.

Prev | Up | Next
9.9. Date/Time Functions and Operators | Home | 9.11. Geometric Functions and Operators


9.11. Geometric Functions and Operators
Prev Up Chapter 9. Functions and Operators Home Next

9.11. Geometric Functions and Operators

The geometric types point, box, lseg, line, path, polygon, and circle have a large set of native support functions and operators, shown in Table 9.35, Table 9.36, and Table 9.37.

Table 9.35. Geometric Operators

Operator

Description

Example(s)

geometric_type + point → geometric_type

Adds the coordinates of the second point to those of each point of the first argument, thus performing translation. Available for point, box, path, circle.

box '(1,1),(0,0)' + point '(2,0)' → (3,1),(2,0)

path + path → path

Concatenates two open paths (returns NULL if either path is closed).

path '[(0,0),(1,1)]' + path '[(2,2),(3,3),(4,4)]' → [(0,0),(1,1),(2,2),(3,3),(4,4)]

geometric_type - point → geometric_type

Subtracts the coordinates of the second point from those of each point of the first argument, thus performing translation. Available for point, box, path, circle.

box '(1,1),(0,0)' - point '(2,0)' → (-1,1),(-2,0)

geometric_type * point → geometric_type

Multiplies each point of the first argument by the second point (treating a point as being a complex number represented by real and imaginary parts, and performing standard complex multiplication). If one interprets the second point as a vector, this is equivalent to scaling the object's size and distance from the origin by the length of the vector, and rotating it counterclockwise around the origin by the vector's angle from the x axis. Available for point, box,[a] path, circle.

path '((0,0),(1,0),(1,1))' * point '(3.0,0)' → ((0,0),(3,0),(3,3))

path '((0,0),(1,0),(1,1))' * point(cosd(45), sind(45)) → ((0,0),​(0.7071067811865475,0.7071067811865475),​(0,1.414213562373095))

geometric_type / point → geometric_type

Divides each point of the first argument by the second point (treating a point as being a complex number represented by real and imaginary parts, and performing standard complex division). If one interprets the second point as a vector, this is equivalent to scaling the object's size and distance from the origin down by the length of the vector, and rotating it clockwise around the origin by the vector's angle from the x axis. Available for point, box,[a] path, circle.

path '((0,0),(1,0),(1,1))' / point '(2.0,0)' → ((0,0),(0.5,0),(0.5,0.5))

path '((0,0),(1,0),(1,1))' / point(cosd(45), sind(45)) → ((0,0),​(0.7071067811865476,-0.7071067811865476),​(1.4142135623730951,0))

@-@ geometric_type → double precision

Computes the total length. Available for lseg, path.

@-@ path '[(0,0),(1,0),(1,1)]' → 2

@@ geometric_type → point

Computes the center point. Available for box, lseg, polygon, circle.

@@ box '(2,2),(0,0)' → (1,1)

# geometric_type → integer

Returns the number of points. Available for path, polygon.

# path '((1,0),(0,1),(-1,0))' → 3

geometric_type # geometric_type → point

Computes the point of intersection, or NULL if there is none. Available for lseg, line.

lseg '[(0,0),(1,1)]' # lseg '[(1,0),(0,1)]' → (0.5,0.5)

box # box → box

Computes the intersection of two boxes, or NULL if there is none.

box '(2,2),(-1,-1)' # box '(1,1),(-2,-2)' → (1,1),(-1,-1)

geometric_type ## geometric_type → point

Computes the closest point to the first object on the second object. Available for these pairs of types: (point, box), (point, lseg), (point, line), (lseg, box), (lseg, lseg), (line, lseg).

point '(0,0)' ## lseg '[(2,0),(0,2)]' → (1,1)

geometric_type <-> geometric_type → double precision

Computes the distance between the objects. Available for all geometric types except polygon, for all combinations of point with another geometric type, and for these additional pairs of types: (box, lseg), (lseg, line), (polygon, circle) (and the commutator cases).

circle '<(0,0),1>' <-> circle '<(5,0),1>' → 3

geometric_type @> geometric_type → boolean

Does first object contain second? Available for these pairs of types: (box, point), (box, box), (path, point), (polygon, point), (polygon, polygon), (circle, point), (circle, circle).

circle '<(0,0),2>' @> point '(1,1)' → t

geometric_type <@ geometric_type → boolean

Is first object contained in or on second? Available for these pairs of types: (point, box), (point, lseg), (point, line), (point, path), (point, polygon), (point, circle), (box, box), (lseg, box), (lseg, line), (polygon, polygon), (circle, circle).

point '(1,1)' <@ circle '<(0,0),2>' → t

geometric_type && geometric_type → boolean

Do these objects overlap? (One point in common makes this true.) Available for box, polygon, circle.

box '(1,1),(0,0)' && box '(2,2),(0,0)' → t

geometric_type << geometric_type → boolean

Is first object strictly left of second? Available for point, box, polygon, circle.

circle '<(0,0),1>' << circle '<(5,0),1>' → t

geometric_type >> geometric_type → boolean

Is first object strictly right of second? Available for point, box, polygon, circle.

circle '<(5,0),1>' >> circle '<(0,0),1>' → t

geometric_type &< geometric_type → boolean

Does first object not extend to the right of second? Available for box, polygon, circle.

box '(1,1),(0,0)' &< box '(2,2),(0,0)' → t

geometric_type &> geometric_type → boolean

Does first object not extend to the left of second? Available for box, polygon, circle.

box '(3,3),(0,0)' &> box '(2,2),(0,0)' → t

geometric_type <<| geometric_type → boolean

Is first object strictly below second? Available for point, box, polygon, circle.

box '(3,3),(0,0)' <<| box '(5,5),(3,4)' → t

geometric_type |>> geometric_type → boolean

Is first object strictly above second? Available for point, box, polygon, circle.

box '(5,5),(3,4)' |>> box '(3,3),(0,0)' → t

geometric_type &<| geometric_type → boolean

Does first object not extend above second? Available for box, polygon, circle.

box '(1,1),(0,0)' &<| box '(2,2),(0,0)' → t

geometric_type |&> geometric_type → boolean

Does first object not extend below second? Available for box, polygon, circle.

box '(3,3),(0,0)' |&> box '(2,2),(0,0)' → t

box <^ box → boolean

Is first object below second (allows edges to touch)?

box '((1,1),(0,0))' <^ box '((2,2),(1,1))' → t

box >^ box → boolean

Is first object above second (allows edges to touch)?

box '((2,2),(1,1))' >^ box '((1,1),(0,0))' → t

geometric_type ?# geometric_type → boolean

Do these objects intersect? Available for these pairs of types: (box, box), (lseg, box), (lseg, lseg), (lseg, line), (line, box), (line, line), (path, path).

lseg '[(-1,0),(1,0)]' ?# box '(2,2),(-2,-2)' → t

?- line → boolean

?- lseg → boolean

Is line horizontal?

?- lseg '[(-1,0),(1,0)]' → t

point ?- point → boolean

Are points horizontally aligned (that is, have same y coordinate)?

point '(1,0)' ?- point '(0,0)' → t

?| line → boolean

?| lseg → boolean

Is line vertical?

?| lseg '[(-1,0),(1,0)]' → f

point ?| point → boolean

Are points vertically aligned (that is, have same x coordinate)?

point '(0,1)' ?| point '(0,0)' → t

line ?-| line → boolean

lseg ?-| lseg → boolean

Are lines perpendicular?

lseg '[(0,0),(0,1)]' ?-| lseg '[(0,0),(1,0)]' → t

line ?|| line → boolean

lseg ?|| lseg → boolean

Are lines parallel?

lseg '[(-1,0),(1,0)]' ?|| lseg '[(-1,2),(1,2)]' → t

geometric_type ~= geometric_type → boolean

Are these objects the same? Available for point, box, polygon, circle.

polygon '((0,0),(1,1))' ~= polygon '((1,1),(0,0))' → t

[a] “Rotating” a box with these operators only moves its corner points: the box is still considered to have sides parallel to the axes. Hence the box's size is not preserved, as a true rotation would do.

Caution

Note that the “same as” operator, ~=, represents the usual notion of equality for the point, box, polygon, and circle types. Some of the geometric types also have an = operator, but = compares for equal areas only. The other scalar comparison operators (<= and so on), where available for these types, likewise compare areas.

Note

Before PostgreSQL 14, the point is strictly below/above comparison operators point <<| point and point |>> point were respectively called <^ and >^. These names are still available, but are deprecated and will eventually be removed.

Table 9.36. Geometric Functions

Function

Description

Example(s)

area ( geometric_type ) → double precision

Computes area. Available for box, path, circle. A path input must be closed, else NULL is returned. Also, if the path is self-intersecting, the result may be meaningless.

area(box '(2,2),(0,0)') → 4

center ( geometric_type ) → point

Computes center point. Available for box, circle.

center(box '(1,2),(0,0)') → (0.5,1)

diagonal ( box ) → lseg

Extracts box's diagonal as a line segment (same as lseg(box)).

diagonal(box '(1,2),(0,0)') → [(1,2),(0,0)]

diameter ( circle ) → double precision

Computes diameter of circle.

diameter(circle '<(0,0),2>') → 4

height ( box ) → double precision

Computes vertical size of box.

height(box '(1,2),(0,0)') → 2

isclosed ( path ) → boolean

Is path closed?

isclosed(path '((0,0),(1,1),(2,0))') → t

isopen ( path ) → boolean

Is path open?

isopen(path '[(0,0),(1,1),(2,0)]') → t

length ( geometric_type ) → double precision

Computes the total length. Available for lseg, path.

length(path '((-1,0),(1,0))') → 4

npoints ( geometric_type ) → integer

Returns the number of points. Available for path, polygon.

npoints(path '[(0,0),(1,1),(2,0)]') → 3

pclose ( path ) → path

Converts path to closed form.

pclose(path '[(0,0),(1,1),(2,0)]') → ((0,0),(1,1),(2,0))

popen ( path ) → path

Converts path to open form.

popen(path '((0,0),(1,1),(2,0))') → [(0,0),(1,1),(2,0)]

radius ( circle ) → double precision

Computes radius of circle.

radius(circle '<(0,0),2>') → 2

slope ( point, point ) → double precision

Computes slope of a line drawn through the two points.

slope(point '(0,0)', point '(2,1)') → 0.5

width ( box ) → double precision

Computes horizontal size of box.

width(box '(1,2),(0,0)') → 1

Table 9.37. Geometric Type Conversion Functions

Function

Description

Example(s)

box ( circle ) → box

Computes box inscribed within the circle.

box(circle '<(0,0),2>') → (1.414213562373095,1.414213562373095),​(-1.414213562373095,-1.414213562373095)

box ( point ) → box

Converts point to empty box.

box(point '(1,0)') → (1,0),(1,0)

box ( point, point ) → box

Converts any two corner points to box.

box(point '(0,1)', point '(1,0)') → (1,1),(0,0)

box ( polygon ) → box

Computes bounding box of polygon.

box(polygon '((0,0),(1,1),(2,0))') → (2,1),(0,0)

bound_box ( box, box ) → box

Computes bounding box of two boxes.

bound_box(box '(1,1),(0,0)', box '(4,4),(3,3)') → (4,4),(0,0)

circle ( box ) → circle

Computes smallest circle enclosing box.

circle(box '(1,1),(0,0)') → <(0.5,0.5),0.7071067811865476>

circle ( point, double precision ) → circle

Constructs circle from center and radius.

circle(point '(0,0)', 2.0) → <(0,0),2>

circle ( polygon ) → circle

Converts polygon to circle. The circle's center is the mean of the positions of the polygon's points, and the radius is the average distance of the polygon's points from that center.

circle(polygon '((0,0),(1,3),(2,0))') → <(1,1),1.6094757082487299>

line ( point, point ) → line

Converts two points to the line through them.

line(point '(-1,0)', point '(1,0)') → {0,-1,0}

lseg ( box ) → lseg

Extracts box's diagonal as a line segment.

lseg(box '(1,0),(-1,0)') → [(1,0),(-1,0)]

lseg ( point, point ) → lseg

Constructs line segment from two endpoints.

lseg(point '(-1,0)', point '(1,0)') → [(-1,0),(1,0)]

path ( polygon ) → path

Converts polygon to a closed path with the same list of points.

path(polygon '((0,0),(1,1),(2,0))') → ((0,0),(1,1),(2,0))

point ( double precision, double precision ) → point

Constructs point from its coordinates.

point(23.4, -44.5) → (23.4,-44.5)

point ( box ) → point

Computes center of box.

point(box '(1,0),(-1,0)') → (0,0)

point ( circle ) → point

Computes center of circle.

point(circle '<(0,0),2>') → (0,0)

point ( lseg ) → point

Computes center of line segment.

point(lseg '[(-1,0),(1,0)]') → (0,0)

point ( polygon ) → point

Computes center of polygon (the mean of the positions of the polygon's points).

point(polygon '((0,0),(1,1),(2,0))') → (1,0.3333333333333333)

polygon ( box ) → polygon

Converts box to a 4-point polygon.

polygon(box '(1,1),(0,0)') → ((0,0),(0,1),(1,1),(1,0))

polygon ( circle ) → polygon

Converts circle to a 12-point polygon.

polygon(circle '<(0,0),2>') → ((-2,0),​(-1.7320508075688774,0.9999999999999999),​(-1.0000000000000002,1.7320508075688772),​(-1.2246063538223773e-16,2),​(0.9999999999999996,1.7320508075688774),​(1.732050807568877,1.0000000000000007),​(2,2.4492127076447545e-16),​(1.7320508075688776,-0.9999999999999994),​(1.0000000000000009,-1.7320508075688767),​(3.673819061467132e-16,-2),​(-0.9999999999999987,-1.732050807568878),​(-1.7320508075688767,-1.0000000000000009))

polygon ( integer, circle ) → polygon

Converts circle to an n-point polygon.

polygon(4, circle '<(3,0),1>') → ((2,0),​(3,1),​(4,1.2246063538223773e-16),​(3,-1))

polygon ( path ) → polygon

Converts closed path to a polygon with the same list of points.

polygon(path '((0,0),(1,1),(2,0))') → ((0,0),(1,1),(2,0))

It is possible to access the two component numbers of a point as though the point were an array with indexes 0 and 1. For example, if t.p is a point column then SELECT p[0] FROM t retrieves the X coordinate and UPDATE t SET p[1] = ... changes the Y coordinate. In the same way, a value of type box or lseg can be treated as an array of two point values.

Prev | Up | Next
9.10. Enum Support Functions | Home | 9.12. Network Address Functions and Operators


9.12. Network Address Functions and Operators
Prev Up Chapter 9. Functions and Operators Home Next

9.12. Network Address Functions and Operators

The IP network address types, cidr and inet, support the usual comparison operators shown in Table 9.1 as well as the specialized operators and functions shown in Table 9.38 and Table 9.39.

Any cidr value can be cast to inet implicitly; therefore, the operators and functions shown below as operating on inet also work on cidr values. (Where there are separate functions for inet and cidr, it is because the behavior should be different for the two cases.) Also, it is permitted to cast an inet value to cidr. When this is done, any bits to the right of the netmask are silently zeroed to create a valid cidr value.

Table 9.38. IP Address Operators

Operator

Description

Example(s)

inet << inet → boolean

Is subnet strictly contained by subnet? This operator, and the next four, test for subnet inclusion. They consider only the network parts of the two addresses (ignoring any bits to the right of the netmasks) and determine whether one network is identical to or a subnet of the other.

inet '192.168.1.5' << inet '192.168.1/24' → t

inet '192.168.0.5' << inet '192.168.1/24' → f

inet '192.168.1/24' << inet '192.168.1/24' → f

inet <<= inet → boolean

Is subnet contained by or equal to subnet?

inet '192.168.1/24' <<= inet '192.168.1/24' → t

inet >> inet → boolean

Does subnet strictly contain subnet?

inet '192.168.1/24' >> inet '192.168.1.5' → t

inet >>= inet → boolean

Does subnet contain or equal subnet?

inet '192.168.1/24' >>= inet '192.168.1/24' → t

inet && inet → boolean

Does either subnet contain or equal the other?

inet '192.168.1/24' && inet '192.168.1.80/28' → t

inet '192.168.1/24' && inet '192.168.2.0/28' → f

~ inet → inet

Computes bitwise NOT.

~ inet '192.168.1.6' → 63.87.254.249

inet & inet → inet

Computes bitwise AND.

inet '192.168.1.6' & inet '0.0.0.255' → 0.0.0.6

inet | inet → inet

Computes bitwise OR.

inet '192.168.1.6' | inet '0.0.0.255' → 192.168.1.255

inet + bigint → inet

Adds an offset to an address.

inet '192.168.1.6' + 25 → 192.168.1.31

bigint + inet → inet

Adds an offset to an address.

200 + inet '::ffff:fff0:1' → ::ffff:255.240.0.201

inet - bigint → inet

Subtracts an offset from an address.

inet '192.168.1.43' - 36 → 192.168.1.7

inet - inet → bigint

Computes the difference of two addresses.

inet '192.168.1.43' - inet '192.168.1.19' → 24

inet '::1' - inet '::ffff:1' → -4294901760

Table 9.39. IP Address Functions

Function

Description

Example(s)

abbrev ( inet ) → text

Creates an abbreviated display format as text. (The result is the same as the inet output function produces; it is “abbreviated” only in comparison to the result of an explicit cast to text, which for historical reasons will never suppress the netmask part.)

abbrev(inet '10.1.0.0/32') → 10.1.0.0

abbrev ( cidr ) → text

Creates an abbreviated display format as text. (The abbreviation consists of dropping all-zero octets to the right of the netmask; more examples are in Table 8.22.)

abbrev(cidr '10.1.0.0/16') → 10.1/16

broadcast ( inet ) → inet

Computes the broadcast address for the address's network.

broadcast(inet '192.168.1.5/24') → 192.168.1.255/24

family ( inet ) → integer

Returns the address's family: 4 for IPv4, 6 for IPv6.

family(inet '::1') → 6

host ( inet ) → text

Returns the IP address as text, ignoring the netmask.

host(inet '192.168.1.0/24') → 192.168.1.0

hostmask ( inet ) → inet

Computes the host mask for the address's network.

hostmask(inet '192.168.23.20/30') → 0.0.0.3

inet_merge ( inet, inet ) → cidr

Computes the smallest network that includes both of the given networks.

inet_merge(inet '192.168.1.5/24', inet '192.168.2.5/24') → 192.168.0.0/22

inet_same_family ( inet, inet ) → boolean

Tests whether the addresses belong to the same IP family.

inet_same_family(inet '192.168.1.5/24', inet '::1') → f

masklen ( inet ) → integer

Returns the netmask length in bits.

masklen(inet '192.168.1.5/24') → 24

netmask ( inet ) → inet

Computes the network mask for the address's network.

netmask(inet '192.168.1.5/24') → 255.255.255.0

network ( inet ) → cidr

Returns the network part of the address, zeroing out whatever is to the right of the netmask. (This is equivalent to casting the value to cidr.)

network(inet '192.168.1.5/24') → 192.168.1.0/24

set_masklen ( inet, integer ) → inet

Sets the netmask length for an inet value. The address part does not change.

set_masklen(inet '192.168.1.5/24', 16) → 192.168.1.5/16

set_masklen ( cidr, integer ) → cidr

Sets the netmask length for a cidr value. Address bits to the right of the new netmask are set to zero.

set_masklen(cidr '192.168.1.0/24', 16) → 192.168.0.0/16

text ( inet ) → text

Returns the unabbreviated IP address and netmask length as text. (This has the same result as an explicit cast to text.)

text(inet '192.168.1.5') → 192.168.1.5/32

Tip

The abbrev, host, and text functions are primarily intended to offer alternative display formats for IP addresses.

The MAC address types, macaddr and macaddr8, support the usual comparison operators shown in Table 9.1 as well as the specialized functions shown in Table 9.40. In addition, they support the bitwise logical operators ~, & and | (NOT, AND and OR), just as shown above for IP addresses.

Table 9.40. MAC Address Functions

Function

Description

Example(s)

trunc ( macaddr ) → macaddr

Sets the last 3 bytes of the address to zero. The remaining prefix can be associated with a particular manufacturer (using data not included in PostgreSQL).

trunc(macaddr '12:34:56:78:90:ab') → 12:34:56:00:00:00

trunc ( macaddr8 ) → macaddr8

Sets the last 5 bytes of the address to zero. The remaining prefix can be associated with a particular manufacturer (using data not included in PostgreSQL).

trunc(macaddr8 '12:34:56:78:90:ab:cd:ef') → 12:34:56:00:00:00:00:00

macaddr8_set7bit ( macaddr8 ) → macaddr8

Sets the 7th bit of the address to one, creating what is known as modified EUI-64, for inclusion in an IPv6 address.

macaddr8_set7bit(macaddr8 '00:34:56:ab:cd:ef') → 02:34:56:ff:fe:ab:cd:ef

Prev | Up | Next
9.11. Geometric Functions and Operators | Home | 9.13. Text Search Functions and Operators


9.13. Text Search Functions and Operators
Prev Up Chapter 9. Functions and Operators Home Next

9.13. Text Search Functions and Operators

Table 9.41, Table 9.42 and Table 9.43 summarize the functions and operators that are provided for full text searching. See Chapter 12 for a detailed explanation of PostgreSQL's text search facility.

Table 9.41. Text Search Operators

Operator

Description

Example(s)

tsvector @@ tsquery → boolean

tsquery @@ tsvector → boolean

Does tsvector match tsquery? (The arguments can be given in either order.)

to_tsvector('fat cats ate rats') @@ to_tsquery('cat & rat') → t

text @@ tsquery → boolean

Does text string, after implicit invocation of to_tsvector(), match tsquery?

'fat cats ate rats' @@ to_tsquery('cat & rat') → t

tsvector @@@ tsquery → boolean

tsquery @@@ tsvector → boolean

This is a deprecated synonym for @@.

to_tsvector('fat cats ate rats') @@@ to_tsquery('cat & rat') → t

tsvector || tsvector → tsvector

Concatenates two tsvectors. If both inputs contain lexeme positions, the second input's positions are adjusted accordingly.

'a:1 b:2'::tsvector || 'c:1 d:2 b:3'::tsvector → 'a':1 'b':2,5 'c':3 'd':4

tsquery && tsquery → tsquery

ANDs two tsquerys together, producing a query that matches documents that match both input queries.

'fat | rat'::tsquery && 'cat'::tsquery → ( 'fat' | 'rat' ) & 'cat'

tsquery || tsquery → tsquery

ORs two tsquerys together, producing a query that matches documents that match either input query.

'fat | rat'::tsquery || 'cat'::tsquery → 'fat' | 'rat' | 'cat'

!! tsquery → tsquery

Negates a tsquery, producing a query that matches documents that do not match the input query.

!! 'cat'::tsquery → !'cat'

tsquery <-> tsquery → tsquery

Constructs a phrase query, which matches if the two input queries match at successive lexemes.

to_tsquery('fat') <-> to_tsquery('rat') → 'fat' <-> 'rat'

tsquery @> tsquery → boolean

Does first tsquery contain the second? (This considers only whether all the lexemes appearing in one query appear in the other, ignoring the combining operators.)

'cat'::tsquery @> 'cat & rat'::tsquery → f

tsquery <@ tsquery → boolean

Is first tsquery contained in the second? (This considers only whether all the lexemes appearing in one query appear in the other, ignoring the combining operators.)

'cat'::tsquery <@ 'cat & rat'::tsquery → t

'cat'::tsquery <@ '!cat & rat'::tsquery → t

In addition to these specialized operators, the usual comparison operators shown in Table 9.1 are available for types tsvector and tsquery. These are not very useful for text searching but allow, for example, unique indexes to be built on columns of these types.

Table 9.42. Text Search Functions

Function

Description

Example(s)

array_to_tsvector ( text[] ) → tsvector

Converts an array of lexemes to a tsvector. The given strings are used as-is without further processing.

array_to_tsvector('{fat,cat,rat}'::text[]) → 'cat' 'fat' 'rat'

get_current_ts_config ( ) → regconfig

Returns the OID of the current default text search configuration (as set by default_text_search_config).

get_current_ts_config() → english

length ( tsvector ) → integer

Returns the number of lexemes in the tsvector.

length('fat:2,4 cat:3 rat:5A'::tsvector) → 3

numnode ( tsquery ) → integer

Returns the number of lexemes plus operators in the tsquery.

numnode('(fat & rat) | cat'::tsquery) → 5

plainto_tsquery ( [ config regconfig, ] query text ) → tsquery

Converts text to a tsquery, normalizing words according to the specified or default configuration. Any punctuation in the string is ignored (it does not determine query operators). The resulting query matches documents containing all non-stopwords in the text.

plainto_tsquery('english', 'The Fat Rats') → 'fat' & 'rat'

phraseto_tsquery ( [ config regconfig, ] query text ) → tsquery

Converts text to a tsquery, normalizing words according to the specified or default configuration. Any punctuation in the string is ignored (it does not determine query operators). The resulting query matches phrases containing all non-stopwords in the text.

phraseto_tsquery('english', 'The Fat Rats') → 'fat' <-> 'rat'

phraseto_tsquery('english', 'The Cat and Rats') → 'cat' <2> 'rat'

websearch_to_tsquery ( [ config regconfig, ] query text ) → tsquery

Converts text to a tsquery, normalizing words according to the specified or default configuration. Quoted word sequences are converted to phrase tests. The word “or” is understood as producing an OR operator, and a dash produces a NOT operator; other punctuation is ignored. This approximates the behavior of some common web search tools.

websearch_to_tsquery('english', '"fat rat" or cat dog') → 'fat' <-> 'rat' | 'cat' & 'dog'

querytree ( tsquery ) → text

Produces a representation of the indexable portion of a tsquery. A result that is empty or just T indicates a non-indexable query.

querytree('foo & ! bar'::tsquery) → 'foo'

setweight ( vector tsvector, weight "char" ) → tsvector

Assigns the specified weight to each element of the vector.

setweight('fat:2,4 cat:3 rat:5B'::tsvector, 'A') → 'cat':3A 'fat':2A,4A 'rat':5A

setweight ( vector tsvector, weight "char", lexemes text[] ) → tsvector

Assigns the specified weight to elements of the vector that are listed in lexemes.

setweight('fat:2,4 cat:3 rat:5,6B'::tsvector, 'A', '{cat,rat}') → 'cat':3A 'fat':2,4 'rat':5A,6A

strip ( tsvector ) → tsvector

Removes positions and weights from the tsvector.

strip('fat:2,4 cat:3 rat:5A'::tsvector) → 'cat' 'fat' 'rat'

to_tsquery ( [ config regconfig, ] query text ) → tsquery

Converts text to a tsquery, normalizing words according to the specified or default configuration. The words must be combined by valid tsquery operators.

to_tsquery('english', 'The & Fat & Rats') → 'fat' & 'rat'

to_tsvector ( [ config regconfig, ] document text ) → tsvector

Converts text to a tsvector, normalizing words according to the specified or default configuration. Position information is included in the result.

to_tsvector('english', 'The Fat Rats') → 'fat':2 'rat':3

to_tsvector ( [ config regconfig, ] document json ) → tsvector

to_tsvector ( [ config regconfig, ] document jsonb ) → tsvector

Converts each string value in the JSON document to a tsvector, normalizing words according to the specified or default configuration. The results are then concatenated in document order to produce the output. Position information is generated as though one stopword exists between each pair of string values. (Beware that “document order” of the fields of a JSON object is implementation-dependent when the input is jsonb; observe the difference in the examples.)

to_tsvector('english', '{"aa": "The Fat Rats", "b": "dog"}'::json) → 'dog':5 'fat':2 'rat':3

to_tsvector('english', '{"aa": "The Fat Rats", "b": "dog"}'::jsonb) → 'dog':1 'fat':4 'rat':5

json_to_tsvector ( [ config regconfig, ] document json, filter jsonb ) → tsvector

jsonb_to_tsvector ( [ config regconfig, ] document jsonb, filter jsonb ) → tsvector

Selects each item in the JSON document that is requested by the filter and converts each one to a tsvector, normalizing words according to the specified or default configuration. The results are then concatenated in document order to produce the output. Position information is generated as though one stopword exists between each pair of selected items. (Beware that “document order” of the fields of a JSON object is implementation-dependent when the input is jsonb.) The filter must be a jsonb array containing zero or more of these keywords: "string" (to include all string values), "numeric" (to include all numeric values), "boolean" (to include all boolean values), "key" (to include all keys), or "all" (to include all the above). As a special case, the filter can also be a simple JSON value that is one of these keywords.

json_to_tsvector('english', '{"a": "The Fat Rats", "b": 123}'::json, '["string", "numeric"]') → '123':5 'fat':2 'rat':3

json_to_tsvector('english', '{"cat": "The Fat Rats", "dog": 123}'::json, '"all"') → '123':9 'cat':1 'dog':7 'fat':4 'rat':5

ts_delete ( vector tsvector, lexeme text ) → tsvector

Removes any occurrence of the given lexeme from the vector.

ts_delete('fat:2,4 cat:3 rat:5A'::tsvector, 'fat') → 'cat':3 'rat':5A

ts_delete ( vector tsvector, lexemes text[] ) → tsvector

Removes any occurrences of the lexemes in lexemes from the vector.

ts_delete('fat:2,4 cat:3 rat:5A'::tsvector, ARRAY['fat','rat']) → 'cat':3

ts_filter ( vector tsvector, weights "char"[] ) → tsvector

Selects only elements with the given weights from the vector.

ts_filter('fat:2,4 cat:3b,7c rat:5A'::tsvector, '{a,b}') → 'cat':3B 'rat':5A

ts_headline ( [ config regconfig, ] document text, query tsquery [, options text ] ) → text

Displays, in an abbreviated form, the match(es) for the query in the document, which must be raw text not a tsvector. Words in the document are normalized according to the specified or default configuration before matching to the query. Use of this function is discussed in Section 12.3.4, which also describes the available options.

ts_headline('The fat cat ate the rat.', 'cat') → The fat <b>cat</b> ate the rat.

ts_headline ( [ config regconfig, ] document json, query tsquery [, options text ] ) → text

ts_headline ( [ config regconfig, ] document jsonb, query tsquery [, options text ] ) → text

Displays, in an abbreviated form, match(es) for the query that occur in string values within the JSON document. See Section 12.3.4 for more details.

ts_headline('{"cat":"raining cats and dogs"}'::jsonb, 'cat') → {"cat": "raining <b>cats</b> and dogs"}

ts_rank ( [ weights real[], ] vector tsvector, query tsquery [, normalization integer ] ) → real

Computes a score showing how well the vector matches the query. See Section 12.3.3 for details.

ts_rank(to_tsvector('raining cats and dogs'), 'cat') → 0.06079271

ts_rank_cd ( [ weights real[], ] vector tsvector, query tsquery [, normalization integer ] ) → real

Computes a score showing how well the vector matches the query, using a cover density algorithm. See Section 12.3.3 for details.

ts_rank_cd(to_tsvector('raining cats and dogs'), 'cat') → 0.1

ts_rewrite ( query tsquery, target tsquery, substitute tsquery ) → tsquery

Replaces occurrences of target with substitute within the query. See Section 12.4.2.1 for details.

ts_rewrite('a & b'::tsquery, 'a'::tsquery, 'foo|bar'::tsquery) → 'b' & ( 'foo' | 'bar' )

ts_rewrite ( query tsquery, select text ) → tsquery

Replaces portions of the query according to target(s) and substitute(s) obtained by executing a SELECT command. See Section 12.4.2.1 for details.

SELECT ts_rewrite('a & b'::tsquery, 'SELECT t,s FROM aliases') → 'b' & ( 'foo' | 'bar' )

tsquery_phrase ( query1 tsquery, query2 tsquery ) → tsquery

Constructs a phrase query that searches for matches of query1 and query2 at successive lexemes (same as <-> operator).

tsquery_phrase(to_tsquery('fat'), to_tsquery('cat')) → 'fat' <-> 'cat'

tsquery_phrase ( query1 tsquery, query2 tsquery, distance integer ) → tsquery

Constructs a phrase query that searches for matches of query1 and query2 that occur exactly distance lexemes apart.

tsquery_phrase(to_tsquery('fat'), to_tsquery('cat'), 10) → 'fat' <10> 'cat'

tsvector_to_array ( tsvector ) → text[]

Converts a tsvector to an array of lexemes.

tsvector_to_array('fat:2,4 cat:3 rat:5A'::tsvector) → {cat,fat,rat}

unnest ( tsvector ) → setof record ( lexeme text, positions smallint[], weights text )

Expands a tsvector into a set of rows, one per lexeme.

select * from unnest('cat:3 fat:2,4 rat:5A'::tsvector) →

 lexeme | positions | weights
--------+-----------+---------
 cat    | {3}       | {D}
 fat    | {2,4}     | {D,D}
 rat    | {5}       | {A}

Note

All the text search functions that accept an optional regconfig argument will use the configuration specified by default_text_search_config when that argument is omitted.

The functions in Table 9.43 are listed separately because they are not usually used in everyday text searching operations. They are primarily helpful for development and debugging of new text search configurations.

Table 9.43. Text Search Debugging Functions

Function

Description

Example(s)

ts_debug ( [ config regconfig, ] document text ) → setof record ( alias text, description text, token text, dictionaries regdictionary[], dictionary regdictionary, lexemes text[] )

Extracts and normalizes tokens from the document according to the specified or default text search configuration, and returns information about how each token was processed. See Section 12.8.1 for details.

ts_debug('english', 'The Brightest supernovaes') → (asciiword,"Word, all ASCII",The,{english_stem},english_stem,{}) ...

ts_lexize ( dict regdictionary, token text ) → text[]

Returns an array of replacement lexemes if the input token is known to the dictionary, or an empty array if the token is known to the dictionary but it is a stop word, or NULL if it is not a known word. See Section 12.8.3 for details.

ts_lexize('english_stem', 'stars') → {star}

ts_parse ( parser_name text, document text ) → setof record ( tokid integer, token text )

Extracts tokens from the document using the named parser. See Section 12.8.2 for details.

ts_parse('default', 'foo - bar') → (1,foo) ...

ts_parse ( parser_oid oid, document text ) → setof record ( tokid integer, token text )

Extracts tokens from the document using a parser specified by OID. See Section 12.8.2 for details.

ts_parse(3722, 'foo - bar') → (1,foo) ...

ts_token_type ( parser_name text ) → setof record ( tokid integer, alias text, description text )

Returns a table that describes each type of token the named parser can recognize. See Section 12.8.2 for details.

ts_token_type('default') → (1,asciiword,"Word, all ASCII") ...

ts_token_type ( parser_oid oid ) → setof record ( tokid integer, alias text, description text )

Returns a table that describes each type of token a parser specified by OID can recognize. See Section 12.8.2 for details.

ts_token_type(3722) → (1,asciiword,"Word, all ASCII") ...

ts_stat ( sqlquery text [, weights text ] ) → setof record ( word text, ndoc integer, nentry integer )

Executes the sqlquery, which must return a single tsvector column, and returns statistics about each distinct lexeme contained in the data. See Section 12.4.4 for details.

ts_stat('SELECT vector FROM apod') → (foo,10,15) ...

Prev | Up | Next
9.12. Network Address Functions and Operators | Home | 9.14. UUID Functions


9.14. UUID Functions
Prev Up Chapter 9. Functions and Operators Home Next

9.14. UUID Functions

PostgreSQL includes one function to generate a UUID:

gen_random_uuid () → uuid

This function returns a version 4 (random) UUID. This is the most commonly used type of UUID and is appropriate for most applications.

The uuid-ossp module provides additional functions that implement other standard algorithms for generating UUIDs.

PostgreSQL also provides the usual comparison operators shown in Table 9.1 for UUIDs.

Prev | Up | Next
9.13. Text Search Functions and Operators | Home | 9.15. XML Functions


9.15. XML Functions
Prev Up Chapter 9. Functions and Operators Home Next

9.15. XML Functions

9.15.1. Producing XML Content
9.15.2. XML Predicates
9.15.3. Processing XML
9.15.4. Mapping Tables to XML

The functions and function-like expressions described in this section operate on values of type xml. See Section 8.13 for information about the xml type. The function-like expressions xmlparse and xmlserialize for converting to and from type xml are documented there, not in this section.

Use of most of these functions requires PostgreSQL to have been built with configure --with-libxml.

9.15.1. Producing XML Content

A set of functions and function-like expressions is available for producing XML content from SQL data. As such, they are particularly suitable for formatting query results into XML documents for processing in client applications.

9.15.1.1. xmlcomment

xmlcomment ( text ) → xml

The function xmlcomment creates an XML value containing an XML comment with the specified text as content. The text cannot contain “--” or end with a “-”, otherwise the resulting construct would not be a valid XML comment. If the argument is null, the result is null.

Example:

SELECT xmlcomment('hello');

  xmlcomment
--------------
 <!--hello-->

9.15.1.2. xmlconcat

xmlconcat ( xml [, ...] ) → xml

The function xmlconcat concatenates a list of individual XML values to create a single value containing an XML content fragment. Null values are omitted; the result is only null if there are no nonnull arguments.

Example:

SELECT xmlconcat('<abc/>', '<bar>foo</bar>');

      xmlconcat
----------------------
 <abc/><bar>foo</bar>

XML declarations, if present, are combined as follows. If all argument values have the same XML version declaration, that version is used in the result, else no version is used. If all argument values have the standalone declaration value “yes”, then that value is used in the result. If all argument values have a standalone declaration value and at least one is “no”, then that is used in the result. Else the result will have no standalone declaration. If the result is determined to require a standalone declaration but no version declaration, a version declaration with version 1.0 will be used because XML requires an XML declaration to contain a version declaration. Encoding declarations are ignored and removed in all cases.

Example:

SELECT xmlconcat('<?xml version="1.1"?><foo/>', '<?xml version="1.1" standalone="no"?><bar/>');

             xmlconcat
-----------------------------------
 <?xml version="1.1"?><foo/><bar/>

9.15.1.3. xmlelement

xmlelement ( NAME name [, XMLATTRIBUTES ( attvalue [ AS attname ] [, ...] ) ] [, content [, ...]] ) → xml

The xmlelement expression produces an XML element with the given name, attributes, and content. The name and attname items shown in the syntax are simple identifiers, not values. The attvalue and content items are expressions, which can yield any PostgreSQL data type. The argument(s) within XMLATTRIBUTES generate attributes of the XML element; the content value(s) are concatenated to form its content.

Examples:

SELECT xmlelement(name foo);

 xmlelement
------------
 <foo/>

SELECT xmlelement(name foo, xmlattributes('xyz' as bar));

    xmlelement
------------------
 <foo bar="xyz"/>

SELECT xmlelement(name foo, xmlattributes(current_date as bar), 'cont', 'ent');

             xmlelement
-------------------------------------
 <foo bar="2007-01-26">content</foo>

Element and attribute names that are not valid XML names are escaped by replacing the offending characters by the sequence _xHHHH_, where HHHH is the character's Unicode codepoint in hexadecimal notation. For example:

SELECT xmlelement(name "foo$bar", xmlattributes('xyz' as "a&b"));

            xmlelement
----------------------------------
 <foo_x0024_bar a_x0026_b="xyz"/>

An explicit attribute name need not be specified if the attribute value is a column reference, in which case the column's name will be used as the attribute name by default. In other cases, the attribute must be given an explicit name. So this example is valid:

CREATE TABLE test (a xml, b xml);
SELECT xmlelement(name test, xmlattributes(a, b)) FROM test;

But these are not:

SELECT xmlelement(name test, xmlattributes('constant'), a, b) FROM test;
SELECT xmlelement(name test, xmlattributes(func(a, b))) FROM test;

Element content, if specified, will be formatted according to its data type. If the content is itself of type xml, complex XML documents can be constructed. For example:

SELECT xmlelement(name foo, xmlattributes('xyz' as bar),
                            xmlelement(name abc),
                            xmlcomment('test'),
                            xmlelement(name xyz));

                  xmlelement
----------------------------------------------
 <foo bar="xyz"><abc/><!--test--><xyz/></foo>

Content of other types will be formatted into valid XML character data. This means in particular that the characters <, >, and & will be converted to entities. Binary data (data type bytea) will be represented in base64 or hex encoding, depending on the setting of the configuration parameter xmlbinary. The particular behavior for individual data types is expected to evolve in order to align the PostgreSQL mappings with those specified in SQL:2006 and later, as discussed in Section D.3.1.3.

9.15.1.4. xmlforest

xmlforest ( content [ AS name ] [, ...] ) → xml

The xmlforest expression produces an XML forest (sequence) of elements using the given names and content. As for xmlelement, each name must be a simple identifier, while the content expressions can have any data type.

Examples:

SELECT xmlforest('abc' AS foo, 123 AS bar);

          xmlforest
------------------------------
 <foo>abc</foo><bar>123</bar>

SELECT xmlforest(table_name, column_name)
FROM information_schema.columns
WHERE table_schema = 'pg_catalog';

                                xmlforest
------------------------------------​-----------------------------------
 <table_name>pg_authid</table_name>​<column_name>rolname</column_name>
 <table_name>pg_authid</table_name>​<column_name>rolsuper</column_name>
 ...

As seen in the second example, the element name can be omitted if the content value is a column reference, in which case the column name is used by default. Otherwise, a name must be specified.

Element names that are not valid XML names are escaped as shown for xmlelement above. Similarly, content data is escaped to make valid XML content, unless it is already of type xml.

Note that XML forests are not valid XML documents if they consist of more than one element, so it might be useful to wrap xmlforest expressions in xmlelement.

9.15.1.5. xmlpi

xmlpi ( NAME name [, content ] ) → xml

The xmlpi expression creates an XML processing instruction. As for xmlelement, the name must be a simple identifier, while the content expression can have any data type. The content, if present, must not contain the character sequence ?>.

Example:

SELECT xmlpi(name php, 'echo "hello world";');

            xmlpi
-----------------------------
 <?php echo "hello world";?>

9.15.1.6. xmlroot

xmlroot ( xml, VERSION {text|NO VALUE} [, STANDALONE {YES|NO|NO VALUE} ] ) → xml

The xmlroot expression alters the properties of the root node of an XML value. If a version is specified, it replaces the value in the root node's version declaration; if a standalone setting is specified, it replaces the value in the root node's standalone declaration.

SELECT xmlroot(xmlparse(document '<?xml version="1.1"?><content>abc</content>'),
               version '1.0', standalone yes);

                xmlroot
----------------------------------------
 <?xml version="1.0" standalone="yes"?>
 <content>abc</content>

9.15.1.7. xmlagg

xmlagg ( xml ) → xml

The function xmlagg is, unlike the other functions described here, an aggregate function. It concatenates the input values to the aggregate function call, much like xmlconcat does, except that concatenation occurs across rows rather than across expressions in a single row. See Section 9.21 for additional information about aggregate functions.

Example:

CREATE TABLE test (y int, x xml);
INSERT INTO test VALUES (1, '<foo>abc</foo>');
INSERT INTO test VALUES (2, '<bar/>');
SELECT xmlagg(x) FROM test;
        xmlagg
----------------------
 <foo>abc</foo><bar/>

To determine the order of the concatenation, an ORDER BY clause may be added to the aggregate call as described in Section 4.2.7. For example:

SELECT xmlagg(x ORDER BY y DESC) FROM test;
        xmlagg
----------------------
 <bar/><foo>abc</foo>

The following non-standard approach used to be recommended in previous versions, and may still be useful in specific cases:

SELECT xmlagg(x) FROM (SELECT * FROM test ORDER BY y DESC) AS tab;
        xmlagg
----------------------
 <bar/><foo>abc</foo>

9.15.2. XML Predicates

The expressions described in this section check properties of xml values.

9.15.2.1. IS DOCUMENT

xml IS DOCUMENT → boolean

The expression IS DOCUMENT returns true if the argument XML value is a proper XML document, false if it is not (that is, it is a content fragment), or null if the argument is null. See Section 8.13 about the difference between documents and content fragments.

9.15.2.2. IS NOT DOCUMENT

xml IS NOT DOCUMENT → boolean

The expression IS NOT DOCUMENT returns false if the argument XML value is a proper XML document, true if it is not (that is, it is a content fragment), or null if the argument is null.

9.15.2.3. XMLEXISTS

XMLEXISTS ( text PASSING [BY {REF|VALUE}] xml [BY {REF|VALUE}] ) → boolean

The function xmlexists evaluates an XPath 1.0 expression (the first argument), with the passed XML value as its context item. The function returns false if the result of that evaluation yields an empty node-set, true if it yields any other value. The function returns null if any argument is null. A nonnull value passed as the context item must be an XML document, not a content fragment or any non-XML value.

Example:

SELECT xmlexists('//town[text() = ''Toronto'']' PASSING BY VALUE '<towns><town>Toronto</town><town>Ottawa</town></towns>');

 xmlexists
------------
 t
(1 row)

The BY REF and BY VALUE clauses are accepted in PostgreSQL, but are ignored, as discussed in Section D.3.2.

In the SQL standard, the xmlexists function evaluates an expression in the XML Query language, but PostgreSQL allows only an XPath 1.0 expression, as discussed in Section D.3.1.

9.15.2.4. xml_is_well_formed

xml_is_well_formed ( text ) → boolean
xml_is_well_formed_document ( text ) → boolean
xml_is_well_formed_content ( text ) → boolean

These functions check whether a text string represents well-formed XML, returning a Boolean result. xml_is_well_formed_document checks for a well-formed document, while xml_is_well_formed_content checks for well-formed content. xml_is_well_formed does the former if the xmloption configuration parameter is set to DOCUMENT, or the latter if it is set to CONTENT. This means that xml_is_well_formed is useful for seeing whether a simple cast to type xml will succeed, whereas the other two functions are useful for seeing whether the corresponding variants of XMLPARSE will succeed.

Examples:

SET xmloption TO DOCUMENT;
SELECT xml_is_well_formed('<>');
 xml_is_well_formed
--------------------
 f
(1 row)

SELECT xml_is_well_formed('<abc/>');
 xml_is_well_formed
--------------------
 t
(1 row)

SET xmloption TO CONTENT;
SELECT xml_is_well_formed('abc');
 xml_is_well_formed
--------------------
 t
(1 row)

SELECT xml_is_well_formed_document('<pg:foo xmlns:pg="http://postgresql.org/stuff">bar</pg:foo>');
 xml_is_well_formed_document
-----------------------------
 t
(1 row)

SELECT xml_is_well_formed_document('<pg:foo xmlns:pg="http://postgresql.org/stuff">bar</my:foo>');
 xml_is_well_formed_document
-----------------------------
 f
(1 row)

The last example shows that the checks include whether namespaces are correctly matched.

9.15.3. Processing XML

To process values of data type xml, PostgreSQL offers the functions xpath and xpath_exists, which evaluate XPath 1.0 expressions, and the XMLTABLE table function.

9.15.3.1. xpath

xpath ( xpath text, xml xml [, nsarray text[] ] ) → xml[]

The function xpath evaluates the XPath 1.0 expression xpath (given as text) against the XML value xml. It returns an array of XML values corresponding to the node-set produced by the XPath expression. If the XPath expression returns a scalar value rather than a node-set, a single-element array is returned.

The second argument must be a well formed XML document. In particular, it must have a single root node element.

The optional third argument of the function is an array of namespace mappings. This array should be a two-dimensional text array with the length of the second axis being equal to 2 (i.e., it should be an array of arrays, each of which consists of exactly 2 elements). The first element of each array entry is the namespace name (alias), the second the namespace URI. It is not required that aliases provided in this array be the same as those being used in the XML document itself (in other words, both in the XML document and in the xpath function context, aliases are local).

Example:

SELECT xpath('/my:a/text()', '<my:a xmlns:my="http://example.com">test</my:a>',
             ARRAY[ARRAY['my', 'http://example.com']]);

 xpath
--------
 {test}
(1 row)

To deal with default (anonymous) namespaces, do something like this:

SELECT xpath('//mydefns:b/text()', '<a xmlns="http://example.com"><b>test</b></a>',
             ARRAY[ARRAY['mydefns', 'http://example.com']]);

 xpath
--------
 {test}
(1 row)

9.15.3.2. xpath_exists

xpath_exists ( xpath text, xml xml [, nsarray text[] ] ) → boolean

The function xpath_exists is a specialized form of the xpath function. Instead of returning the individual XML values that satisfy the XPath 1.0 expression, this function returns a Boolean indicating whether the query was satisfied or not (specifically, whether it produced any value other than an empty node-set). This function is equivalent to the XMLEXISTS predicate, except that it also offers support for a namespace mapping argument.

Example:

SELECT xpath_exists('/my:a/text()', '<my:a xmlns:my="http://example.com">test</my:a>',
                     ARRAY[ARRAY['my', 'http://example.com']]);

 xpath_exists
--------------
 t
(1 row)

9.15.3.3. xmltable

XMLTABLE (
    [ XMLNAMESPACES ( namespace_uri AS namespace_name [, ...] ), ]
    row_expression PASSING [BY {REF|VALUE}] document_expression [BY {REF|VALUE}]
    COLUMNS name { type [PATH column_expression] [DEFAULT default_expression] [NOT NULL | NULL]
                  | FOR ORDINALITY }
            [, ...]
) → setof record

The xmltable expression produces a table based on an XML value, an XPath filter to extract rows, and a set of column definitions. Although it syntactically resembles a function, it can only appear as a table in a query's FROM clause.

The optional XMLNAMESPACES clause gives a comma-separated list of namespace definitions, where each namespace_uri is a text expression and each namespace_name is a simple identifier. It specifies the XML namespaces used in the document and their aliases. A default namespace specification is not currently supported.

The required row_expression argument is an XPath 1.0 expression (given as text) that is evaluated, passing the XML value document_expression as its context item, to obtain a set of XML nodes. These nodes are what xmltable transforms into output rows. No rows will be produced if the document_expression is null, nor if the row_expression produces an empty node-set or any value other than a node-set.

document_expression provides the context item for the row_expression. It must be a well-formed XML document; fragments/forests are not accepted. The BY REF and BY VALUE clauses are accepted but ignored, as discussed in Section D.3.2.

In the SQL standard, the xmltable function evaluates expressions in the XML Query language, but PostgreSQL allows only XPath 1.0 expressions, as discussed in Section D.3.1.

The required COLUMNS clause specifies the column(s) that will be produced in the output table. See the syntax summary above for the format. A name is required for each column, as is a data type (unless FOR ORDINALITY is specified, in which case type integer is implicit). The path, default and nullability clauses are optional.

A column marked FOR ORDINALITY will be populated with row numbers, starting with 1, in the order of nodes retrieved from the row_expression's result node-set. At most one column may be marked FOR ORDINALITY.

Note

XPath 1.0 does not specify an order for nodes in a node-set, so code that relies on a particular order of the results will be implementation-dependent. Details can be found in Section D.3.1.2.

The column_expression for a column is an XPath 1.0 expression that is evaluated for each row, with the current node from the row_expression result as its context item, to find the value of the column. If no column_expression is given, then the column name is used as an implicit path.

If a column's XPath expression returns a non-XML value (which is limited to string, boolean, or double in XPath 1.0) and the column has a PostgreSQL type other than xml, the column will be set as if by assigning the value's string representation to the PostgreSQL type. (If the value is a boolean, its string representation is taken to be 1 or 0 if the output column's type category is numeric, otherwise true or false.)

If a column's XPath expression returns a non-empty set of XML nodes and the column's PostgreSQL type is xml, the column will be assigned the expression result exactly, if it is of document or content form. [8]

A non-XML result assigned to an xml output column produces content, a single text node with the string value of the result. An XML result assigned to a column of any other type may not have more than one node, or an error is raised. If there is exactly one node, the column will be set as if by assigning the node's string value (as defined for the XPath 1.0 string function) to the PostgreSQL type.

The string value of an XML element is the concatenation, in document order, of all text nodes contained in that element and its descendants. The string value of an element with no descendant text nodes is an empty string (not NULL). Any xsi:nil attributes are ignored. Note that the whitespace-only text() node between two non-text elements is preserved, and that leading whitespace on a text() node is not flattened. The XPath 1.0 string function may be consulted for the rules defining the string value of other XML node types and non-XML values.

The conversion rules presented here are not exactly those of the SQL standard, as discussed in Section D.3.1.3.

If the path expression returns an empty node-set (typically, when it does not match) for a given row, the column will be set to NULL, unless a default_expression is specified; then the value resulting from evaluating that expression is used.

A default_expression, rather than being evaluated immediately when xmltable is called, is evaluated each time a default is needed for the column. If the expression qualifies as stable or immutable, the repeat evaluation may be skipped. This means that you can usefully use volatile functions like nextval in default_expression.

Columns may be marked NOT NULL. If the column_expression for a NOT NULL column does not match anything and there is no DEFAULT or the default_expression also evaluates to null, an error is reported.

Examples:

CREATE TABLE xmldata AS SELECT
xml $$
<ROWS>
  <ROW id="1">
    <COUNTRY_ID>AU</COUNTRY_ID>
    <COUNTRY_NAME>Australia</COUNTRY_NAME>
  </ROW>
  <ROW id="5">
    <COUNTRY_ID>JP</COUNTRY_ID>
    <COUNTRY_NAME>Japan</COUNTRY_NAME>
    <PREMIER_NAME>Shinzo Abe</PREMIER_NAME>
    <SIZE unit="sq_mi">145935</SIZE>
  </ROW>
  <ROW id="6">
    <COUNTRY_ID>SG</COUNTRY_ID>
    <COUNTRY_NAME>Singapore</COUNTRY_NAME>
    <SIZE unit="sq_km">697</SIZE>
  </ROW>
</ROWS>
$$ AS data;

SELECT xmltable.*
  FROM xmldata,
       XMLTABLE('//ROWS/ROW'
                PASSING data
                COLUMNS id int PATH '@id',
                        ordinality FOR ORDINALITY,
                        "COUNTRY_NAME" text,
                        country_id text PATH 'COUNTRY_ID',
                        size_sq_km float PATH 'SIZE[@unit = "sq_km"]',
                        size_other text PATH
                             'concat(SIZE[@unit!="sq_km"], " ", SIZE[@unit!="sq_km"]/@unit)',
                        premier_name text PATH 'PREMIER_NAME' DEFAULT 'not specified');

 id | ordinality | COUNTRY_NAME | country_id | size_sq_km |  size_other  | premier_name
----+------------+--------------+------------+------------+--------------+---------------
  1 |          1 | Australia    | AU         |            |              | not specified
  5 |          2 | Japan        | JP         |            | 145935 sq_mi | Shinzo Abe
  6 |          3 | Singapore    | SG         |        697 |              | not specified

The following example shows concatenation of multiple text() nodes, usage of the column name as XPath filter, and the treatment of whitespace, XML comments and processing instructions:

CREATE TABLE xmlelements AS SELECT
xml $$
  <root>
   <element>  Hello<!-- xyxxz -->2a2<?aaaaa?> <!--x-->  bbb<x>xxx</x>CC  </element>
  </root>
$$ AS data;

SELECT xmltable.*
  FROM xmlelements, XMLTABLE('/root' PASSING data COLUMNS element text);
         element
-------------------------
   Hello2a2   bbbxxxCC

The following example illustrates how the XMLNAMESPACES clause can be used to specify a list of namespaces used in the XML document as well as in the XPath expressions:

WITH xmldata(data) AS (VALUES ('
<example xmlns="http://example.com/myns" xmlns:B="http://example.com/b">
 <item foo="1" B:bar="2"/>
 <item foo="3" B:bar="4"/>
 <item foo="4" B:bar="5"/>
</example>'::xml)
)
SELECT xmltable.*
  FROM XMLTABLE(XMLNAMESPACES('http://example.com/myns' AS x,
                              'http://example.com/b' AS "B"),
             '/x:example/x:item'
                PASSING (SELECT data FROM xmldata)
                COLUMNS foo int PATH '@foo',
                  bar int PATH '@B:bar');
 foo | bar
-----+-----
   1 |   2
   3 |   4
   4 |   5
(3 rows)

9.15.4. Mapping Tables to XML

The following functions map the contents of relational tables to XML values. They can be thought of as XML export functionality:

table_to_xml ( table regclass, nulls boolean,
               tableforest boolean, targetns text ) → xml
query_to_xml ( query text, nulls boolean,
               tableforest boolean, targetns text ) → xml
cursor_to_xml ( cursor refcursor, count integer, nulls boolean,
                tableforest boolean, targetns text ) → xml

table_to_xml maps the content of the named table, passed as parameter table. The regclass type accepts strings identifying tables using the usual notation, including optional schema qualification and double quotes (see Section 8.19 for details). query_to_xml executes the query whose text is passed as parameter query and maps the result set. cursor_to_xml fetches the indicated number of rows from the cursor specified by the parameter cursor. This variant is recommended if large tables have to be mapped, because the result value is built up in memory by each function.

If tableforest is false, then the resulting XML document looks like this:

<tablename>
  <row>
    <columnname1>data</columnname1>
    <columnname2>data</columnname2>
  </row>

  <row>
    ...
  </row>

  ...
</tablename>

If tableforest is true, the result is an XML content fragment that looks like this:

<tablename>
  <columnname1>data</columnname1>
  <columnname2>data</columnname2>
</tablename>

<tablename>
  ...
</tablename>

...

If no table name is available, that is, when mapping a query or a cursor, the string table is used in the first format, row in the second format.

The choice between these formats is up to the user. The first format is a proper XML document, which will be important in many applications. The second format tends to be more useful in the cursor_to_xml function if the result values are to be reassembled into one document later on. The functions for producing XML content discussed above, in particular xmlelement, can be used to alter the results to taste.

The data values are mapped in the same way as described for the function xmlelement above.

The parameter nulls determines whether null values should be included in the output. If true, null values in columns are represented as:

<columnname xsi:nil="true"/>

where xsi is the XML namespace prefix for XML Schema Instance. An appropriate namespace declaration will be added to the result value. If false, columns containing null values are simply omitted from the output.

The parameter targetns specifies the desired XML namespace of the result. If no particular namespace is wanted, an empty string should be passed.

The following functions return XML Schema documents describing the mappings performed by the corresponding functions above:

table_to_xmlschema ( table regclass, nulls boolean,
                     tableforest boolean, targetns text ) → xml
query_to_xmlschema ( query text, nulls boolean,
                     tableforest boolean, targetns text ) → xml
cursor_to_xmlschema ( cursor refcursor, nulls boolean,
                      tableforest boolean, targetns text ) → xml

It is essential that the same parameters are passed in order to obtain matching XML data mappings and XML Schema documents.

The following functions produce XML data mappings and the corresponding XML Schema in one document (or forest), linked together. They can be useful where self-contained and self-describing results are wanted:

table_to_xml_and_xmlschema ( table regclass, nulls boolean,
                             tableforest boolean, targetns text ) → xml
query_to_xml_and_xmlschema ( query text, nulls boolean,
                             tableforest boolean, targetns text ) → xml

In addition, the following functions are available to produce analogous mappings of entire schemas or the entire current database:

schema_to_xml ( schema name, nulls boolean,
                tableforest boolean, targetns text ) → xml
schema_to_xmlschema ( schema name, nulls boolean,
                      tableforest boolean, targetns text ) → xml
schema_to_xml_and_xmlschema ( schema name, nulls boolean,
                              tableforest boolean, targetns text ) → xml

database_to_xml ( nulls boolean,
                  tableforest boolean, targetns text ) → xml
database_to_xmlschema ( nulls boolean,
                        tableforest boolean, targetns text ) → xml
database_to_xml_and_xmlschema ( nulls boolean,
                                tableforest boolean, targetns text ) → xml

These functions ignore tables that are not readable by the current user. The database-wide functions additionally ignore schemas that the current user does not have USAGE (lookup) privilege for.

Note that these potentially produce a lot of data, which needs to be built up in memory. When requesting content mappings of large schemas or databases, it might be worthwhile to consider mapping the tables separately instead, possibly even through a cursor.

The result of a schema content mapping looks like this:

<schemaname>

table1-mapping

table2-mapping

...

</schemaname>

where the format of a table mapping depends on the tableforest parameter as explained above.

The result of a database content mapping looks like this:

<dbname>

<schema1name>
  ...
</schema1name>

<schema2name>
  ...
</schema2name>

...

</dbname>

where the schema mapping is as above.

As an example of using the output produced by these functions, Example 9.1 shows an XSLT stylesheet that converts the output of table_to_xml_and_xmlschema to an HTML document containing a tabular rendition of the table data. In a similar manner, the results from these functions can be converted into other XML-based formats.

Example 9.1. XSLT Stylesheet for Converting SQL/XML Output to HTML

<?xml version="1.0"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns="http://www.w3.org/1999/xhtml"
>

  <xsl:output method="xml"
      doctype-system="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"
      doctype-public="-//W3C/DTD XHTML 1.0 Strict//EN"
      indent="yes"/>

  <xsl:template match="/*">
    <xsl:variable name="schema" select="//xsd:schema"/>
    <xsl:variable name="tabletypename"
                  select="$schema/xsd:element[@name=name(current())]/@type"/>
    <xsl:variable name="rowtypename"
                  select="$schema/xsd:complexType[@name=$tabletypename]/xsd:sequence/xsd:element[@name='row']/@type"/>

    <html>
      <head>
        <title><xsl:value-of select="name(current())"/></title>
      </head>
      <body>
        <table>
          <tr>
            <xsl:for-each select="$schema/xsd:complexType[@name=$rowtypename]/xsd:sequence/xsd:element/@name">
              <th><xsl:value-of select="."/></th>
            </xsl:for-each>
          </tr>

          <xsl:for-each select="row">
            <tr>
              <xsl:for-each select="*">
                <td><xsl:value-of select="."/></td>
              </xsl:for-each>
            </tr>
          </xsl:for-each>
        </table>
      </body>
    </html>
  </xsl:template>

</xsl:stylesheet>

[8] A result containing more than one element node at the top level, or non-whitespace text outside of an element, is an example of content form. An XPath result can be of neither form, for example if it returns an attribute node selected from the element that contains it. Such a result will be put into content form with each such disallowed node replaced by its string value, as defined for the XPath 1.0 string function.

Prev | Up | Next
9.14. UUID Functions | Home | 9.16. JSON Functions and Operators


9.16. JSON Functions and Operators
Prev Up Chapter 9. Functions and Operators Home Next

9.16. JSON Functions and Operators

9.16.1. Processing and Creating JSON Data
9.16.2. The SQL/JSON Path Language

This section describes:

  • functions and operators for processing and creating JSON data

  • the SQL/JSON path language

To learn more about the SQL/JSON standard, see [sqltr-19075-6]. For details on JSON types supported in PostgreSQL, see Section 8.14.

9.16.1. Processing and Creating JSON Data

Table 9.44 shows the operators that are available for use with JSON data types (see Section 8.14). In addition, the usual comparison operators shown in Table 9.1 are available for jsonb, though not for json. The comparison operators follow the ordering rules for B-tree operations outlined in Section 8.14.4.

Table 9.44. json and jsonb Operators

Operator

Description

Example(s)

json -> integer → json

jsonb -> integer → jsonb

Extracts n'th element of JSON array (array elements are indexed from zero, but negative integers count from the end).

'[{"a":"foo"},{"b":"bar"},{"c":"baz"}]'::json -> 2 → {"c":"baz"}

'[{"a":"foo"},{"b":"bar"},{"c":"baz"}]'::json -> -3 → {"a":"foo"}

json -> text → json

jsonb -> text → jsonb

Extracts JSON object field with the given key.

'{"a": {"b":"foo"}}'::json -> 'a' → {"b":"foo"}

json ->> integer → text

jsonb ->> integer → text

Extracts n'th element of JSON array, as text.

'[1,2,3]'::json ->> 2 → 3

json ->> text → text

jsonb ->> text → text

Extracts JSON object field with the given key, as text.

'{"a":1,"b":2}'::json ->> 'b' → 2

json #> text[] → json

jsonb #> text[] → jsonb

Extracts JSON sub-object at the specified path, where path elements can be either field keys or array indexes.

'{"a": {"b": ["foo","bar"]}}'::json #> '{a,b,1}' → "bar"

json #>> text[] → text

jsonb #>> text[] → text

Extracts JSON sub-object at the specified path as text.

'{"a": {"b": ["foo","bar"]}}'::json #>> '{a,b,1}' → bar

Note

The field/element/path extraction operators return NULL, rather than failing, if the JSON input does not have the right structure to match the request; for example if no such key or array element exists.

Some further operators exist only for jsonb, as shown in Table 9.45. Section 8.14.4 describes how these operators can be used to effectively search indexed jsonb data.

Table 9.45. Additional jsonb Operators

Operator

Description

Example(s)

jsonb @> jsonb → boolean

Does the first JSON value contain the second? (See Section 8.14.3 for details about containment.)

'{"a":1, "b":2}'::jsonb @> '{"b":2}'::jsonb → t

jsonb <@ jsonb → boolean

Is the first JSON value contained in the second?

'{"b":2}'::jsonb <@ '{"a":1, "b":2}'::jsonb → t

jsonb ? text → boolean

Does the text string exist as a top-level key or array element within the JSON value?

'{"a":1, "b":2}'::jsonb ? 'b' → t

'["a", "b", "c"]'::jsonb ? 'b' → t

jsonb ?| text[] → boolean

Do any of the strings in the text array exist as top-level keys or array elements?

'{"a":1, "b":2, "c":3}'::jsonb ?| array['b', 'd'] → t

jsonb ?& text[] → boolean

Do all of the strings in the text array exist as top-level keys or array elements?

'["a", "b", "c"]'::jsonb ?& array['a', 'b'] → t

jsonb || jsonb → jsonb

Concatenates two jsonb values. Concatenating two arrays generates an array containing all the elements of each input. Concatenating two objects generates an object containing the union of their keys, taking the second object's value when there are duplicate keys. All other cases are treated by converting a non-array input into a single-element array, and then proceeding as for two arrays. Does not operate recursively: only the top-level array or object structure is merged.

'["a", "b"]'::jsonb || '["a", "d"]'::jsonb → ["a", "b", "a", "d"]

'{"a": "b"}'::jsonb || '{"c": "d"}'::jsonb → {"a": "b", "c": "d"}

'[1, 2]'::jsonb || '3'::jsonb → [1, 2, 3]

'{"a": "b"}'::jsonb || '42'::jsonb → [{"a": "b"}, 42]

To append an array to another array as a single entry, wrap it in an additional layer of array, for example:

'[1, 2]'::jsonb || jsonb_build_array('[3, 4]'::jsonb) → [1, 2, [3, 4]]

jsonb - text → jsonb

Deletes a key (and its value) from a JSON object, or matching string value(s) from a JSON array.

'{"a": "b", "c": "d"}'::jsonb - 'a' → {"c": "d"}

'["a", "b", "c", "b"]'::jsonb - 'b' → ["a", "c"]

jsonb - text[] → jsonb

Deletes all matching keys or array elements from the left operand.

'{"a": "b", "c": "d"}'::jsonb - '{a,c}'::text[] → {}

jsonb - integer → jsonb

Deletes the array element with specified index (negative integers count from the end). Throws an error if JSON value is not an array.

'["a", "b"]'::jsonb - 1 → ["a"]

jsonb #- text[] → jsonb

Deletes the field or array element at the specified path, where path elements can be either field keys or array indexes.

'["a", {"b":1}]'::jsonb #- '{1,b}' → ["a", {}]

jsonb @? jsonpath → boolean

Does JSON path return any item for the specified JSON value?

'{"a":[1,2,3,4,5]}'::jsonb @? '$.a[*] ? (@ > 2)' → t

jsonb @@ jsonpath → boolean

Returns the result of a JSON path predicate check for the specified JSON value. Only the first item of the result is taken into account. If the result is not Boolean, then NULL is returned.

'{"a":[1,2,3,4,5]}'::jsonb @@ '$.a[*] > 2' → t

Note

The jsonpath operators @? and @@ suppress the following errors: missing object field or array element, unexpected JSON item type, datetime and numeric errors. The jsonpath-related functions described below can also be told to suppress these types of errors. This behavior might be helpful when searching JSON document collections of varying structure.

Table 9.46 shows the functions that are available for constructing json and jsonb values.

Table 9.46. JSON Creation Functions

Function

Description

Example(s)

to_json ( anyelement ) → json

to_jsonb ( anyelement ) → jsonb

Converts any SQL value to json or jsonb. Arrays and composites are converted recursively to arrays and objects (multidimensional arrays become arrays of arrays in JSON). Otherwise, if there is a cast from the SQL data type to json, the cast function will be used to perform the conversion;[a] otherwise, a scalar JSON value is produced. For any scalar other than a number, a Boolean, or a null value, the text representation will be used, with escaping as necessary to make it a valid JSON string value.

to_json('Fred said "Hi."'::text) → "Fred said \"Hi.\""

to_jsonb(row(42, 'Fred said "Hi."'::text)) → {"f1": 42, "f2": "Fred said \"Hi.\""}

array_to_json ( anyarray [, boolean ] ) → json

Converts an SQL array to a JSON array. The behavior is the same as to_json except that line feeds will be added between top-level array elements if the optional boolean parameter is true.

array_to_json('{{1,5},{99,100}}'::int[]) → [[1,5],[99,100]]

row_to_json ( record [, boolean ] ) → json

Converts an SQL composite value to a JSON object. The behavior is the same as to_json except that line feeds will be added between top-level elements if the optional boolean parameter is true.

row_to_json(row(1,'foo')) → {"f1":1,"f2":"foo"}

json_build_array ( VARIADIC "any" ) → json

jsonb_build_array ( VARIADIC "any" ) → jsonb

Builds a possibly-heterogeneously-typed JSON array out of a variadic argument list. Each argument is converted as per to_json or to_jsonb.

json_build_array(1, 2, 'foo', 4, 5) → [1, 2, "foo", 4, 5]

json_build_object ( VARIADIC "any" ) → json

jsonb_build_object ( VARIADIC "any" ) → jsonb

Builds a JSON object out of a variadic argument list. By convention, the argument list consists of alternating keys and values. Key arguments are coerced to text; value arguments are converted as per to_json or to_jsonb.

json_build_object('foo', 1, 2, row(3,'bar')) → {"foo" : 1, "2" : {"f1":3,"f2":"bar"}}

json_object ( text[] ) → json

jsonb_object ( text[] ) → jsonb

Builds a JSON object out of a text array. The array must have either exactly one dimension with an even number of members, in which case they are taken as alternating key/value pairs, or two dimensions such that each inner array has exactly two elements, which are taken as a key/value pair. All values are converted to JSON strings.

json_object('{a, 1, b, "def", c, 3.5}') → {"a" : "1", "b" : "def", "c" : "3.5"}

json_object('{{a, 1}, {b, "def"}, {c, 3.5}}') → {"a" : "1", "b" : "def", "c" : "3.5"}

json_object ( keys text[], values text[] ) → json

jsonb_object ( keys text[], values text[] ) → jsonb

This form of json_object takes keys and values pairwise from separate text arrays. Otherwise it is identical to the one-argument form.

json_object('{a,b}', '{1,2}') → {"a": "1", "b": "2"}

[a] For example, the hstore extension has a cast from hstore to json, so that hstore values converted via the JSON creation functions will be represented as JSON objects, not as primitive string values.

Table 9.47 shows the functions that are available for processing json and jsonb values.

Table 9.47. JSON Processing Functions

Function

Description

Example(s)

json_array_elements ( json ) → setof json

jsonb_array_elements ( jsonb ) → setof jsonb

Expands the top-level JSON array into a set of JSON values.

select * from json_array_elements('[1,true, [2,false]]') →

   value
-----------
 1
 true
 [2,false]

json_array_elements_text ( json ) → setof text

jsonb_array_elements_text ( jsonb ) → setof text

Expands the top-level JSON array into a set of text values.

select * from json_array_elements_text('["foo", "bar"]') →

   value
-----------
 foo
 bar

json_array_length ( json ) → integer

jsonb_array_length ( jsonb ) → integer

Returns the number of elements in the top-level JSON array.

json_array_length('[1,2,3,{"f1":1,"f2":[5,6]},4]') → 5

json_each ( json ) → setof record ( key text, value json )

jsonb_each ( jsonb ) → setof record ( key text, value jsonb )

Expands the top-level JSON object into a set of key/value pairs.

select * from json_each('{"a":"foo", "b":"bar"}') →

 key | value
-----+-------
 a   | "foo"
 b   | "bar"

json_each_text ( json ) → setof record ( key text, value text )

jsonb_each_text ( jsonb ) → setof record ( key text, value text )

Expands the top-level JSON object into a set of key/value pairs. The returned values will be of type text.

select * from json_each_text('{"a":"foo", "b":"bar"}') →

 key | value
-----+-------
 a   | foo
 b   | bar

json_extract_path ( from_json json, VARIADIC path_elems text[] ) → json

jsonb_extract_path ( from_json jsonb, VARIADIC path_elems text[] ) → jsonb

Extracts JSON sub-object at the specified path. (This is functionally equivalent to the #> operator, but writing the path out as a variadic list can be more convenient in some cases.)

json_extract_path('{"f2":{"f3":1},"f4":{"f5":99,"f6":"foo"}}', 'f4', 'f6') → "foo"

json_extract_path_text ( from_json json, VARIADIC path_elems text[] ) → text

jsonb_extract_path_text ( from_json jsonb, VARIADIC path_elems text[] ) → text

Extracts JSON sub-object at the specified path as text. (This is functionally equivalent to the #>> operator.)

json_extract_path_text('{"f2":{"f3":1},"f4":{"f5":99,"f6":"foo"}}', 'f4', 'f6') → foo

json_object_keys ( json ) → setof text

jsonb_object_keys ( jsonb ) → setof text

Returns the set of keys in the top-level JSON object.

select * from json_object_keys('{"f1":"abc","f2":{"f3":"a", "f4":"b"}}') →

 json_object_keys
------------------
 f1
 f2

json_populate_record ( base anyelement, from_json json ) → anyelement

jsonb_populate_record ( base anyelement, from_json jsonb ) → anyelement

Expands the top-level JSON object to a row having the composite type of the base argument. The JSON object is scanned for fields whose names match column names of the output row type, and their values are inserted into those columns of the output. (Fields that do not correspond to any output column name are ignored.) In typical use, the value of base is just NULL, which means that any output columns that do not match any object field will be filled with nulls. However, if base isn't NULL then the values it contains will be used for unmatched columns.

To convert a JSON value to the SQL type of an output column, the following rules are applied in sequence:

  • A JSON null value is converted to an SQL null in all cases.

  • If the output column is of type json or jsonb, the JSON value is just reproduced exactly.

  • If the output column is a composite (row) type, and the JSON value is a JSON object, the fields of the object are converted to columns of the output row type by recursive application of these rules.

  • Likewise, if the output column is an array type and the JSON value is a JSON array, the elements of the JSON array are converted to elements of the output array by recursive application of these rules.

  • Otherwise, if the JSON value is a string, the contents of the string are fed to the input conversion function for the column's data type.

  • Otherwise, the ordinary text representation of the JSON value is fed to the input conversion function for the column's data type.

While the example below uses a constant JSON value, typical use would be to reference a json or jsonb column laterally from another table in the query's FROM clause. Writing json_populate_record in the FROM clause is good practice, since all of the extracted columns are available for use without duplicate function calls.

create type subrowtype as (d int, e text); create type myrowtype as (a int, b text[], c subrowtype);

select * from json_populate_record(null::myrowtype, '{"a": 1, "b": ["2", "a b"], "c": {"d": 4, "e": "a b c"}, "x": "foo"}') →

 a |   b       |      c
---+-----------+-------------
 1 | {2,"a b"} | (4,"a b c")

json_populate_recordset ( base anyelement, from_json json ) → setof anyelement

jsonb_populate_recordset ( base anyelement, from_json jsonb ) → setof anyelement

Expands the top-level JSON array of objects to a set of rows having the composite type of the base argument. Each element of the JSON array is processed as described above for json[b]_populate_record.

create type twoints as (a int, b int);

select * from json_populate_recordset(null::twoints, '[{"a":1,"b":2}, {"a":3,"b":4}]') →

 a | b
---+---
 1 | 2
 3 | 4

json_to_record ( json ) → record

jsonb_to_record ( jsonb ) → record

Expands the top-level JSON object to a row having the composite type defined by an AS clause. (As with all functions returning record, the calling query must explicitly define the structure of the record with an AS clause.) The output record is filled from fields of the JSON object, in the same way as described above for json[b]_populate_record. Since there is no input record value, unmatched columns are always filled with nulls.

create type myrowtype as (a int, b text);

select * from json_to_record('{"a":1,"b":[1,2,3],"c":[1,2,3],"e":"bar","r": {"a": 123, "b": "a b c"}}') as x(a int, b text, c int[], d text, r myrowtype) →

 a |    b    |    c    | d |       r
---+---------+---------+---+---------------
 1 | [1,2,3] | {1,2,3} |   | (123,"a b c")

json_to_recordset ( json ) → setof record

jsonb_to_recordset ( jsonb ) → setof record

Expands the top-level JSON array of objects to a set of rows having the composite type defined by an AS clause. (As with all functions returning record, the calling query must explicitly define the structure of the record with an AS clause.) Each element of the JSON array is processed as described above for json[b]_populate_record.

select * from json_to_recordset('[{"a":1,"b":"foo"}, {"a":"2","c":"bar"}]') as x(a int, b text) →

 a |  b
---+-----
 1 | foo
 2 |

jsonb_set ( target jsonb, path text[], new_value jsonb [, create_if_missing boolean ] ) → jsonb

Returns target with the item designated by path replaced by new_value, or with new_value added if create_if_missing is true (which is the default) and the item designated by path does not exist. All earlier steps in the path must exist, or the target is returned unchanged. As with the path oriented operators, negative integers that appear in the path count from the end of JSON arrays. If the last path step is an array index that is out of range, and create_if_missing is true, the new value is added at the beginning of the array if the index is negative, or at the end of the array if it is positive.

jsonb_set('[{"f1":1,"f2":null},2,null,3]', '{0,f1}', '[2,3,4]', false) → [{"f1": [2, 3, 4], "f2": null}, 2, null, 3]

jsonb_set('[{"f1":1,"f2":null},2]', '{0,f3}', '[2,3,4]') → [{"f1": 1, "f2": null, "f3": [2, 3, 4]}, 2]

jsonb_set_lax ( target jsonb, path text[], new_value jsonb [, create_if_missing boolean [, null_value_treatment text ]] ) → jsonb

If new_value is not NULL, behaves identically to jsonb_set. Otherwise behaves according to the value of null_value_treatment which must be one of 'raise_exception', 'use_json_null', 'delete_key', or 'return_target'. The default is 'use_json_null'.

jsonb_set_lax('[{"f1":1,"f2":null},2,null,3]', '{0,f1}', null) → [{"f1":null,"f2":null},2,null,3]

jsonb_set_lax('[{"f1":99,"f2":null},2]', '{0,f3}', null, true, 'return_target') → [{"f1": 99, "f2": null}, 2]

jsonb_insert ( target jsonb, path text[], new_value jsonb [, insert_after boolean ] ) → jsonb

Returns target with new_value inserted. If the item designated by the path is an array element, new_value will be inserted before that item if insert_after is false (which is the default), or after it if insert_after is true. If the item designated by the path is an object field, new_value will be inserted only if the object does not already contain that key. All earlier steps in the path must exist, or the target is returned unchanged. As with the path oriented operators, negative integers that appear in the path count from the end of JSON arrays. If the last path step is an array index that is out of range, the new value is added at the beginning of the array if the index is negative, or at the end of the array if it is positive.

jsonb_insert('{"a": [0,1,2]}', '{a, 1}', '"new_value"') → {"a": [0, "new_value", 1, 2]}

jsonb_insert('{"a": [0,1,2]}', '{a, 1}', '"new_value"', true) → {"a": [0, 1, "new_value", 2]}

json_strip_nulls ( json ) → json

jsonb_strip_nulls ( jsonb ) → jsonb

Deletes all object fields that have null values from the given JSON value, recursively. Null values that are not object fields are untouched.

json_strip_nulls('[{"f1":1, "f2":null}, 2, null, 3]') → [{"f1":1},2,null,3]

jsonb_path_exists ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → boolean

Checks whether the JSON path returns any item for the specified JSON value. If the vars argument is specified, it must be a JSON object, and its fields provide named values to be substituted into the jsonpath expression. If the silent argument is specified and is true, the function suppresses the same errors as the @? and @@ operators do.

jsonb_path_exists('{"a":[1,2,3,4,5]}', '$.a[*] ? (@ >= $min && @ <= $max)', '{"min":2, "max":4}') → t

jsonb_path_match ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → boolean

Returns the result of a JSON path predicate check for the specified JSON value. Only the first item of the result is taken into account. If the result is not Boolean, then NULL is returned. The optional vars and silent arguments act the same as for jsonb_path_exists.

jsonb_path_match('{"a":[1,2,3,4,5]}', 'exists($.a[*] ? (@ >= $min && @ <= $max))', '{"min":2, "max":4}') → t

jsonb_path_query ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → setof jsonb

Returns all JSON items returned by the JSON path for the specified JSON value. The optional vars and silent arguments act the same as for jsonb_path_exists.

select * from jsonb_path_query('{"a":[1,2,3,4,5]}', '$.a[*] ? (@ >= $min && @ <= $max)', '{"min":2, "max":4}') →

 jsonb_path_query
------------------
 2
 3
 4

jsonb_path_query_array ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → jsonb

Returns all JSON items returned by the JSON path for the specified JSON value, as a JSON array. The optional vars and silent arguments act the same as for jsonb_path_exists.

jsonb_path_query_array('{"a":[1,2,3,4,5]}', '$.a[*] ? (@ >= $min && @ <= $max)', '{"min":2, "max":4}') → [2, 3, 4]

jsonb_path_query_first ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → jsonb

Returns the first JSON item returned by the JSON path for the specified JSON value. Returns NULL if there are no results. The optional vars and silent arguments act the same as for jsonb_path_exists.

jsonb_path_query_first('{"a":[1,2,3,4,5]}', '$.a[*] ? (@ >= $min && @ <= $max)', '{"min":2, "max":4}') → 2

jsonb_path_exists_tz ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → boolean

jsonb_path_match_tz ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → boolean

jsonb_path_query_tz ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → setof jsonb

jsonb_path_query_array_tz ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → jsonb

jsonb_path_query_first_tz ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → jsonb

These functions act like their counterparts described above without the _tz suffix, except that these functions support comparisons of date/time values that require timezone-aware conversions. The example below requires interpretation of the date-only value 2015-08-02 as a timestamp with time zone, so the result depends on the current TimeZone setting. Due to this dependency, these functions are marked as stable, which means these functions cannot be used in indexes. Their counterparts are immutable, and so can be used in indexes; but they will throw errors if asked to make such comparisons.

jsonb_path_exists_tz('["2015-08-01 12:00:00 -05"]', '$[*] ? (@.datetime() < "2015-08-02".datetime())') → t

jsonb_pretty ( jsonb ) → text

Converts the given JSON value to pretty-printed, indented text.

jsonb_pretty('[{"f1":1,"f2":null}, 2]') →

[
    {
        "f1": 1,
        "f2": null
    },
    2
]

json_typeof ( json ) → text

jsonb_typeof ( jsonb ) → text

Returns the type of the top-level JSON value as a text string. Possible types are object, array, string, number, boolean, and null. (The null result should not be confused with an SQL NULL; see the examples.)

json_typeof('-123.4') → number

json_typeof('null'::json) → null

json_typeof(NULL::json) IS NULL → t

See also Section 9.21 for the aggregate function json_agg which aggregates record values as JSON, the aggregate function json_object_agg which aggregates pairs of values into a JSON object, and their jsonb equivalents, jsonb_agg and jsonb_object_agg.

9.16.2. The SQL/JSON Path Language

SQL/JSON path expressions specify the items to be retrieved from the JSON data, similar to XPath expressions used for SQL access to XML. In PostgreSQL, path expressions are implemented as the jsonpath data type and can use any elements described in Section 8.14.7.

JSON query functions and operators pass the provided path expression to the path engine for evaluation. If the expression matches the queried JSON data, the corresponding JSON item, or set of items, is returned. Path expressions are written in the SQL/JSON path language and can include arithmetic expressions and functions.

A path expression consists of a sequence of elements allowed by the jsonpath data type. The path expression is normally evaluated from left to right, but you can use parentheses to change the order of operations. If the evaluation is successful, a sequence of JSON items is produced, and the evaluation result is returned to the JSON query function that completes the specified computation.

To refer to the JSON value being queried (the context item), use the $ variable in the path expression. It can be followed by one or more accessor operators, which go down the JSON structure level by level to retrieve sub-items of the context item. Each operator that follows deals with the result of the previous evaluation step.

For example, suppose you have some JSON data from a GPS tracker that you would like to parse, such as:

{
  "track": {
    "segments": [
      {
        "location":   [ 47.763, 13.4034 ],
        "start time": "2018-10-14 10:05:14",
        "HR": 73
      },
      {
        "location":   [ 47.706, 13.2635 ],
        "start time": "2018-10-14 10:39:21",
        "HR": 135
      }
    ]
  }
}

To retrieve the available track segments, you need to use the .key accessor operator to descend through surrounding JSON objects:

$.track.segments

To retrieve the contents of an array, you typically use the [*] operator. For example, the following path will return the location coordinates for all the available track segments:

$.track.segments[*].location

To return the coordinates of the first segment only, you can specify the corresponding subscript in the [] accessor operator. Recall that JSON array indexes are 0-relative:

$.track.segments[0].location

The result of each path evaluation step can be processed by one or more jsonpath operators and methods listed in Section 9.16.2.2. Each method name must be preceded by a dot. For example, you can get the size of an array:

$.track.segments.size()

More examples of using jsonpath operators and methods within path expressions appear below in Section 9.16.2.2.

When defining a path, you can also use one or more filter expressions that work similarly to the WHERE clause in SQL. A filter expression begins with a question mark and provides a condition in parentheses:

? (condition)

Filter expressions must be written just after the path evaluation step to which they should apply. The result of that step is filtered to include only those items that satisfy the provided condition. SQL/JSON defines three-valued logic, so the condition can be true, false, or unknown. The unknown value plays the same role as SQL NULL and can be tested for with the is unknown predicate. Further path evaluation steps use only those items for which the filter expression returned true.

The functions and operators that can be used in filter expressions are listed in Table 9.49. Within a filter expression, the @ variable denotes the value being filtered (i.e., one result of the preceding path step). You can write accessor operators after @ to retrieve component items.

For example, suppose you would like to retrieve all heart rate values higher than 130. You can achieve this using the following expression:

$.track.segments[*].HR ? (@ > 130)

To get the start times of segments with such values, you have to filter out irrelevant segments before returning the start times, so the filter expression is applied to the previous step, and the path used in the condition is different:

$.track.segments[*] ? (@.HR > 130)."start time"

You can use several filter expressions in sequence, if required. For example, the following expression selects start times of all segments that contain locations with relevant coordinates and high heart rate values:

$.track.segments[*] ? (@.location[1] < 13.4) ? (@.HR > 130)."start time"

Using filter expressions at different nesting levels is also allowed. The following example first filters all segments by location, and then returns high heart rate values for these segments, if available:

$.track.segments[*] ? (@.location[1] < 13.4).HR ? (@ > 130)

You can also nest filter expressions within each other:

$.track ? (exists(@.segments[*] ? (@.HR > 130))).segments.size()

This expression returns the size of the track if it contains any segments with high heart rate values, or an empty sequence otherwise.

PostgreSQL's implementation of the SQL/JSON path language has the following deviations from the SQL/JSON standard:

  • A path expression can be a Boolean predicate, although the SQL/JSON standard allows predicates only in filters. This is necessary for implementation of the @@ operator. For example, the following jsonpath expression is valid in PostgreSQL:

    $.track.segments[*].HR < 70

  • There are minor differences in the interpretation of regular expression patterns used in like_regex filters, as described in Section 9.16.2.3.

9.16.2.1. Strict and Lax Modes

When you query JSON data, the path expression may not match the actual JSON data structure. An attempt to access a non-existent member of an object or element of an array results in a structural error. SQL/JSON path expressions have two modes of handling structural errors:

  • lax (default) — the path engine implicitly adapts the queried data to the specified path. Any remaining structural errors are suppressed and converted to empty SQL/JSON sequences.

  • strict — if a structural error occurs, an error is raised.

The lax mode facilitates matching of a JSON document structure and path expression if the JSON data does not conform to the expected schema. If an operand does not match the requirements of a particular operation, it can be automatically wrapped as an SQL/JSON array or unwrapped by converting its elements into an SQL/JSON sequence before performing this operation. Besides, comparison operators automatically unwrap their operands in the lax mode, so you can compare SQL/JSON arrays out-of-the-box. An array of size 1 is considered equal to its sole element. Automatic unwrapping is not performed only when:

  • The path expression contains type() or size() methods that return the type and the number of elements in the array, respectively.

  • The queried JSON data contain nested arrays. In this case, only the outermost array is unwrapped, while all the inner arrays remain unchanged. Thus, implicit unwrapping can only go one level down within each path evaluation step.

For example, when querying the GPS data listed above, you can abstract from the fact that it stores an array of segments when using the lax mode:

lax $.track.segments.location

In the strict mode, the specified path must exactly match the structure of the queried JSON document to return an SQL/JSON item, so using this path expression will cause an error. To get the same result as in the lax mode, you have to explicitly unwrap the segments array:

strict $.track.segments[*].location

The .** accessor can lead to surprising results when using the lax mode. For instance, the following query selects every HR value twice:

lax $.**.HR

This happens because the .** accessor selects both the segments array and each of its elements, while the .HR accessor automatically unwraps arrays when using the lax mode. To avoid surprising results, we recommend using the .** accessor only in the strict mode. The following query selects each HR value just once:

strict $.**.HR

9.16.2.2. SQL/JSON Path Operators and Methods

Table 9.48 shows the operators and methods available in jsonpath. Note that while the unary operators and methods can be applied to multiple values resulting from a preceding path step, the binary operators (addition etc.) can only be applied to single values.

Table 9.48. jsonpath Operators and Methods

Operator/Method

Description

Example(s)

number + number → number

Addition

jsonb_path_query('[2]', '$[0] + 3') → 5

+ number → number

Unary plus (no operation); unlike addition, this can iterate over multiple values

jsonb_path_query_array('{"x": [2,3,4]}', '+ $.x') → [2, 3, 4]

number - number → number

Subtraction

jsonb_path_query('[2]', '7 - $[0]') → 5

- number → number

Negation; unlike subtraction, this can iterate over multiple values

jsonb_path_query_array('{"x": [2,3,4]}', '- $.x') → [-2, -3, -4]

number * number → number

Multiplication

jsonb_path_query('[4]', '2 * $[0]') → 8

number / number → number

Division

jsonb_path_query('[8.5]', '$[0] / 2') → 4.2500000000000000

number % number → number

Modulo (remainder)

jsonb_path_query('[32]', '$[0] % 10') → 2

value . type() → string

Type of the JSON item (see json_typeof)

jsonb_path_query_array('[1, "2", {}]', '$[*].type()') → ["number", "string", "object"]

value . size() → number

Size of the JSON item (number of array elements, or 1 if not an array)

jsonb_path_query('{"m": [11, 15]}', '$.m.size()') → 2

value . double() → number

Approximate floating-point number converted from a JSON number or string

jsonb_path_query('{"len": "1.9"}', '$.len.double() * 2') → 3.8

number . ceiling() → number

Nearest integer greater than or equal to the given number

jsonb_path_query('{"h": 1.3}', '$.h.ceiling()') → 2

number . floor() → number

Nearest integer less than or equal to the given number

jsonb_path_query('{"h": 1.7}', '$.h.floor()') → 1

number . abs() → number

Absolute value of the given number

jsonb_path_query('{"z": -0.3}', '$.z.abs()') → 0.3

string . datetime() → datetime_type (see note)

Date/time value converted from a string

jsonb_path_query('["2015-8-1", "2015-08-12"]', '$[*] ? (@.datetime() < "2015-08-2".datetime())') → "2015-8-1"

string . datetime(template) → datetime_type (see note)

Date/time value converted from a string using the specified to_timestamp template

jsonb_path_query_array('["12:30", "18:40"]', '$[*].datetime("HH24:MI")') → ["12:30:00", "18:40:00"]

object . keyvalue() → array

The object's key-value pairs, represented as an array of objects containing three fields: "key", "value", and "id"; "id" is a unique identifier of the object the key-value pair belongs to

jsonb_path_query_array('{"x": "20", "y": 32}', '$.keyvalue()') → [{"id": 0, "key": "x", "value": "20"}, {"id": 0, "key": "y", "value": 32}]

Note

The result type of the datetime() and datetime(template) methods can be date, timetz, time, timestamptz, or timestamp. Both methods determine their result type dynamically.

The datetime() method sequentially tries to match its input string to the ISO formats for date, timetz, time, timestamptz, and timestamp. It stops on the first matching format and emits the corresponding data type.

The datetime(template) method determines the result type according to the fields used in the provided template string.

The datetime() and datetime(template) methods use the same parsing rules as the to_timestamp SQL function does (see Section 9.8), with three exceptions. First, these methods don't allow unmatched template patterns. Second, only the following separators are allowed in the template string: minus sign, period, solidus (slash), comma, apostrophe, semicolon, colon and space. Third, separators in the template string must exactly match the input string.

If different date/time types need to be compared, an implicit cast is applied. A date value can be cast to timestamp or timestamptz, timestamp can be cast to timestamptz, and time to timetz. However, all but the first of these conversions depend on the current TimeZone setting, and thus can only be performed within timezone-aware jsonpath functions.

Table 9.49 shows the available filter expression elements.

Table 9.49. jsonpath Filter Expression Elements

Predicate/Value

Description

Example(s)

value == value → boolean

Equality comparison (this, and the other comparison operators, work on all JSON scalar values)

jsonb_path_query_array('[1, "a", 1, 3]', '$[*] ? (@ == 1)') → [1, 1]

jsonb_path_query_array('[1, "a", 1, 3]', '$[*] ? (@ == "a")') → ["a"]

value != value → boolean

value <> value → boolean

Non-equality comparison

jsonb_path_query_array('[1, 2, 1, 3]', '$[*] ? (@ != 1)') → [2, 3]

jsonb_path_query_array('["a", "b", "c"]', '$[*] ? (@ <> "b")') → ["a", "c"]

value < value → boolean

Less-than comparison

jsonb_path_query_array('[1, 2, 3]', '$[*] ? (@ < 2)') → [1]

value <= value → boolean

Less-than-or-equal-to comparison

jsonb_path_query_array('["a", "b", "c"]', '$[*] ? (@ <= "b")') → ["a", "b"]

value > value → boolean

Greater-than comparison

jsonb_path_query_array('[1, 2, 3]', '$[*] ? (@ > 2)') → [3]

value >= value → boolean

Greater-than-or-equal-to comparison

jsonb_path_query_array('[1, 2, 3]', '$[*] ? (@ >= 2)') → [2, 3]

true → boolean

JSON constant true

jsonb_path_query('[{"name": "John", "parent": false}, {"name": "Chris", "parent": true}]', '$[*] ? (@.parent == true)') → {"name": "Chris", "parent": true}

false → boolean

JSON constant false

jsonb_path_query('[{"name": "John", "parent": false}, {"name": "Chris", "parent": true}]', '$[*] ? (@.parent == false)') → {"name": "John", "parent": false}

null → value

JSON constant null (note that, unlike in SQL, comparison to null works normally)

jsonb_path_query('[{"name": "Mary", "job": null}, {"name": "Michael", "job": "driver"}]', '$[*] ? (@.job == null) .name') → "Mary"

boolean && boolean → boolean

Boolean AND

jsonb_path_query('[1, 3, 7]', '$[*] ? (@ > 1 && @ < 5)') → 3

boolean || boolean → boolean

Boolean OR

jsonb_path_query('[1, 3, 7]', '$[*] ? (@ < 1 || @ > 5)') → 7

! boolean → boolean

Boolean NOT

jsonb_path_query('[1, 3, 7]', '$[*] ? (!(@ < 5))') → 7

boolean is unknown → boolean

Tests whether a Boolean condition is unknown.

jsonb_path_query('[-1, 2, 7, "foo"]', '$[*] ? ((@ > 0) is unknown)') → "foo"

string like_regex string [ flag string ] → boolean

Tests whether the first operand matches the regular expression given by the second operand, optionally with modifications described by a string of flag characters (see Section 9.16.2.3).

jsonb_path_query_array('["abc", "abd", "aBdC", "abdacb", "babc"]', '$[*] ? (@ like_regex "^ab.*c")') → ["abc", "abdacb"]

jsonb_path_query_array('["abc", "abd", "aBdC", "abdacb", "babc"]', '$[*] ? (@ like_regex "^ab.*c" flag "i")') → ["abc", "aBdC", "abdacb"]

string starts with string → boolean

Tests whether the second operand is an initial substring of the first operand.

jsonb_path_query('["John Smith", "Mary Stone", "Bob Johnson"]', '$[*] ? (@ starts with "John")') → "John Smith"

exists ( path_expression ) → boolean

Tests whether a path expression matches at least one SQL/JSON item. Returns unknown if the path expression would result in an error; the second example uses this to avoid a no-such-key error in strict mode.

jsonb_path_query('{"x": [1, 2], "y": [2, 4]}', 'strict $.* ? (exists (@ ? (@[*] > 2)))') → [2, 4]

jsonb_path_query_array('{"value": 41}', 'strict $ ? (exists (@.name)) .name') → []

9.16.2.3. SQL/JSON Regular Expressions

SQL/JSON path expressions allow matching text to a regular expression with the like_regex filter. For example, the following SQL/JSON path query would case-insensitively match all strings in an array that start with an English vowel:

$[*] ? (@ like_regex "^[aeiou]" flag "i")

The optional flag string may include one or more of the characters i for case-insensitive match, m to allow ^ and $ to match at newlines, s to allow . to match a newline, and q to quote the whole pattern (reducing the behavior to a simple substring match).

The SQL/JSON standard borrows its definition for regular expressions from the LIKE_REGEX operator, which in turn uses the XQuery standard. PostgreSQL does not currently support the LIKE_REGEX operator. Therefore, the like_regex filter is implemented using the POSIX regular expression engine described in Section 9.7.3. This leads to various minor discrepancies from standard SQL/JSON behavior, which are cataloged in Section 9.7.3.8. Note, however, that the flag-letter incompatibilities described there do not apply to SQL/JSON, as it translates the XQuery flag letters to match what the POSIX engine expects.

Keep in mind that the pattern argument of like_regex is a JSON path string literal, written according to the rules given in Section 8.14.7. This means in particular that any backslashes you want to use in the regular expression must be doubled. For example, to match string values of the root document that contain only digits:

$.* ? (@ like_regex "^\\d+$")

Prev | Up | Next
9.15. XML Functions | Home | 9.17. Sequence Manipulation Functions


9.17. Sequence Manipulation Functions
Prev Up Chapter 9. Functions and Operators Home Next

9.17. Sequence Manipulation Functions

This section describes functions for operating on sequence objects, also called sequence generators or just sequences. Sequence objects are special single-row tables created with CREATE SEQUENCE. Sequence objects are commonly used to generate unique identifiers for rows of a table. The sequence functions, listed in Table 9.50, provide simple, multiuser-safe methods for obtaining successive sequence values from sequence objects.

Table 9.50. Sequence Functions

Function

Description

nextval ( regclass ) → bigint

Advances the sequence object to its next value and returns that value. This is done atomically: even if multiple sessions execute nextval concurrently, each will safely receive a distinct sequence value. If the sequence object has been created with default parameters, successive nextval calls will return successive values beginning with 1. Other behaviors can be obtained by using appropriate parameters in the CREATE SEQUENCE command.

This function requires USAGE or UPDATE privilege on the sequence.

setval ( regclass, bigint [, boolean ] ) → bigint

Sets the sequence object's current value, and optionally its is_called flag. The two-parameter form sets the sequence's last_value field to the specified value and sets its is_called field to true, meaning that the next nextval will advance the sequence before returning a value. The value that will be reported by currval is also set to the specified value. In the three-parameter form, is_called can be set to either true or false. true has the same effect as the two-parameter form. If it is set to false, the next nextval will return exactly the specified value, and sequence advancement commences with the following nextval. Furthermore, the value reported by currval is not changed in this case. For example,

SELECT setval('myseq', 42);           Next nextval will return 43
SELECT setval('myseq', 42, true);     Same as above
SELECT setval('myseq', 42, false);    Next nextval will return 42

The result returned by setval is just the value of its second argument.

This function requires UPDATE privilege on the sequence.

currval ( regclass ) → bigint

Returns the value most recently obtained by nextval for this sequence in the current session. (An error is reported if nextval has never been called for this sequence in this session.) Because this is returning a session-local value, it gives a predictable answer whether or not other sessions have executed nextval since the current session did.

This function requires USAGE or SELECT privilege on the sequence.

lastval () → bigint

Returns the value most recently returned by nextval in the current session. This function is identical to currval, except that instead of taking the sequence name as an argument it refers to whichever sequence nextval was most recently applied to in the current session. It is an error to call lastval if nextval has not yet been called in the current session.

This function requires USAGE or SELECT privilege on the last used sequence.

Caution

To avoid blocking concurrent transactions that obtain numbers from the same sequence, the value obtained by nextval is not reclaimed for re-use if the calling transaction later aborts. This means that transaction aborts or database crashes can result in gaps in the sequence of assigned values. That can happen without a transaction abort, too. For example an INSERT with an ON CONFLICT clause will compute the to-be-inserted tuple, including doing any required nextval calls, before detecting any conflict that would cause it to follow the ON CONFLICT rule instead. Thus, PostgreSQL sequence objects cannot be used to obtain “gapless” sequences.

Likewise, sequence state changes made by setval are immediately visible to other transactions, and are not undone if the calling transaction rolls back.

If the database cluster crashes before committing a transaction containing a nextval or setval call, the sequence state change might not have made its way to persistent storage, so that it is uncertain whether the sequence will have its original or updated state after the cluster restarts. This is harmless for usage of the sequence within the database, since other effects of uncommitted transactions will not be visible either. However, if you wish to use a sequence value for persistent outside-the-database purposes, make sure that the nextval call has been committed before doing so.

The sequence to be operated on by a sequence function is specified by a regclass argument, which is simply the OID of the sequence in the pg_class system catalog. You do not have to look up the OID by hand, however, since the regclass data type's input converter will do the work for you. See Section 8.19 for details.

Prev | Up | Next
9.16. JSON Functions and Operators | Home | 9.18. Conditional Expressions


9.18. Conditional Expressions
Prev Up Chapter 9. Functions and Operators Home Next

9.18. Conditional Expressions

9.18.1. CASE
9.18.2. COALESCE
9.18.3. NULLIF
9.18.4. GREATEST and LEAST

This section describes the SQL-compliant conditional expressions available in PostgreSQL.

Tip

If your needs go beyond the capabilities of these conditional expressions, you might want to consider writing a server-side function in a more expressive programming language.

Note

Although COALESCE, GREATEST, and LEAST are syntactically similar to functions, they are not ordinary functions, and thus cannot be used with explicit VARIADIC array arguments.

9.18.1. CASE

The SQL CASE expression is a generic conditional expression, similar to if/else statements in other programming languages:

CASE WHEN condition THEN result
     [WHEN ...]
     [ELSE result]
END

CASE clauses can be used wherever an expression is valid. Each condition is an expression that returns a boolean result. If the condition's result is true, the value of the CASE expression is the result that follows the condition, and the remainder of the CASE expression is not processed. If the condition's result is not true, any subsequent WHEN clauses are examined in the same manner. If no WHEN condition yields true, the value of the CASE expression is the result of the ELSE clause. If the ELSE clause is omitted and no condition is true, the result is null.

An example:

SELECT * FROM test;

 a
---
 1
 2
 3

SELECT a,
       CASE WHEN a=1 THEN 'one'
            WHEN a=2 THEN 'two'
            ELSE 'other'
       END
    FROM test;

 a | case
---+-------
 1 | one
 2 | two
 3 | other

The data types of all the result expressions must be convertible to a single output type. See Section 10.5 for more details.

There is a “simple” form of CASE expression that is a variant of the general form above:

CASE expression
    WHEN value THEN result
    [WHEN ...]
    [ELSE result]
END

The first expression is computed, then compared to each of the value expressions in the WHEN clauses until one is found that is equal to it. If no match is found, the result of the ELSE clause (or a null value) is returned. This is similar to the switch statement in C.

The example above can be written using the simple CASE syntax:

SELECT a,
       CASE a WHEN 1 THEN 'one'
              WHEN 2 THEN 'two'
              ELSE 'other'
       END
    FROM test;

 a | case
---+-------
 1 | one
 2 | two
 3 | other

A CASE expression does not evaluate any subexpressions that are not needed to determine the result. For example, this is a possible way of avoiding a division-by-zero failure:

SELECT ... WHERE CASE WHEN x <> 0 THEN y/x > 1.5 ELSE false END;

Note

As described in Section 4.2.14, there are various situations in which subexpressions of an expression are evaluated at different times, so that the principle that “CASE evaluates only necessary subexpressions” is not ironclad. For example a constant 1/0 subexpression will usually result in a division-by-zero failure at planning time, even if it's within a CASE arm that would never be entered at run time.

9.18.2. COALESCE

COALESCE(value [, ...])

The COALESCE function returns the first of its arguments that is not null. Null is returned only if all arguments are null. It is often used to substitute a default value for null values when data is retrieved for display, for example:

SELECT COALESCE(description, short_description, '(none)') ...

This returns description if it is not null, otherwise short_description if it is not null, otherwise (none).

The arguments must all be convertible to a common data type, which will be the type of the result (see Section 10.5 for details).

Like a CASE expression, COALESCE only evaluates the arguments that are needed to determine the result; that is, arguments to the right of the first non-null argument are not evaluated. This SQL-standard function provides capabilities similar to NVL and IFNULL, which are used in some other database systems.

9.18.3. NULLIF

NULLIF(value1, value2)

The NULLIF function returns a null value if value1 equals value2; otherwise it returns value1. This can be used to perform the inverse operation of the COALESCE example given above:

SELECT NULLIF(value, '(none)') ...

In this example, if value is (none), null is returned, otherwise the value of value is returned.

The two arguments must be of comparable types. To be specific, they are compared exactly as if you had written value1 = value2, so there must be a suitable = operator available.

The result has the same type as the first argument — but there is a subtlety. What is actually returned is the first argument of the implied = operator, and in some cases that will have been promoted to match the second argument's type. For example, NULLIF(1, 2.2) yields numeric, because there is no integer = numeric operator, only numeric = numeric.

9.18.4. GREATEST and LEAST

GREATEST(value [, ...])

LEAST(value [, ...])

The GREATEST and LEAST functions select the largest or smallest value from a list of any number of expressions. The expressions must all be convertible to a common data type, which will be the type of the result (see Section 10.5 for details). NULL values in the list are ignored. The result will be NULL only if all the expressions evaluate to NULL.

Note that GREATEST and LEAST are not in the SQL standard, but are a common extension. Some other databases make them return NULL if any argument is NULL, rather than only when all are NULL.

Prev | Up | Next
9.17. Sequence Manipulation Functions | Home | 9.19. Array Functions and Operators


9.19. Array Functions and Operators
Prev Up Chapter 9. Functions and Operators Home Next

9.19. Array Functions and Operators

Table 9.51 shows the specialized operators available for array types. In addition to those, the usual comparison operators shown in Table 9.1 are available for arrays. The comparison operators compare the array contents element-by-element, using the default B-tree comparison function for the element data type, and sort based on the first difference. In multidimensional arrays the elements are visited in row-major order (last subscript varies most rapidly). If the contents of two arrays are equal but the dimensionality is different, the first difference in the dimensionality information determines the sort order.

Table 9.51. Array Operators

Operator

Description

Example(s)

anyarray @> anyarray → boolean

Does the first array contain the second, that is, does each element appearing in the second array equal some element of the first array? (Duplicates are not treated specially, thus ARRAY[1] and ARRAY[1,1] are each considered to contain the other.)

ARRAY[1,4,3] @> ARRAY[3,1,3] → t

anyarray <@ anyarray → boolean

Is the first array contained by the second?

ARRAY[2,2,7] <@ ARRAY[1,7,4,2,6] → t

anyarray && anyarray → boolean

Do the arrays overlap, that is, have any elements in common?

ARRAY[1,4,3] && ARRAY[2,1] → t

anycompatiblearray || anycompatiblearray → anycompatiblearray

Concatenates the two arrays. Concatenating a null or empty array is a no-op; otherwise the arrays must have the same number of dimensions (as illustrated by the first example) or differ in number of dimensions by one (as illustrated by the second). If the arrays are not of identical element types, they will be coerced to a common type (see Section 10.5).

ARRAY[1,2,3] || ARRAY[4,5,6,7] → {1,2,3,4,5,6,7}

ARRAY[1,2,3] || ARRAY[[4,5,6],[7,8,9.9]] → {{1,2,3},{4,5,6},{7,8,9.9}}

anycompatible || anycompatiblearray → anycompatiblearray

Concatenates an element onto the front of an array (which must be empty or one-dimensional).

3 || ARRAY[4,5,6] → {3,4,5,6}

anycompatiblearray || anycompatible → anycompatiblearray

Concatenates an element onto the end of an array (which must be empty or one-dimensional).

ARRAY[4,5,6] || 7 → {4,5,6,7}

See Section 8.15 for more details about array operator behavior. See Section 11.2 for more details about which operators support indexed operations.

Table 9.52 shows the functions available for use with array types. See Section 8.15 for more information and examples of the use of these functions.

Table 9.52. Array Functions

Function

Description

Example(s)

array_append ( anycompatiblearray, anycompatible ) → anycompatiblearray

Appends an element to the end of an array (same as the anycompatiblearray || anycompatible operator).

array_append(ARRAY[1,2], 3) → {1,2,3}

array_cat ( anycompatiblearray, anycompatiblearray ) → anycompatiblearray

Concatenates two arrays (same as the anycompatiblearray || anycompatiblearray operator).

array_cat(ARRAY[1,2,3], ARRAY[4,5]) → {1,2,3,4,5}

array_dims ( anyarray ) → text

Returns a text representation of the array's dimensions.

array_dims(ARRAY[[1,2,3], [4,5,6]]) → [1:2][1:3]

array_fill ( anyelement, integer[] [, integer[] ] ) → anyarray

Returns an array filled with copies of the given value, having dimensions of the lengths specified by the second argument. The optional third argument supplies lower-bound values for each dimension (which default to all 1).

array_fill(11, ARRAY[2,3]) → {{11,11,11},{11,11,11}}

array_fill(7, ARRAY[3], ARRAY[2]) → [2:4]={7,7,7}

array_length ( anyarray, integer ) → integer

Returns the length of the requested array dimension.

array_length(array[1,2,3], 1) → 3

array_lower ( anyarray, integer ) → integer

Returns the lower bound of the requested array dimension.

array_lower('[0:2]={1,2,3}'::integer[], 1) → 0

array_ndims ( anyarray ) → integer

Returns the number of dimensions of the array.

array_ndims(ARRAY[[1,2,3], [4,5,6]]) → 2

array_position ( anycompatiblearray, anycompatible [, integer ] ) → integer

Returns the subscript of the first occurrence of the second argument in the array, or NULL if it's not present. If the third argument is given, the search begins at that subscript. The array must be one-dimensional. Comparisons are done using IS NOT DISTINCT FROM semantics, so it is possible to search for NULL.

array_position(ARRAY['sun', 'mon', 'tue', 'wed', 'thu', 'fri', 'sat'], 'mon') → 2

array_positions ( anycompatiblearray, anycompatible ) → integer[]

Returns an array of the subscripts of all occurrences of the second argument in the array given as first argument. The array must be one-dimensional. Comparisons are done using IS NOT DISTINCT FROM semantics, so it is possible to search for NULL. NULL is returned only if the array is NULL; if the value is not found in the array, an empty array is returned.

array_positions(ARRAY['A','A','B','A'], 'A') → {1,2,4}

array_prepend ( anycompatible, anycompatiblearray ) → anycompatiblearray

Prepends an element to the beginning of an array (same as the anycompatible || anycompatiblearray operator).

array_prepend(1, ARRAY[2,3]) → {1,2,3}

array_remove ( anycompatiblearray, anycompatible ) → anycompatiblearray

Removes all elements equal to the given value from the array. The array must be one-dimensional. Comparisons are done using IS NOT DISTINCT FROM semantics, so it is possible to remove NULLs.

array_remove(ARRAY[1,2,3,2], 2) → {1,3}

array_replace ( anycompatiblearray, anycompatible, anycompatible ) → anycompatiblearray

Replaces each array element equal to the second argument with the third argument.

array_replace(ARRAY[1,2,5,4], 5, 3) → {1,2,3,4}

array_to_string ( array anyarray, delimiter text [, null_string text ] ) → text

Converts each array element to its text representation, and concatenates those separated by the delimiter string. If null_string is given and is not NULL, then NULL array entries are represented by that string; otherwise, they are omitted.

array_to_string(ARRAY[1, 2, 3, NULL, 5], ',', '*') → 1,2,3,*,5

array_upper ( anyarray, integer ) → integer

Returns the upper bound of the requested array dimension.

array_upper(ARRAY[1,8,3,7], 1) → 4

cardinality ( anyarray ) → integer

Returns the total number of elements in the array, or 0 if the array is empty.

cardinality(ARRAY[[1,2],[3,4]]) → 4

trim_array ( array anyarray, n integer ) → anyarray

Trims an array by removing the last n elements. If the array is multidimensional, only the first dimension is trimmed.

trim_array(ARRAY[1,2,3,4,5,6], 2) → {1,2,3,4}

unnest ( anyarray ) → setof anyelement

Expands an array into a set of rows. The array's elements are read out in storage order.

unnest(ARRAY[1,2]) →

 1
 2

unnest(ARRAY[['foo','bar'],['baz','quux']]) →

 foo
 bar
 baz
 quux

unnest ( anyarray, anyarray [, ... ] ) → setof anyelement, anyelement [, ... ]

Expands multiple arrays (possibly of different data types) into a set of rows. If the arrays are not all the same length then the shorter ones are padded with NULLs. This form is only allowed in a query's FROM clause; see Section 7.2.1.4.

select * from unnest(ARRAY[1,2], ARRAY['foo','bar','baz']) as x(a,b) →

 a |  b
---+-----
 1 | foo
 2 | bar
   | baz

Note

There are two differences in the behavior of string_to_array from pre-9.1 versions of PostgreSQL. First, it will return an empty (zero-element) array rather than NULL when the input string is of zero length. Second, if the delimiter string is NULL, the function splits the input into individual characters, rather than returning NULL as before.

See also Section 9.21 about the aggregate function array_agg for use with arrays.

Prev | Up | Next
9.18. Conditional Expressions | Home | 9.20. Range/Multirange Functions and Operators


9.20. Range/Multirange Functions and Operators
Prev Up Chapter 9. Functions and Operators Home Next

9.20. Range/Multirange Functions and Operators

See Section 8.17 for an overview of range types.

Table 9.53 shows the specialized operators available for range types. Table 9.54 shows the specialized operators available for multirange types. In addition to those, the usual comparison operators shown in Table 9.1 are available for range and multirange types. The comparison operators order first by the range lower bounds, and only if those are equal do they compare the upper bounds. The multirange operators compare each range until one is unequal. This does not usually result in a useful overall ordering, but the operators are provided to allow unique indexes to be constructed on ranges.

Table 9.53. Range Operators

Operator

Description

Example(s)

anyrange @> anyrange → boolean

Does the first range contain the second?

int4range(2,4) @> int4range(2,3) → t

anyrange @> anyelement → boolean

Does the range contain the element?

'[2011-01-01,2011-03-01)'::tsrange @> '2011-01-10'::timestamp → t

anyrange <@ anyrange → boolean

Is the first range contained by the second?

int4range(2,4) <@ int4range(1,7) → t

anyelement <@ anyrange → boolean

Is the element contained in the range?

42 <@ int4range(1,7) → f

anyrange && anyrange → boolean

Do the ranges overlap, that is, have any elements in common?

int8range(3,7) && int8range(4,12) → t

anyrange << anyrange → boolean

Is the first range strictly left of the second?

int8range(1,10) << int8range(100,110) → t

anyrange >> anyrange → boolean

Is the first range strictly right of the second?

int8range(50,60) >> int8range(20,30) → t

anyrange &< anyrange → boolean

Does the first range not extend to the right of the second?

int8range(1,20) &< int8range(18,20) → t

anyrange &> anyrange → boolean

Does the first range not extend to the left of the second?

int8range(7,20) &> int8range(5,10) → t

anyrange -|- anyrange → boolean

Are the ranges adjacent?

numrange(1.1,2.2) -|- numrange(2.2,3.3) → t

anyrange + anyrange → anyrange

Computes the union of the ranges. The ranges must overlap or be adjacent, so that the union is a single range (but see range_merge()).

numrange(5,15) + numrange(10,20) → [5,20)

anyrange * anyrange → anyrange

Computes the intersection of the ranges.

int8range(5,15) * int8range(10,20) → [10,15)

anyrange - anyrange → anyrange

Computes the difference of the ranges. The second range must not be contained in the first in such a way that the difference would not be a single range.

int8range(5,15) - int8range(10,20) → [5,10)

Table 9.54. Multirange Operators

Operator

Description

Example(s)

anymultirange @> anymultirange → boolean

Does the first multirange contain the second?

'{[2,4)}'::int4multirange @> '{[2,3)}'::int4multirange → t

anymultirange @> anyrange → boolean

Does the multirange contain the range?

'{[2,4)}'::int4multirange @> int4range(2,3) → t

anymultirange @> anyelement → boolean

Does the multirange contain the element?

'{[2011-01-01,2011-03-01)}'::tsmultirange @> '2011-01-10'::timestamp → t

anyrange @> anymultirange → boolean

Does the range contain the multirange?

'[2,4)'::int4range @> '{[2,3)}'::int4multirange → t

anymultirange <@ anymultirange → boolean

Is the first multirange contained by the second?

'{[2,4)}'::int4multirange <@ '{[1,7)}'::int4multirange → t

anymultirange <@ anyrange → boolean

Is the multirange contained by the range?

'{[2,4)}'::int4multirange <@ int4range(1,7) → t

anyrange <@ anymultirange → boolean

Is the range contained by the multirange?

int4range(2,4) <@ '{[1,7)}'::int4multirange → t

anyelement <@ anymultirange → boolean

Is the element contained by the multirange?

4 <@ '{[1,7)}'::int4multirange → t

anymultirange && anymultirange → boolean

Do the multiranges overlap, that is, have any elements in common?

'{[3,7)}'::int8multirange && '{[4,12)}'::int8multirange → t

anymultirange && anyrange → boolean

Does the multirange overlap the range?

'{[3,7)}'::int8multirange && int8range(4,12) → t

anyrange && anymultirange → boolean

Does the range overlap the multirange?

int8range(3,7) && '{[4,12)}'::int8multirange → t

anymultirange << anymultirange → boolean

Is the first multirange strictly left of the second?

'{[1,10)}'::int8multirange << '{[100,110)}'::int8multirange → t

anymultirange << anyrange → boolean

Is the multirange strictly left of the range?

'{[1,10)}'::int8multirange << int8range(100,110) → t

anyrange << anymultirange → boolean

Is the range strictly left of the multirange?

int8range(1,10) << '{[100,110)}'::int8multirange → t

anymultirange >> anymultirange → boolean

Is the first multirange strictly right of the second?

'{[50,60)}'::int8multirange >> '{[20,30)}'::int8multirange → t

anymultirange >> anyrange → boolean

Is the multirange strictly right of the range?

'{[50,60)}'::int8multirange >> int8range(20,30) → t

anyrange >> anymultirange → boolean

Is the range strictly right of the multirange?

int8range(50,60) >> '{[20,30)}'::int8multirange → t

anymultirange &< anymultirange → boolean

Does the first multirange not extend to the right of the second?

'{[1,20)}'::int8multirange &< '{[18,20)}'::int8multirange → t

anymultirange &< anyrange → boolean

Does the multirange not extend to the right of the range?

'{[1,20)}'::int8multirange &< int8range(18,20) → t

anyrange &< anymultirange → boolean

Does the range not extend to the right of the multirange?

int8range(1,20) &< '{[18,20)}'::int8multirange → t

anymultirange &> anymultirange → boolean

Does the first multirange not extend to the left of the second?

'{[7,20)}'::int8multirange &> '{[5,10)}'::int8multirange → t

anymultirange &> anyrange → boolean

Does the multirange not extend to the left of the range?

'{[7,20)}'::int8multirange &> int8range(5,10) → t

anyrange &> anymultirange → boolean

Does the range not extend to the left of the multirange?

int8range(7,20) &> '{[5,10)}'::int8multirange → t

anymultirange -|- anymultirange → boolean

Are the multiranges adjacent?

'{[1.1,2.2)}'::nummultirange -|- '{[2.2,3.3)}'::nummultirange → t

anymultirange -|- anyrange → boolean

Is the multirange adjacent to the range?

'{[1.1,2.2)}'::nummultirange -|- numrange(2.2,3.3) → t

anyrange -|- anymultirange → boolean

Is the range adjacent to the multirange?

numrange(1.1,2.2) -|- '{[2.2,3.3)}'::nummultirange → t

anymultirange + anymultirange → anymultirange

Computes the union of the multiranges. The multiranges need not overlap or be adjacent.

'{[5,10)}'::nummultirange + '{[15,20)}'::nummultirange → {[5,10), [15,20)}

anymultirange * anymultirange → anymultirange

Computes the intersection of the multiranges.

'{[5,15)}'::int8multirange * '{[10,20)}'::int8multirange → {[10,15)}

anymultirange - anymultirange → anymultirange

Computes the difference of the multiranges.

'{[5,20)}'::int8multirange - '{[10,15)}'::int8multirange → {[5,10), [15,20)}

The left-of/right-of/adjacent operators always return false when an empty range or multirange is involved; that is, an empty range is not considered to be either before or after any other range.

Elsewhere empty ranges and multiranges are treated as the additive identity: anything unioned with an empty value is itself. Anything minus an empty value is itself. An empty multirange has exactly the same points as an empty range. Every range contains the empty range. Every multirange contains as many empty ranges as you like.

The range union and difference operators will fail if the resulting range would need to contain two disjoint sub-ranges, as such a range cannot be represented. There are separate operators for union and difference that take multirange parameters and return a multirange, and they do not fail even if their arguments are disjoint. So if you need a union or difference operation for ranges that may be disjoint, you can avoid errors by first casting your ranges to multiranges.

Table 9.55 shows the functions available for use with range types. Table 9.56 shows the functions available for use with multirange types.

Table 9.55. Range Functions

Function

Description

Example(s)

lower ( anyrange ) → anyelement

Extracts the lower bound of the range (NULL if the range is empty or the lower bound is infinite).

lower(numrange(1.1,2.2)) → 1.1

upper ( anyrange ) → anyelement

Extracts the upper bound of the range (NULL if the range is empty or the upper bound is infinite).

upper(numrange(1.1,2.2)) → 2.2

isempty ( anyrange ) → boolean

Is the range empty?

isempty(numrange(1.1,2.2)) → f

lower_inc ( anyrange ) → boolean

Is the range's lower bound inclusive?

lower_inc(numrange(1.1,2.2)) → t

upper_inc ( anyrange ) → boolean

Is the range's upper bound inclusive?

upper_inc(numrange(1.1,2.2)) → f

lower_inf ( anyrange ) → boolean

Is the range's lower bound infinite?

lower_inf('(,)'::daterange) → t

upper_inf ( anyrange ) → boolean

Is the range's upper bound infinite?

upper_inf('(,)'::daterange) → t

range_merge ( anyrange, anyrange ) → anyrange

Computes the smallest range that includes both of the given ranges.

range_merge('[1,2)'::int4range, '[3,4)'::int4range) → [1,4)

Table 9.56. Multirange Functions

Function

Description

Example(s)

lower ( anymultirange ) → anyelement

Extracts the lower bound of the multirange (NULL if the multirange is empty or the lower bound is infinite).

lower('{[1.1,2.2)}'::nummultirange) → 1.1

upper ( anymultirange ) → anyelement

Extracts the upper bound of the multirange (NULL if the multirange is empty or the upper bound is infinite).

upper('{[1.1,2.2)}'::nummultirange) → 2.2

isempty ( anymultirange ) → boolean

Is the multirange empty?

isempty('{[1.1,2.2)}'::nummultirange) → f

lower_inc ( anymultirange ) → boolean

Is the multirange's lower bound inclusive?

lower_inc('{[1.1,2.2)}'::nummultirange) → t

upper_inc ( anymultirange ) → boolean

Is the multirange's upper bound inclusive?

upper_inc('{[1.1,2.2)}'::nummultirange) → f

lower_inf ( anymultirange ) → boolean

Is the multirange's lower bound infinite?

lower_inf('{(,)}'::datemultirange) → t

upper_inf ( anymultirange ) → boolean

Is the multirange's upper bound infinite?

upper_inf('{(,)}'::datemultirange) → t

range_merge ( anymultirange ) → anyrange

Computes the smallest range that includes the entire multirange.

range_merge('{[1,2), [3,4)}'::int4multirange) → [1,4)

multirange ( anyrange ) → anymultirange

Returns a multirange containing just the given range.

multirange('[1,2)'::int4range) → {[1,2)}

unnest ( anymultirange ) → setof anyrange

Expands a multirange into a set of ranges. The ranges are read out in storage order (ascending).

unnest('{[1,2), [3,4)}'::int4multirange) →

 [1,2)
 [3,4)

The lower_inc, upper_inc, lower_inf, and upper_inf functions all return false for an empty range or multirange.

Prev | Up | Next
9.19. Array Functions and Operators | Home | 9.21. Aggregate Functions


9.21. Aggregate Functions
Prev Up Chapter 9. Functions and Operators Home Next

9.21. Aggregate Functions

Aggregate functions compute a single result from a set of input values. The built-in general-purpose aggregate functions are listed in Table 9.57 while statistical aggregates are in Table 9.58. The built-in within-group ordered-set aggregate functions are listed in Table 9.59 while the built-in within-group hypothetical-set ones are in Table 9.60. Grouping operations, which are closely related to aggregate functions, are listed in Table 9.61. The special syntax considerations for aggregate functions are explained in Section 4.2.7. Consult Section 2.7 for additional introductory information.

Aggregate functions that support Partial Mode are eligible to participate in various optimizations, such as parallel aggregation.

Table 9.57. General-Purpose Aggregate Functions

Function Description | Partial Mode
array_agg ( anynonarray ) → anyarray Collects all the input values, including nulls, into an array. | No
array_agg ( anyarray ) → anyarray Concatenates all the input arrays into an array of one higher dimension. (The inputs must all have the same dimensionality, and cannot be empty or null.) | No
avg ( smallint ) → numeric avg ( integer ) → numeric avg ( bigint ) → numeric avg ( numeric ) → numeric avg ( real ) → double precision avg ( double precision ) → double precision avg ( interval ) → interval Computes the average (arithmetic mean) of all the non-null input values. | Yes
bit_and ( smallint ) → smallint bit_and ( integer ) → integer bit_and ( bigint ) → bigint bit_and ( bit ) → bit Computes the bitwise AND of all non-null input values. | Yes
bit_or ( smallint ) → smallint bit_or ( integer ) → integer bit_or ( bigint ) → bigint bit_or ( bit ) → bit Computes the bitwise OR of all non-null input values. | Yes
bit_xor ( smallint ) → smallint bit_xor ( integer ) → integer bit_xor ( bigint ) → bigint bit_xor ( bit ) → bit Computes the bitwise exclusive OR of all non-null input values. Can be useful as a checksum for an unordered set of values. | Yes
bool_and ( boolean ) → boolean Returns true if all non-null input values are true, otherwise false. | Yes
bool_or ( boolean ) → boolean Returns true if any non-null input value is true, otherwise false. | Yes
count ( * ) → bigint Computes the number of input rows. | Yes
count ( "any" ) → bigint Computes the number of input rows in which the input value is not null. | Yes
every ( boolean ) → boolean This is the SQL standard's equivalent to bool_and. | Yes
json_agg ( anyelement ) → json jsonb_agg ( anyelement ) → jsonb Collects all the input values, including nulls, into a JSON array. Values are converted to JSON as per to_json or to_jsonb. | No
json_object_agg ( key "any", value "any" ) → json jsonb_object_agg ( key "any", value "any" ) → jsonb Collects all the key/value pairs into a JSON object. Key arguments are coerced to text; value arguments are converted as per to_json or to_jsonb. Values can be null, but not keys. | No
max ( see text ) → same as input type Computes the maximum of the non-null input values. Available for any numeric, string, date/time, or enum type, as well as inet, interval, money, oid, pg_lsn, tid, and arrays of any of these types. | Yes
min ( see text ) → same as input type Computes the minimum of the non-null input values. Available for any numeric, string, date/time, or enum type, as well as inet, interval, money, oid, pg_lsn, tid, and arrays of any of these types. | Yes
range_agg ( value anyrange ) → anymultirange Computes the union of the non-null input values. | No
range_intersect_agg ( value anyrange ) → anyrange range_intersect_agg ( value anymultirange ) → anymultirange Computes the intersection of the non-null input values. | No
string_agg ( value text, delimiter text ) → text string_agg ( value bytea, delimiter bytea ) → bytea Concatenates the non-null input values into a string. Each value after the first is preceded by the corresponding delimiter (if it's not null). | No
sum ( smallint ) → bigint sum ( integer ) → bigint sum ( bigint ) → numeric sum ( numeric ) → numeric sum ( real ) → real sum ( double precision ) → double precision sum ( interval ) → interval sum ( money ) → money Computes the sum of the non-null input values. | Yes
xmlagg ( xml ) → xml Concatenates the non-null XML input values (see Section 9.15.1.7). | No

It should be noted that except for count, these functions return a null value when no rows are selected. In particular, sum of no rows returns null, not zero as one might expect, and array_agg returns null rather than an empty array when there are no input rows. The coalesce function can be used to substitute zero or an empty array for null when necessary.

The aggregate functions array_agg, json_agg, jsonb_agg, json_object_agg, jsonb_object_agg, string_agg, and xmlagg, as well as similar user-defined aggregate functions, produce meaningfully different result values depending on the order of the input values. This ordering is unspecified by default, but can be controlled by writing an ORDER BY clause within the aggregate call, as shown in Section 4.2.7. Alternatively, supplying the input values from a sorted subquery will usually work. For example:

SELECT xmlagg(x) FROM (SELECT x FROM test ORDER BY y DESC) AS tab;

Beware that this approach can fail if the outer query level contains additional processing, such as a join, because that might cause the subquery's output to be reordered before the aggregate is computed.

Note

The boolean aggregates bool_and and bool_or correspond to the standard SQL aggregates every and any or some. PostgreSQL supports every, but not any or some, because there is an ambiguity built into the standard syntax:

SELECT b1 = ANY((SELECT b2 FROM t2 ...)) FROM t1 ...;

Here ANY can be considered either as introducing a subquery, or as being an aggregate function, if the subquery returns one row with a Boolean value. Thus the standard name cannot be given to these aggregates.

Note

Users accustomed to working with other SQL database management systems might be disappointed by the performance of the count aggregate when it is applied to the entire table. A query like:

SELECT count(*) FROM sometable;

will require effort proportional to the size of the table: PostgreSQL will need to scan either the entire table or the entirety of an index that includes all rows in the table.

Table 9.58 shows aggregate functions typically used in statistical analysis. (These are separated out merely to avoid cluttering the listing of more-commonly-used aggregates.) Functions shown as accepting numeric_type are available for all the types smallint, integer, bigint, numeric, real, and double precision. Where the description mentions N, it means the number of input rows for which all the input expressions are non-null. In all cases, null is returned if the computation is meaningless, for example when N is zero.

Table 9.58. Aggregate Functions for Statistics

Function Description | Partial Mode
corr ( Y double precision, X double precision ) → double precision Computes the correlation coefficient. | Yes
covar_pop ( Y double precision, X double precision ) → double precision Computes the population covariance. | Yes
covar_samp ( Y double precision, X double precision ) → double precision Computes the sample covariance. | Yes
regr_avgx ( Y double precision, X double precision ) → double precision Computes the average of the independent variable, sum(X)/N. | Yes
regr_avgy ( Y double precision, X double precision ) → double precision Computes the average of the dependent variable, sum(Y)/N. | Yes
regr_count ( Y double precision, X double precision ) → bigint Computes the number of rows in which both inputs are non-null. | Yes
regr_intercept ( Y double precision, X double precision ) → double precision Computes the y-intercept of the least-squares-fit linear equation determined by the (X, Y) pairs. | Yes
regr_r2 ( Y double precision, X double precision ) → double precision Computes the square of the correlation coefficient. | Yes
regr_slope ( Y double precision, X double precision ) → double precision Computes the slope of the least-squares-fit linear equation determined by the (X, Y) pairs. | Yes
regr_sxx ( Y double precision, X double precision ) → double precision Computes the “sum of squares” of the independent variable, sum(X^2) - sum(X)^2/N. | Yes
regr_sxy ( Y double precision, X double precision ) → double precision Computes the “sum of products” of independent times dependent variables, sum(X*Y) - sum(X) * sum(Y)/N. | Yes
regr_syy ( Y double precision, X double precision ) → double precision Computes the “sum of squares” of the dependent variable, sum(Y^2) - sum(Y)^2/N. | Yes
stddev ( numeric_type ) → double precision for real or double precision, otherwise numeric This is a historical alias for stddev_samp. | Yes
stddev_pop ( numeric_type ) → double precision for real or double precision, otherwise numeric Computes the population standard deviation of the input values. | Yes
stddev_samp ( numeric_type ) → double precision for real or double precision, otherwise numeric Computes the sample standard deviation of the input values. | Yes
variance ( numeric_type ) → double precision for real or double precision, otherwise numeric This is a historical alias for var_samp. | Yes
var_pop ( numeric_type ) → double precision for real or double precision, otherwise numeric Computes the population variance of the input values (square of the population standard deviation). | Yes
var_samp ( numeric_type ) → double precision for real or double precision, otherwise numeric Computes the sample variance of the input values (square of the sample standard deviation). | Yes

Table 9.59 shows some aggregate functions that use the ordered-set aggregate syntax. These functions are sometimes referred to as “inverse distribution” functions. Their aggregated input is introduced by ORDER BY, and they may also take a direct argument that is not aggregated, but is computed only once. All these functions ignore null values in their aggregated input. For those that take a fraction parameter, the fraction value must be between 0 and 1; an error is thrown if not. However, a null fraction value simply produces a null result.

Table 9.59. Ordered-Set Aggregate Functions

Function

Description

Partial Mode

mode () WITHIN GROUP ( ORDER BY anyelement ) → anyelement

Computes the mode, the most frequent value of the aggregated argument (arbitrarily choosing the first one if there are multiple equally-frequent values). The aggregated argument must be of a sortable type.

No

percentile_cont ( fraction double precision ) WITHIN GROUP ( ORDER BY double precision ) → double precision

percentile_cont ( fraction double precision ) WITHIN GROUP ( ORDER BY interval ) → interval

Computes the continuous percentile, a value corresponding to the specified fraction within the ordered set of aggregated argument values. This will interpolate between adjacent input items if needed.

No

percentile_cont ( fractions double precision[] ) WITHIN GROUP ( ORDER BY double precision ) → double precision[]

percentile_cont ( fractions double precision[] ) WITHIN GROUP ( ORDER BY interval ) → interval[]

Computes multiple continuous percentiles. The result is an array of the same dimensions as the fractions parameter, with each non-null element replaced by the (possibly interpolated) value corresponding to that percentile.

No

percentile_disc ( fraction double precision ) WITHIN GROUP ( ORDER BY anyelement ) → anyelement

Computes the discrete percentile, the first value within the ordered set of aggregated argument values whose position in the ordering equals or exceeds the specified fraction. The aggregated argument must be of a sortable type.

No

percentile_disc ( fractions double precision[] ) WITHIN GROUP ( ORDER BY anyelement ) → anyarray

Computes multiple discrete percentiles. The result is an array of the same dimensions as the fractions parameter, with each non-null element replaced by the input value corresponding to that percentile. The aggregated argument must be of a sortable type.

No

Each of the “hypothetical-set” aggregates listed in Table 9.60 is associated with a window function of the same name defined in Section 9.22. In each case, the aggregate's result is the value that the associated window function would have returned for the “hypothetical” row constructed from args, if such a row had been added to the sorted group of rows represented by the sorted_args. For each of these functions, the list of direct arguments given in args must match the number and types of the aggregated arguments given in sorted_args. Unlike most built-in aggregates, these aggregates are not strict, that is they do not drop input rows containing nulls. Null values sort according to the rule specified in the ORDER BY clause.

Table 9.60. Hypothetical-Set Aggregate Functions

Function Description | Partial Mode
rank ( args ) WITHIN GROUP ( ORDER BY sorted_args ) → bigint Computes the rank of the hypothetical row, with gaps; that is, the row number of the first row in its peer group. | No
dense_rank ( args ) WITHIN GROUP ( ORDER BY sorted_args ) → bigint Computes the rank of the hypothetical row, without gaps; this function effectively counts peer groups. | No
percent_rank ( args ) WITHIN GROUP ( ORDER BY sorted_args ) → double precision Computes the relative rank of the hypothetical row, that is (rank - 1) / (total rows - 1). The value thus ranges from 0 to 1 inclusive. | No
cume_dist ( args ) WITHIN GROUP ( ORDER BY sorted_args ) → double precision Computes the cumulative distribution, that is (number of rows preceding or peers with hypothetical row) / (total rows). The value thus ranges from 1/N to 1. | No

Table 9.61. Grouping Operations

Function

Description

GROUPING ( group_by_expression(s) ) → integer

Returns a bit mask indicating which GROUP BY expressions are not included in the current grouping set. Bits are assigned with the rightmost argument corresponding to the least-significant bit; each bit is 0 if the corresponding expression is included in the grouping criteria of the grouping set generating the current result row, and 1 if it is not included.

The grouping operations shown in Table 9.61 are used in conjunction with grouping sets (see Section 7.2.4) to distinguish result rows. The arguments to the GROUPING function are not actually evaluated, but they must exactly match expressions given in the GROUP BY clause of the associated query level. For example:

=> SELECT * FROM items_sold;
 make  | model | sales
-------+-------+-------
 Foo   | GT    |  10
 Foo   | Tour  |  20
 Bar   | City  |  15
 Bar   | Sport |  5
(4 rows)

=> SELECT make, model, GROUPING(make,model), sum(sales) FROM items_sold GROUP BY ROLLUP(make,model);
 make  | model | grouping | sum
-------+-------+----------+-----
 Foo   | GT    |        0 | 10
 Foo   | Tour  |        0 | 20
 Bar   | City  |        0 | 15
 Bar   | Sport |        0 | 5
 Foo   |       |        1 | 30
 Bar   |       |        1 | 20
       |       |        3 | 50
(7 rows)

Here, the grouping value 0 in the first four rows shows that those have been grouped normally, over both the grouping columns. The value 1 indicates that model was not grouped by in the next-to-last two rows, and the value 3 indicates that neither make nor model was grouped by in the last row (which therefore is an aggregate over all the input rows).

Prev | Up | Next
9.20. Range/Multirange Functions and Operators | Home | 9.22. Window Functions


9.22. Window Functions
Prev Up Chapter 9. Functions and Operators Home Next

9.22. Window Functions

Window functions provide the ability to perform calculations across sets of rows that are related to the current query row. See Section 3.5 for an introduction to this feature, and Section 4.2.8 for syntax details.

The built-in window functions are listed in Table 9.62. Note that these functions must be invoked using window function syntax, i.e., an OVER clause is required.

In addition to these functions, any built-in or user-defined ordinary aggregate (i.e., not ordered-set or hypothetical-set aggregates) can be used as a window function; see Section 9.21 for a list of the built-in aggregates. Aggregate functions act as window functions only when an OVER clause follows the call; otherwise they act as plain aggregates and return a single row for the entire set.

Table 9.62. General-Purpose Window Functions

Function

Description

row_number () → bigint

Returns the number of the current row within its partition, counting from 1.

rank () → bigint

Returns the rank of the current row, with gaps; that is, the row_number of the first row in its peer group.

dense_rank () → bigint

Returns the rank of the current row, without gaps; this function effectively counts peer groups.

percent_rank () → double precision

Returns the relative rank of the current row, that is (rank - 1) / (total partition rows - 1). The value thus ranges from 0 to 1 inclusive.

cume_dist () → double precision

Returns the cumulative distribution, that is (number of partition rows preceding or peers with current row) / (total partition rows). The value thus ranges from 1/N to 1.

ntile ( num_buckets integer ) → integer

Returns an integer ranging from 1 to the argument value, dividing the partition as equally as possible.

lag ( value anycompatible [, offset integer [, default anycompatible ]] ) → anycompatible

Returns value evaluated at the row that is offset rows before the current row within the partition; if there is no such row, instead returns default (which must be of a type compatible with value). Both offset and default are evaluated with respect to the current row. If omitted, offset defaults to 1 and default to NULL.

lead ( value anycompatible [, offset integer [, default anycompatible ]] ) → anycompatible

Returns value evaluated at the row that is offset rows after the current row within the partition; if there is no such row, instead returns default (which must be of a type compatible with value). Both offset and default are evaluated with respect to the current row. If omitted, offset defaults to 1 and default to NULL.

first_value ( value anyelement ) → anyelement

Returns value evaluated at the row that is the first row of the window frame.

last_value ( value anyelement ) → anyelement

Returns value evaluated at the row that is the last row of the window frame.

nth_value ( value anyelement, n integer ) → anyelement

Returns value evaluated at the row that is the n'th row of the window frame (counting from 1); returns NULL if there is no such row.

All of the functions listed in Table 9.62 depend on the sort ordering specified by the ORDER BY clause of the associated window definition. Rows that are not distinct when considering only the ORDER BY columns are said to be peers. The four ranking functions (including cume_dist) are defined so that they give the same answer for all rows of a peer group.

Note that first_value, last_value, and nth_value consider only the rows within the “window frame”, which by default contains the rows from the start of the partition through the last peer of the current row. This is likely to give unhelpful results for last_value and sometimes also nth_value. You can redefine the frame by adding a suitable frame specification (RANGE, ROWS or GROUPS) to the OVER clause. See Section 4.2.8 for more information about frame specifications.

When an aggregate function is used as a window function, it aggregates over the rows within the current row's window frame. An aggregate used with ORDER BY and the default window frame definition produces a “running sum” type of behavior, which may or may not be what's wanted. To obtain aggregation over the whole partition, omit ORDER BY or use ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING. Other frame specifications can be used to obtain other effects.

Note

The SQL standard defines a RESPECT NULLS or IGNORE NULLS option for lead, lag, first_value, last_value, and nth_value. This is not implemented in PostgreSQL: the behavior is always the same as the standard's default, namely RESPECT NULLS. Likewise, the standard's FROM FIRST or FROM LAST option for nth_value is not implemented: only the default FROM FIRST behavior is supported. (You can achieve the result of FROM LAST by reversing the ORDER BY ordering.)

Prev | Up | Next
9.21. Aggregate Functions | Home | 9.23. Subquery Expressions


9.23. Subquery Expressions
Prev Up Chapter 9. Functions and Operators Home Next

9.23. Subquery Expressions

9.23.1. EXISTS
9.23.2. IN
9.23.3. NOT IN
9.23.4. ANY/SOME
9.23.5. ALL
9.23.6. Single-Row Comparison

This section describes the SQL-compliant subquery expressions available in PostgreSQL. All of the expression forms documented in this section return Boolean (true/false) results.

9.23.1. EXISTS

EXISTS (subquery)

The argument of EXISTS is an arbitrary SELECT statement, or subquery. The subquery is evaluated to determine whether it returns any rows. If it returns at least one row, the result of EXISTS is “true”; if the subquery returns no rows, the result of EXISTS is “false”.

The subquery can refer to variables from the surrounding query, which will act as constants during any one evaluation of the subquery.

The subquery will generally only be executed long enough to determine whether at least one row is returned, not all the way to completion. It is unwise to write a subquery that has side effects (such as calling sequence functions); whether the side effects occur might be unpredictable.

Since the result depends only on whether any rows are returned, and not on the contents of those rows, the output list of the subquery is normally unimportant. A common coding convention is to write all EXISTS tests in the form EXISTS(SELECT 1 WHERE ...). There are exceptions to this rule however, such as subqueries that use INTERSECT.

This simple example is like an inner join on col2, but it produces at most one output row for each tab1 row, even if there are several matching tab2 rows:

SELECT col1
FROM tab1
WHERE EXISTS (SELECT 1 FROM tab2 WHERE col2 = tab1.col2);

9.23.2. IN

expression IN (subquery)

The right-hand side is a parenthesized subquery, which must return exactly one column. The left-hand expression is evaluated and compared to each row of the subquery result. The result of IN is “true” if any equal subquery row is found. The result is “false” if no equal row is found (including the case where the subquery returns no rows).

Note that if the left-hand expression yields null, or if there are no equal right-hand values and at least one right-hand row yields null, the result of the IN construct will be null, not false. This is in accordance with SQL's normal rules for Boolean combinations of null values.

As with EXISTS, it's unwise to assume that the subquery will be evaluated completely.

row_constructor IN (subquery)

The left-hand side of this form of IN is a row constructor, as described in Section 4.2.13. The right-hand side is a parenthesized subquery, which must return exactly as many columns as there are expressions in the left-hand row. The left-hand expressions are evaluated and compared row-wise to each row of the subquery result. The result of IN is “true” if any equal subquery row is found. The result is “false” if no equal row is found (including the case where the subquery returns no rows).

As usual, null values in the rows are combined per the normal rules of SQL Boolean expressions. Two rows are considered equal if all their corresponding members are non-null and equal; the rows are unequal if any corresponding members are non-null and unequal; otherwise the result of that row comparison is unknown (null). If all the per-row results are either unequal or null, with at least one null, then the result of IN is null.

9.23.3. NOT IN

expression NOT IN (subquery)

The right-hand side is a parenthesized subquery, which must return exactly one column. The left-hand expression is evaluated and compared to each row of the subquery result. The result of NOT IN is “true” if only unequal subquery rows are found (including the case where the subquery returns no rows). The result is “false” if any equal row is found.

Note that if the left-hand expression yields null, or if there are no equal right-hand values and at least one right-hand row yields null, the result of the NOT IN construct will be null, not true. This is in accordance with SQL's normal rules for Boolean combinations of null values.

As with EXISTS, it's unwise to assume that the subquery will be evaluated completely.

row_constructor NOT IN (subquery)

The left-hand side of this form of NOT IN is a row constructor, as described in Section 4.2.13. The right-hand side is a parenthesized subquery, which must return exactly as many columns as there are expressions in the left-hand row. The left-hand expressions are evaluated and compared row-wise to each row of the subquery result. The result of NOT IN is “true” if only unequal subquery rows are found (including the case where the subquery returns no rows). The result is “false” if any equal row is found.

As usual, null values in the rows are combined per the normal rules of SQL Boolean expressions. Two rows are considered equal if all their corresponding members are non-null and equal; the rows are unequal if any corresponding members are non-null and unequal; otherwise the result of that row comparison is unknown (null). If all the per-row results are either unequal or null, with at least one null, then the result of NOT IN is null.

9.23.4. ANY/SOME

expression operator ANY (subquery)
expression operator SOME (subquery)

The right-hand side is a parenthesized subquery, which must return exactly one column. The left-hand expression is evaluated and compared to each row of the subquery result using the given operator, which must yield a Boolean result. The result of ANY is “true” if any true result is obtained. The result is “false” if no true result is found (including the case where the subquery returns no rows).

SOME is a synonym for ANY. IN is equivalent to = ANY.

Note that if there are no successes and at least one right-hand row yields null for the operator's result, the result of the ANY construct will be null, not false. This is in accordance with SQL's normal rules for Boolean combinations of null values.

As with EXISTS, it's unwise to assume that the subquery will be evaluated completely.

row_constructor operator ANY (subquery)
row_constructor operator SOME (subquery)

The left-hand side of this form of ANY is a row constructor, as described in Section 4.2.13. The right-hand side is a parenthesized subquery, which must return exactly as many columns as there are expressions in the left-hand row. The left-hand expressions are evaluated and compared row-wise to each row of the subquery result, using the given operator. The result of ANY is “true” if the comparison returns true for any subquery row. The result is “false” if the comparison returns false for every subquery row (including the case where the subquery returns no rows). The result is NULL if no comparison with a subquery row returns true, and at least one comparison returns NULL.

See Section 9.24.5 for details about the meaning of a row constructor comparison.

9.23.5. ALL

expression operator ALL (subquery)

The right-hand side is a parenthesized subquery, which must return exactly one column. The left-hand expression is evaluated and compared to each row of the subquery result using the given operator, which must yield a Boolean result. The result of ALL is “true” if all rows yield true (including the case where the subquery returns no rows). The result is “false” if any false result is found. The result is NULL if no comparison with a subquery row returns false, and at least one comparison returns NULL.

NOT IN is equivalent to <> ALL.

As with EXISTS, it's unwise to assume that the subquery will be evaluated completely.

row_constructor operator ALL (subquery)

The left-hand side of this form of ALL is a row constructor, as described in Section 4.2.13. The right-hand side is a parenthesized subquery, which must return exactly as many columns as there are expressions in the left-hand row. The left-hand expressions are evaluated and compared row-wise to each row of the subquery result, using the given operator. The result of ALL is “true” if the comparison returns true for all subquery rows (including the case where the subquery returns no rows). The result is “false” if the comparison returns false for any subquery row. The result is NULL if no comparison with a subquery row returns false, and at least one comparison returns NULL.

See Section 9.24.5 for details about the meaning of a row constructor comparison.

9.23.6. Single-Row Comparison

row_constructor operator (subquery)

The left-hand side is a row constructor, as described in Section 4.2.13. The right-hand side is a parenthesized subquery, which must return exactly as many columns as there are expressions in the left-hand row. Furthermore, the subquery cannot return more than one row. (If it returns zero rows, the result is taken to be null.) The left-hand side is evaluated and compared row-wise to the single subquery result row.

See Section 9.24.5 for details about the meaning of a row constructor comparison.

Prev | Up | Next
9.22. Window Functions | Home | 9.24. Row and Array Comparisons


9.24. Row and Array Comparisons
Prev Up Chapter 9. Functions and Operators Home Next

9.24. Row and Array Comparisons

9.24.1. IN
9.24.2. NOT IN
9.24.3. ANY/SOME (array)
9.24.4. ALL (array)
9.24.5. Row Constructor Comparison
9.24.6. Composite Type Comparison

This section describes several specialized constructs for making multiple comparisons between groups of values. These forms are syntactically related to the subquery forms of the previous section, but do not involve subqueries. The forms involving array subexpressions are PostgreSQL extensions; the rest are SQL-compliant. All of the expression forms documented in this section return Boolean (true/false) results.

9.24.1. IN

expression IN (value [, ...])

The right-hand side is a parenthesized list of scalar expressions. The result is “true” if the left-hand expression's result is equal to any of the right-hand expressions. This is a shorthand notation for

expression = value1
OR
expression = value2
OR
...

Note that if the left-hand expression yields null, or if there are no equal right-hand values and at least one right-hand expression yields null, the result of the IN construct will be null, not false. This is in accordance with SQL's normal rules for Boolean combinations of null values.

9.24.2. NOT IN

expression NOT IN (value [, ...])

The right-hand side is a parenthesized list of scalar expressions. The result is “true” if the left-hand expression's result is unequal to all of the right-hand expressions. This is a shorthand notation for

expression <> value1
AND
expression <> value2
AND
...

Note that if the left-hand expression yields null, or if there are no equal right-hand values and at least one right-hand expression yields null, the result of the NOT IN construct will be null, not true as one might naively expect. This is in accordance with SQL's normal rules for Boolean combinations of null values.

Tip

x NOT IN y is equivalent to NOT (x IN y) in all cases. However, null values are much more likely to trip up the novice when working with NOT IN than when working with IN. It is best to express your condition positively if possible.

9.24.3. ANY/SOME (array)

expression operator ANY (array expression)
expression operator SOME (array expression)

The right-hand side is a parenthesized expression, which must yield an array value. The left-hand expression is evaluated and compared to each element of the array using the given operator, which must yield a Boolean result. The result of ANY is “true” if any true result is obtained. The result is “false” if no true result is found (including the case where the array has zero elements).

If the array expression yields a null array, the result of ANY will be null. If the left-hand expression yields null, the result of ANY is ordinarily null (though a non-strict comparison operator could possibly yield a different result). Also, if the right-hand array contains any null elements and no true comparison result is obtained, the result of ANY will be null, not false (again, assuming a strict comparison operator). This is in accordance with SQL's normal rules for Boolean combinations of null values.

SOME is a synonym for ANY.

9.24.4. ALL (array)

expression operator ALL (array expression)

The right-hand side is a parenthesized expression, which must yield an array value. The left-hand expression is evaluated and compared to each element of the array using the given operator, which must yield a Boolean result. The result of ALL is “true” if all comparisons yield true (including the case where the array has zero elements). The result is “false” if any false result is found.

If the array expression yields a null array, the result of ALL will be null. If the left-hand expression yields null, the result of ALL is ordinarily null (though a non-strict comparison operator could possibly yield a different result). Also, if the right-hand array contains any null elements and no false comparison result is obtained, the result of ALL will be null, not true (again, assuming a strict comparison operator). This is in accordance with SQL's normal rules for Boolean combinations of null values.

9.24.5. Row Constructor Comparison

row_constructor operator row_constructor

Each side is a row constructor, as described in Section 4.2.13. The two row values must have the same number of fields. Each side is evaluated and they are compared row-wise. Row constructor comparisons are allowed when the operator is =, <>, <, <=, > or >=. Every row element must be of a type which has a default B-tree operator class or the attempted comparison may generate an error.

Note

Errors related to the number or types of elements might not occur if the comparison is resolved using earlier columns.

The = and <> cases work slightly differently from the others. Two rows are considered equal if all their corresponding members are non-null and equal; the rows are unequal if any corresponding members are non-null and unequal; otherwise the result of the row comparison is unknown (null).

For the <, <=, > and >= cases, the row elements are compared left-to-right, stopping as soon as an unequal or null pair of elements is found. If either of this pair of elements is null, the result of the row comparison is unknown (null); otherwise comparison of this pair of elements determines the result. For example, ROW(1,2,NULL) < ROW(1,3,0) yields true, not null, because the third pair of elements are not considered.

Note

Prior to PostgreSQL 8.2, the <, <=, > and >= cases were not handled per SQL specification. A comparison like ROW(a,b) < ROW(c,d) was implemented as a < c AND b < d whereas the correct behavior is equivalent to a < c OR (a = c AND b < d).

row_constructor IS DISTINCT FROM row_constructor

This construct is similar to a <> row comparison, but it does not yield null for null inputs. Instead, any null value is considered unequal to (distinct from) any non-null value, and any two nulls are considered equal (not distinct). Thus the result will either be true or false, never null.

row_constructor IS NOT DISTINCT FROM row_constructor

This construct is similar to a = row comparison, but it does not yield null for null inputs. Instead, any null value is considered unequal to (distinct from) any non-null value, and any two nulls are considered equal (not distinct). Thus the result will always be either true or false, never null.

9.24.6. Composite Type Comparison

record operator record

The SQL specification requires row-wise comparison to return NULL if the result depends on comparing two NULL values or a NULL and a non-NULL. PostgreSQL does this only when comparing the results of two row constructors (as in Section 9.24.5) or comparing a row constructor to the output of a subquery (as in Section 9.23). In other contexts where two composite-type values are compared, two NULL field values are considered equal, and a NULL is considered larger than a non-NULL. This is necessary in order to have consistent sorting and indexing behavior for composite types.

Each side is evaluated and they are compared row-wise. Composite type comparisons are allowed when the operator is =, <>, <, <=, > or >=, or has semantics similar to one of these. (To be specific, an operator can be a row comparison operator if it is a member of a B-tree operator class, or is the negator of the = member of a B-tree operator class.) The default behavior of the above operators is the same as for IS [ NOT ] DISTINCT FROM for row constructors (see Section 9.24.5).

To support matching of rows which include elements without a default B-tree operator class, the following operators are defined for composite type comparison: *=, *<>, *<, *<=, *>, and *>=. These operators compare the internal binary representation of the two rows. Two rows might have a different binary representation even though comparisons of the two rows with the equality operator is true. The ordering of rows under these comparison operators is deterministic but not otherwise meaningful. These operators are used internally for materialized views and might be useful for other specialized purposes such as replication and B-Tree deduplication (see Section 64.4.3). They are not intended to be generally useful for writing queries, though.

Prev | Up | Next
9.23. Subquery Expressions | Home | 9.25. Set Returning Functions


9.25. Set Returning Functions
Prev Up Chapter 9. Functions and Operators Home Next

9.25. Set Returning Functions

This section describes functions that possibly return more than one row. The most widely used functions in this class are series generating functions, as detailed in Table 9.63 and Table 9.64. Other, more specialized set-returning functions are described elsewhere in this manual. See Section 7.2.1.4 for ways to combine multiple set-returning functions.

Table 9.63. Series Generating Functions

Function

Description

generate_series ( start integer, stop integer [, step integer ] ) → setof integer

generate_series ( start bigint, stop bigint [, step bigint ] ) → setof bigint

generate_series ( start numeric, stop numeric [, step numeric ] ) → setof numeric

Generates a series of values from start to stop, with a step size of step. step defaults to 1.

generate_series ( start timestamp, stop timestamp, step interval ) → setof timestamp

generate_series ( start timestamp with time zone, stop timestamp with time zone, step interval ) → setof timestamp with time zone

Generates a series of values from start to stop, with a step size of step.

When step is positive, zero rows are returned if start is greater than stop. Conversely, when step is negative, zero rows are returned if start is less than stop. Zero rows are also returned if any input is NULL. It is an error for step to be zero. Some examples follow:

SELECT * FROM generate_series(2,4);
 generate_series
-----------------
               2
               3
               4
(3 rows)

SELECT * FROM generate_series(5,1,-2);
 generate_series
-----------------
               5
               3
               1
(3 rows)

SELECT * FROM generate_series(4,3);
 generate_series
-----------------
(0 rows)

SELECT generate_series(1.1, 4, 1.3);
 generate_series
-----------------
             1.1
             2.4
             3.7
(3 rows)

-- this example relies on the date-plus-integer operator:
SELECT current_date + s.a AS dates FROM generate_series(0,14,7) AS s(a);
   dates
------------
 2004-02-05
 2004-02-12
 2004-02-19
(3 rows)

SELECT * FROM generate_series('2008-03-01 00:00'::timestamp,
                              '2008-03-04 12:00', '10 hours');
   generate_series
---------------------
 2008-03-01 00:00:00
 2008-03-01 10:00:00
 2008-03-01 20:00:00
 2008-03-02 06:00:00
 2008-03-02 16:00:00
 2008-03-03 02:00:00
 2008-03-03 12:00:00
 2008-03-03 22:00:00
 2008-03-04 08:00:00
(9 rows)

Table 9.64. Subscript Generating Functions

Function

Description

generate_subscripts ( array anyarray, dim integer ) → setof integer

Generates a series comprising the valid subscripts of the dim'th dimension of the given array.

generate_subscripts ( array anyarray, dim integer, reverse boolean ) → setof integer

Generates a series comprising the valid subscripts of the dim'th dimension of the given array. When reverse is true, returns the series in reverse order.

generate_subscripts is a convenience function that generates the set of valid subscripts for the specified dimension of the given array. Zero rows are returned for arrays that do not have the requested dimension, or if any input is NULL. Some examples follow:

-- basic usage:
SELECT generate_subscripts('{NULL,1,NULL,2}'::int[], 1) AS s;
 s
---
 1
 2
 3
 4
(4 rows)

-- presenting an array, the subscript and the subscripted
-- value requires a subquery:
SELECT * FROM arrays;
         a
--------------------
 {-1,-2}
 {100,200,300}
(2 rows)

SELECT a AS array, s AS subscript, a[s] AS value
FROM (SELECT generate_subscripts(a, 1) AS s, a FROM arrays) foo;
     array     | subscript | value
---------------+-----------+-------
 {-1,-2}       |         1 |    -1
 {-1,-2}       |         2 |    -2
 {100,200,300} |         1 |   100
 {100,200,300} |         2 |   200
 {100,200,300} |         3 |   300
(5 rows)

-- unnest a 2D array:
CREATE OR REPLACE FUNCTION unnest2(anyarray)
RETURNS SETOF anyelement AS $$
select $1[i][j]
   from generate_subscripts($1,1) g1(i),
        generate_subscripts($1,2) g2(j);
$$ LANGUAGE sql IMMUTABLE;
CREATE FUNCTION
SELECT * FROM unnest2(ARRAY[[1,2],[3,4]]);
 unnest2
---------
       1
       2
       3
       4
(4 rows)

When a function in the FROM clause is suffixed by WITH ORDINALITY, a bigint column is appended to the function's output column(s), which starts from 1 and increments by 1 for each row of the function's output. This is most useful in the case of set returning functions such as unnest().

-- set returning function WITH ORDINALITY:
SELECT * FROM pg_ls_dir('.') WITH ORDINALITY AS t(ls,n);
       ls        | n
-----------------+----
 pg_serial       |  1
 pg_twophase     |  2
 postmaster.opts |  3
 pg_notify       |  4
 postgresql.conf |  5
 pg_tblspc       |  6
 logfile         |  7
 base            |  8
 postmaster.pid  |  9
 pg_ident.conf   | 10
 global          | 11
 pg_xact         | 12
 pg_snapshots    | 13
 pg_multixact    | 14
 PG_VERSION      | 15
 pg_wal          | 16
 pg_hba.conf     | 17
 pg_stat_tmp     | 18
 pg_subtrans     | 19
(19 rows)

Prev | Up | Next
9.24. Row and Array Comparisons | Home | 9.26. System Information Functions and Operators


9.26. System Information Functions and Operators
Prev Up Chapter 9. Functions and Operators Home Next

9.26. System Information Functions and Operators

Table 9.65 shows several functions that extract session and system information.

In addition to the functions listed in this section, there are a number of functions related to the statistics system that also provide system information. See Section 28.2.2 for more information.

Table 9.65. Session Information Functions

Function

Description

current_catalog → name

current_database () → name

Returns the name of the current database. (Databases are called “catalogs” in the SQL standard, so current_catalog is the standard's spelling.)

current_query () → text

Returns the text of the currently executing query, as submitted by the client (which might contain more than one statement).

current_role → name

This is equivalent to current_user.

current_schema → name

current_schema () → name

Returns the name of the schema that is first in the search path (or a null value if the search path is empty). This is the schema that will be used for any tables or other named objects that are created without specifying a target schema.

current_schemas ( include_implicit boolean ) → name[]

Returns an array of the names of all schemas presently in the effective search path, in their priority order. (Items in the current search_path setting that do not correspond to existing, searchable schemas are omitted.) If the Boolean argument is true, then implicitly-searched system schemas such as pg_catalog are included in the result.

current_user → name

Returns the user name of the current execution context.

inet_client_addr () → inet

Returns the IP address of the current client, or NULL if the current connection is via a Unix-domain socket.

inet_client_port () → integer

Returns the IP port number of the current client, or NULL if the current connection is via a Unix-domain socket.

inet_server_addr () → inet

Returns the IP address on which the server accepted the current connection, or NULL if the current connection is via a Unix-domain socket.

inet_server_port () → integer

Returns the IP port number on which the server accepted the current connection, or NULL if the current connection is via a Unix-domain socket.

pg_backend_pid () → integer

Returns the process ID of the server process attached to the current session.

pg_blocking_pids ( integer ) → integer[]

Returns an array of the process ID(s) of the sessions that are blocking the server process with the specified process ID from acquiring a lock, or an empty array if there is no such server process or it is not blocked.

One server process blocks another if it either holds a lock that conflicts with the blocked process's lock request (hard block), or is waiting for a lock that would conflict with the blocked process's lock request and is ahead of it in the wait queue (soft block). When using parallel queries the result always lists client-visible process IDs (that is, pg_backend_pid results) even if the actual lock is held or awaited by a child worker process. As a result of that, there may be duplicated PIDs in the result. Also note that when a prepared transaction holds a conflicting lock, it will be represented by a zero process ID.

Frequent calls to this function could have some impact on database performance, because it needs exclusive access to the lock manager's shared state for a short time.

pg_conf_load_time () → timestamp with time zone

Returns the time when the server configuration files were last loaded. If the current session was alive at the time, this will be the time when the session itself re-read the configuration files (so the reading will vary a little in different sessions). Otherwise it is the time when the postmaster process re-read the configuration files.

pg_current_logfile ( [ text ] ) → text

Returns the path name of the log file currently in use by the logging collector. The path includes the log_directory directory and the individual log file name. The result is NULL if the logging collector is disabled. When multiple log files exist, each in a different format, pg_current_logfile without an argument returns the path of the file having the first format found in the ordered list: stderr, csvlog. NULL is returned if no log file has any of these formats. To request information about a specific log file format, supply either csvlog or stderr as the value of the optional parameter. The result is NULL if the log format requested is not configured in log_destination. The result reflects the contents of the current_logfiles file.

pg_my_temp_schema () → oid

Returns the OID of the current session's temporary schema, or zero if it has none (because it has not created any temporary tables).

pg_is_other_temp_schema ( oid ) → boolean

Returns true if the given OID is the OID of another session's temporary schema. (This can be useful, for example, to exclude other sessions' temporary tables from a catalog display.)

pg_jit_available () → boolean

Returns true if a JIT compiler extension is available (see Chapter 32) and the jit configuration parameter is set to on.

pg_listening_channels () → setof text

Returns the set of names of asynchronous notification channels that the current session is listening to.

pg_notification_queue_usage () → double precision

Returns the fraction (0–1) of the asynchronous notification queue's maximum size that is currently occupied by notifications that are waiting to be processed. See LISTEN and NOTIFY for more information.

pg_postmaster_start_time () → timestamp with time zone

Returns the time when the server started.

pg_safe_snapshot_blocking_pids ( integer ) → integer[]

Returns an array of the process ID(s) of the sessions that are blocking the server process with the specified process ID from acquiring a safe snapshot, or an empty array if there is no such server process or it is not blocked.

A session running a SERIALIZABLE transaction blocks a SERIALIZABLE READ ONLY DEFERRABLE transaction from acquiring a snapshot until the latter determines that it is safe to avoid taking any predicate locks. See Section 13.2.3 for more information about serializable and deferrable transactions.

Frequent calls to this function could have some impact on database performance, because it needs access to the predicate lock manager's shared state for a short time.

pg_trigger_depth () → integer

Returns the current nesting level of PostgreSQL triggers (0 if not called, directly or indirectly, from inside a trigger).

session_user → name

Returns the session user's name.

user → name

This is equivalent to current_user.

version () → text

Returns a string describing the PostgreSQL server's version. You can also get this information from server_version, or for a machine-readable version use server_version_num. Software developers should use server_version_num (available since 8.2) or PQserverVersion instead of parsing the text version.

Note

current_catalog, current_role, current_schema, current_user, session_user, and user have special syntactic status in SQL: they must be called without trailing parentheses. In PostgreSQL, parentheses can optionally be used with current_schema, but not with the others.

The session_user is normally the user who initiated the current database connection; but superusers can change this setting with SET SESSION AUTHORIZATION. The current_user is the user identifier that is applicable for permission checking. Normally it is equal to the session user, but it can be changed with SET ROLE. It also changes during the execution of functions with the attribute SECURITY DEFINER. In Unix parlance, the session user is the “real user” and the current user is the “effective user”. current_role and user are synonyms for current_user. (The SQL standard draws a distinction between current_role and current_user, but PostgreSQL does not, since it unifies users and roles into a single kind of entity.)

Table 9.66 lists functions that allow querying object access privileges programmatically. (See Section 5.7 for more information about privileges.) In these functions, the user whose privileges are being inquired about can be specified by name or by OID (pg_authid.oid), or if the name is given as public then the privileges of the PUBLIC pseudo-role are checked. Also, the user argument can be omitted entirely, in which case the current_user is assumed. The object that is being inquired about can be specified either by name or by OID, too. When specifying by name, a schema name can be included if relevant. The access privilege of interest is specified by a text string, which must evaluate to one of the appropriate privilege keywords for the object's type (e.g., SELECT). Optionally, WITH GRANT OPTION can be added to a privilege type to test whether the privilege is held with grant option. Also, multiple privilege types can be listed separated by commas, in which case the result will be true if any of the listed privileges is held. (Case of the privilege string is not significant, and extra whitespace is allowed between but not within privilege names.) Some examples:

SELECT has_table_privilege('myschema.mytable', 'select');
SELECT has_table_privilege('joe', 'mytable', 'INSERT, SELECT WITH GRANT OPTION');

Table 9.66. Access Privilege Inquiry Functions

Function

Description

has_any_column_privilege ( [ user name or oid, ] table text or oid, privilege text ) → boolean

Does user have privilege for any column of table? This succeeds either if the privilege is held for the whole table, or if there is a column-level grant of the privilege for at least one column. Allowable privilege types are SELECT, INSERT, UPDATE, and REFERENCES.

has_column_privilege ( [ user name or oid, ] table text or oid, column text or smallint, privilege text ) → boolean

Does user have privilege for the specified table column? This succeeds either if the privilege is held for the whole table, or if there is a column-level grant of the privilege for the column. The column can be specified by name or by attribute number (pg_attribute.attnum). Allowable privilege types are SELECT, INSERT, UPDATE, and REFERENCES.

has_database_privilege ( [ user name or oid, ] database text or oid, privilege text ) → boolean

Does user have privilege for database? Allowable privilege types are CREATE, CONNECT, TEMPORARY, and TEMP (which is equivalent to TEMPORARY).

has_foreign_data_wrapper_privilege ( [ user name or oid, ] fdw text or oid, privilege text ) → boolean

Does user have privilege for foreign-data wrapper? The only allowable privilege type is USAGE.

has_function_privilege ( [ user name or oid, ] function text or oid, privilege text ) → boolean

Does user have privilege for function? The only allowable privilege type is EXECUTE.

When specifying a function by name rather than by OID, the allowed input is the same as for the regprocedure data type (see Section 8.19). An example is:

SELECT has_function_privilege('joeuser', 'myfunc(int, text)', 'execute');

has_language_privilege ( [ user name or oid, ] language text or oid, privilege text ) → boolean

Does user have privilege for language? The only allowable privilege type is USAGE.

has_schema_privilege ( [ user name or oid, ] schema text or oid, privilege text ) → boolean

Does user have privilege for schema? Allowable privilege types are CREATE and USAGE.

has_sequence_privilege ( [ user name or oid, ] sequence text or oid, privilege text ) → boolean

Does user have privilege for sequence? Allowable privilege types are USAGE, SELECT, and UPDATE.

has_server_privilege ( [ user name or oid, ] server text or oid, privilege text ) → boolean

Does user have privilege for foreign server? The only allowable privilege type is USAGE.

has_table_privilege ( [ user name or oid, ] table text or oid, privilege text ) → boolean

Does user have privilege for table? Allowable privilege types are SELECT, INSERT, UPDATE, DELETE, TRUNCATE, REFERENCES, and TRIGGER.

has_tablespace_privilege ( [ user name or oid, ] tablespace text or oid, privilege text ) → boolean

Does user have privilege for tablespace? The only allowable privilege type is CREATE.

has_type_privilege ( [ user name or oid, ] type text or oid, privilege text ) → boolean

Does user have privilege for data type? The only allowable privilege type is USAGE. When specifying a type by name rather than by OID, the allowed input is the same as for the regtype data type (see Section 8.19).

pg_has_role ( [ user name or oid, ] role text or oid, privilege text ) → boolean

Does user have privilege for role? Allowable privilege types are MEMBER and USAGE. MEMBER denotes direct or indirect membership in the role (that is, the right to do SET ROLE), while USAGE denotes whether the privileges of the role are immediately available without doing SET ROLE. This function does not allow the special case of setting user to public, because the PUBLIC pseudo-role can never be a member of real roles.

row_security_active ( table text or oid ) → boolean

Is row-level security active for the specified table in the context of the current user and current environment?

Table 9.67 shows the operators available for the aclitem type, which is the catalog representation of access privileges. See Section 5.7 for information about how to read access privilege values.

Table 9.67. aclitem Operators

Operator

Description

Example(s)

aclitem = aclitem → boolean

Are aclitems equal? (Notice that type aclitem lacks the usual set of comparison operators; it has only equality. In turn, aclitem arrays can only be compared for equality.)

'calvin=r*w/hobbes'::aclitem = 'calvin=r*w*/hobbes'::aclitem → f

aclitem[] @> aclitem → boolean

Does array contain the specified privileges? (This is true if there is an array entry that matches the aclitem's grantee and grantor, and has at least the specified set of privileges.)

'{calvin=r*w/hobbes,hobbes=r*w*/postgres}'::aclitem[] @> 'calvin=r*/hobbes'::aclitem → t

aclitem[] ~ aclitem → boolean

This is a deprecated alias for @>.

'{calvin=r*w/hobbes,hobbes=r*w*/postgres}'::aclitem[] ~ 'calvin=r*/hobbes'::aclitem → t

Table 9.68 shows some additional functions to manage the aclitem type.

Table 9.68. aclitem Functions

Function

Description

acldefault ( type "char", ownerId oid ) → aclitem[]

Constructs an aclitem array holding the default access privileges for an object of type type belonging to the role with OID ownerId. This represents the access privileges that will be assumed when an object's ACL entry is null. (The default access privileges are described in Section 5.7.) The type parameter must be one of 'c' for COLUMN, 'r' for TABLE and table-like objects, 's' for SEQUENCE, 'd' for DATABASE, 'f' for FUNCTION or PROCEDURE, 'l' for LANGUAGE, 'L' for LARGE OBJECT, 'n' for SCHEMA, 't' for TABLESPACE, 'F' for FOREIGN DATA WRAPPER, 'S' for FOREIGN SERVER, or 'T' for TYPE or DOMAIN.

aclexplode ( aclitem[] ) → setof record ( grantor oid, grantee oid, privilege_type text, is_grantable boolean )

Returns the aclitem array as a set of rows. If the grantee is the pseudo-role PUBLIC, it is represented by zero in the grantee column. Each granted privilege is represented as SELECT, INSERT, etc. Note that each privilege is broken out as a separate row, so only one keyword appears in the privilege_type column.

makeaclitem ( grantee oid, grantor oid, privileges text, is_grantable boolean ) → aclitem

Constructs an aclitem with the given properties.

Table 9.69 shows functions that determine whether a certain object is visible in the current schema search path. For example, a table is said to be visible if its containing schema is in the search path and no table of the same name appears earlier in the search path. This is equivalent to the statement that the table can be referenced by name without explicit schema qualification. Thus, to list the names of all visible tables:

SELECT relname FROM pg_class WHERE pg_table_is_visible(oid);

For functions and operators, an object in the search path is said to be visible if there is no object of the same name and argument data type(s) earlier in the path. For operator classes and families, both the name and the associated index access method are considered.

Table 9.69. Schema Visibility Inquiry Functions

Function

Description

pg_collation_is_visible ( collation oid ) → boolean

Is collation visible in search path?

pg_conversion_is_visible ( conversion oid ) → boolean

Is conversion visible in search path?

pg_function_is_visible ( function oid ) → boolean

Is function visible in search path? (This also works for procedures and aggregates.)

pg_opclass_is_visible ( opclass oid ) → boolean

Is operator class visible in search path?

pg_operator_is_visible ( operator oid ) → boolean

Is operator visible in search path?

pg_opfamily_is_visible ( opclass oid ) → boolean

Is operator family visible in search path?

pg_statistics_obj_is_visible ( stat oid ) → boolean

Is statistics object visible in search path?

pg_table_is_visible ( table oid ) → boolean

Is table visible in search path? (This works for all types of relations, including views, materialized views, indexes, sequences and foreign tables.)

pg_ts_config_is_visible ( config oid ) → boolean

Is text search configuration visible in search path?

pg_ts_dict_is_visible ( dict oid ) → boolean

Is text search dictionary visible in search path?

pg_ts_parser_is_visible ( parser oid ) → boolean

Is text search parser visible in search path?

pg_ts_template_is_visible ( template oid ) → boolean

Is text search template visible in search path?

pg_type_is_visible ( type oid ) → boolean

Is type (or domain) visible in search path?

All these functions require object OIDs to identify the object to be checked. If you want to test an object by name, it is convenient to use the OID alias types (regclass, regtype, regprocedure, regoperator, regconfig, or regdictionary), for example:

SELECT pg_type_is_visible('myschema.widget'::regtype);

Note that it would not make much sense to test a non-schema-qualified type name in this way — if the name can be recognized at all, it must be visible.

Table 9.70 lists functions that extract information from the system catalogs.

Table 9.70. System Catalog Information Functions

Function

Description

format_type ( type oid, typemod integer ) → text

Returns the SQL name for a data type that is identified by its type OID and possibly a type modifier. Pass NULL for the type modifier if no specific modifier is known.

pg_get_catalog_foreign_keys () → setof record ( fktable regclass, fkcols text[], pktable regclass, pkcols text[], is_array boolean, is_opt boolean )

Returns a set of records describing the foreign key relationships that exist within the PostgreSQL system catalogs. The fktable column contains the name of the referencing catalog, and the fkcols column contains the name(s) of the referencing column(s). Similarly, the pktable column contains the name of the referenced catalog, and the pkcols column contains the name(s) of the referenced column(s). If is_array is true, the last referencing column is an array, each of whose elements should match some entry in the referenced catalog. If is_opt is true, the referencing column(s) are allowed to contain zeroes instead of a valid reference.

pg_get_constraintdef ( constraint oid [, pretty boolean ] ) → text

Reconstructs the creating command for a constraint. (This is a decompiled reconstruction, not the original text of the command.)

pg_get_expr ( expr pg_node_tree, relation oid [, pretty boolean ] ) → text

Decompiles the internal form of an expression stored in the system catalogs, such as the default value for a column. If the expression might contain Vars, specify the OID of the relation they refer to as the second parameter; if no Vars are expected, passing zero is sufficient.

pg_get_functiondef ( func oid ) → text

Reconstructs the creating command for a function or procedure. (This is a decompiled reconstruction, not the original text of the command.) The result is a complete CREATE OR REPLACE FUNCTION or CREATE OR REPLACE PROCEDURE statement.

pg_get_function_arguments ( func oid ) → text

Reconstructs the argument list of a function or procedure, in the form it would need to appear in within CREATE FUNCTION (including default values).

pg_get_function_identity_arguments ( func oid ) → text

Reconstructs the argument list necessary to identify a function or procedure, in the form it would need to appear in within commands such as ALTER FUNCTION. This form omits default values.

pg_get_function_result ( func oid ) → text

Reconstructs the RETURNS clause of a function, in the form it would need to appear in within CREATE FUNCTION. Returns NULL for a procedure.

pg_get_indexdef ( index oid [, column integer, pretty boolean ] ) → text

Reconstructs the creating command for an index. (This is a decompiled reconstruction, not the original text of the command.) If column is supplied and is not zero, only the definition of that column is reconstructed.

pg_get_keywords () → setof record ( word text, catcode "char", barelabel boolean, catdesc text, baredesc text )

Returns a set of records describing the SQL keywords recognized by the server. The word column contains the keyword. The catcode column contains a category code: U for an unreserved keyword, C for a keyword that can be a column name, T for a keyword that can be a type or function name, or R for a fully reserved keyword. The barelabel column contains true if the keyword can be used as a “bare” column label in SELECT lists, or false if it can only be used after AS. The catdesc column contains a possibly-localized string describing the keyword's category. The baredesc column contains a possibly-localized string describing the keyword's column label status.

pg_get_ruledef ( rule oid [, pretty boolean ] ) → text

Reconstructs the creating command for a rule. (This is a decompiled reconstruction, not the original text of the command.)

pg_get_serial_sequence ( table text, column text ) → text

Returns the name of the sequence associated with a column, or NULL if no sequence is associated with the column. If the column is an identity column, the associated sequence is the sequence internally created for that column. For columns created using one of the serial types (serial, smallserial, bigserial), it is the sequence created for that serial column definition. In the latter case, the association can be modified or removed with ALTER SEQUENCE OWNED BY. (This function probably should have been called pg_get_owned_sequence; its current name reflects the fact that it has historically been used with serial-type columns.) The first parameter is a table name with optional schema, and the second parameter is a column name. Because the first parameter potentially contains both schema and table names, it is parsed per usual SQL rules, meaning it is lower-cased by default. The second parameter, being just a column name, is treated literally and so has its case preserved. The result is suitably formatted for passing to the sequence functions (see Section 9.17).

A typical use is in reading the current value of the sequence for an identity or serial column, for example:

SELECT currval(pg_get_serial_sequence('sometable', 'id'));

pg_get_statisticsobjdef ( statobj oid ) → text

Reconstructs the creating command for an extended statistics object. (This is a decompiled reconstruction, not the original text of the command.)

pg_get_triggerdef ( trigger oid [, pretty boolean ] ) → text

Reconstructs the creating command for a trigger. (This is a decompiled reconstruction, not the original text of the command.)

pg_get_userbyid ( role oid ) → name

Returns a role's name given its OID.

pg_get_viewdef ( view oid [, pretty boolean ] ) → text

Reconstructs the underlying SELECT command for a view or materialized view. (This is a decompiled reconstruction, not the original text of the command.)

pg_get_viewdef ( view oid, wrap_column integer ) → text

Reconstructs the underlying SELECT command for a view or materialized view. (This is a decompiled reconstruction, not the original text of the command.) In this form of the function, pretty-printing is always enabled, and long lines are wrapped to try to keep them shorter than the specified number of columns.

pg_get_viewdef ( view text [, pretty boolean ] ) → text

Reconstructs the underlying SELECT command for a view or materialized view, working from a textual name for the view rather than its OID. (This is deprecated; use the OID variant instead.)

pg_index_column_has_property ( index regclass, column integer, property text ) → boolean

Tests whether an index column has the named property. Common index column properties are listed in Table 9.71. (Note that extension access methods can define additional property names for their indexes.) NULL is returned if the property name is not known or does not apply to the particular object, or if the OID or column number does not identify a valid object.

pg_index_has_property ( index regclass, property text ) → boolean

Tests whether an index has the named property. Common index properties are listed in Table 9.72. (Note that extension access methods can define additional property names for their indexes.) NULL is returned if the property name is not known or does not apply to the particular object, or if the OID does not identify a valid object.

pg_indexam_has_property ( am oid, property text ) → boolean

Tests whether an index access method has the named property. Access method properties are listed in Table 9.73. NULL is returned if the property name is not known or does not apply to the particular object, or if the OID does not identify a valid object.

pg_options_to_table ( options_array text[] ) → setof record ( option_name text, option_value text )

Returns the set of storage options represented by a value from pg_class.reloptions or pg_attribute.attoptions.

pg_tablespace_databases ( tablespace oid ) → setof oid

Returns the set of OIDs of databases that have objects stored in the specified tablespace. If this function returns any rows, the tablespace is not empty and cannot be dropped. To identify the specific objects populating the tablespace, you will need to connect to the database(s) identified by pg_tablespace_databases and query their pg_class catalogs.

pg_tablespace_location ( tablespace oid ) → text

Returns the file system path that this tablespace is located in.

pg_typeof ( "any" ) → regtype

Returns the OID of the data type of the value that is passed to it. This can be helpful for troubleshooting or dynamically constructing SQL queries. The function is declared as returning regtype, which is an OID alias type (see Section 8.19); this means that it is the same as an OID for comparison purposes but displays as a type name.

For example:

SELECT pg_typeof(33);
 pg_typeof
-----------
 integer

SELECT typlen FROM pg_type WHERE oid = pg_typeof(33);
 typlen
--------
      4

COLLATION FOR ( "any" ) → text

Returns the name of the collation of the value that is passed to it. The value is quoted and schema-qualified if necessary. If no collation was derived for the argument expression, then NULL is returned. If the argument is not of a collatable data type, then an error is raised.

For example:

SELECT collation for (description) FROM pg_description LIMIT 1;
 pg_collation_for
------------------
 "default"

SELECT collation for ('foo' COLLATE "de_DE");
 pg_collation_for
------------------
 "de_DE"

to_regclass ( text ) → regclass

Translates a textual relation name to its OID. A similar result is obtained by casting the string to type regclass (see Section 8.19); however, this function will return NULL rather than throwing an error if the name is not found. Also unlike the cast, this does not accept a numeric OID as input.

to_regcollation ( text ) → regcollation

Translates a textual collation name to its OID. A similar result is obtained by casting the string to type regcollation (see Section 8.19); however, this function will return NULL rather than throwing an error if the name is not found. Also unlike the cast, this does not accept a numeric OID as input.

to_regnamespace ( text ) → regnamespace

Translates a textual schema name to its OID. A similar result is obtained by casting the string to type regnamespace (see Section 8.19); however, this function will return NULL rather than throwing an error if the name is not found. Also unlike the cast, this does not accept a numeric OID as input.

to_regoper ( text ) → regoper

Translates a textual operator name to its OID. A similar result is obtained by casting the string to type regoper (see Section 8.19); however, this function will return NULL rather than throwing an error if the name is not found or is ambiguous. Also unlike the cast, this does not accept a numeric OID as input.

to_regoperator ( text ) → regoperator

Translates a textual operator name (with parameter types) to its OID. A similar result is obtained by casting the string to type regoperator (see Section 8.19); however, this function will return NULL rather than throwing an error if the name is not found. Also unlike the cast, this does not accept a numeric OID as input.

to_regproc ( text ) → regproc

Translates a textual function or procedure name to its OID. A similar result is obtained by casting the string to type regproc (see Section 8.19); however, this function will return NULL rather than throwing an error if the name is not found or is ambiguous. Also unlike the cast, this does not accept a numeric OID as input.

to_regprocedure ( text ) → regprocedure

Translates a textual function or procedure name (with argument types) to its OID. A similar result is obtained by casting the string to type regprocedure (see Section 8.19); however, this function will return NULL rather than throwing an error if the name is not found. Also unlike the cast, this does not accept a numeric OID as input.

to_regrole ( text ) → regrole

Translates a textual role name to its OID. A similar result is obtained by casting the string to type regrole (see Section 8.19); however, this function will return NULL rather than throwing an error if the name is not found. Also unlike the cast, this does not accept a numeric OID as input.

to_regtype ( text ) → regtype

Translates a textual type name to its OID. A similar result is obtained by casting the string to type regtype (see Section 8.19); however, this function will return NULL rather than throwing an error if the name is not found. Also unlike the cast, this does not accept a numeric OID as input.

Most of the functions that reconstruct (decompile) database objects have an optional pretty flag, which if true causes the result to be “pretty-printed”. Pretty-printing suppresses unnecessary parentheses and adds whitespace for legibility. The pretty-printed format is more readable, but the default format is more likely to be interpreted the same way by future versions of PostgreSQL; so avoid using pretty-printed output for dump purposes. Passing false for the pretty parameter yields the same result as omitting the parameter.

Table 9.71. Index Column Properties

Name | Description
asc | Does the column sort in ascending order on a forward scan?
desc | Does the column sort in descending order on a forward scan?
nulls_first | Does the column sort with nulls first on a forward scan?
nulls_last | Does the column sort with nulls last on a forward scan?
orderable | Does the column possess any defined sort ordering?
distance_orderable | Can the column be scanned in order by a “distance” operator, for example ORDER BY col <-> constant ?
returnable | Can the column value be returned by an index-only scan?
search_array | Does the column natively support col = ANY(array) searches?
search_nulls | Does the column support IS NULL and IS NOT NULL searches?

Table 9.72. Index Properties

Name | Description
clusterable | Can the index be used in a CLUSTER command?
index_scan | Does the index support plain (non-bitmap) scans?
bitmap_scan | Does the index support bitmap scans?
backward_scan | Can the scan direction be changed in mid-scan (to support FETCH BACKWARD on a cursor without needing materialization)?

Table 9.73. Index Access Method Properties

Name | Description
can_order | Does the access method support ASC, DESC and related keywords in CREATE INDEX?
can_unique | Does the access method support unique indexes?
can_multi_col | Does the access method support indexes with multiple columns?
can_exclude | Does the access method support exclusion constraints?
can_include | Does the access method support the INCLUDE clause of CREATE INDEX?

Table 9.74 lists functions related to database object identification and addressing.

Table 9.74. Object Information and Addressing Functions

Function

Description

pg_describe_object ( classid oid, objid oid, objsubid integer ) → text

Returns a textual description of a database object identified by catalog OID, object OID, and sub-object ID (such as a column number within a table; the sub-object ID is zero when referring to a whole object). This description is intended to be human-readable, and might be translated, depending on server configuration. This is especially useful to determine the identity of an object referenced in the pg_depend catalog. This function returns NULL values for undefined objects.

pg_identify_object ( classid oid, objid oid, objsubid integer ) → record ( type text, schema text, name text, identity text )

Returns a row containing enough information to uniquely identify the database object specified by catalog OID, object OID and sub-object ID. This information is intended to be machine-readable, and is never translated. type identifies the type of database object; schema is the schema name that the object belongs in, or NULL for object types that do not belong to schemas; name is the name of the object, quoted if necessary, if the name (along with schema name, if pertinent) is sufficient to uniquely identify the object, otherwise NULL; identity is the complete object identity, with the precise format depending on object type, and each name within the format being schema-qualified and quoted as necessary. Undefined objects are identified with NULL values.

pg_identify_object_as_address ( classid oid, objid oid, objsubid integer ) → record ( type text, object_names text[], object_args text[] )

Returns a row containing enough information to uniquely identify the database object specified by catalog OID, object OID and sub-object ID. The returned information is independent of the current server, that is, it could be used to identify an identically named object in another server. type identifies the type of database object; object_names and object_args are text arrays that together form a reference to the object. These three values can be passed to pg_get_object_address to obtain the internal address of the object.

pg_get_object_address ( type text, object_names text[], object_args text[] ) → record ( classid oid, objid oid, objsubid integer )

Returns a row containing enough information to uniquely identify the database object specified by a type code and object name and argument arrays. The returned values are the ones that would be used in system catalogs such as pg_depend; they can be passed to other system functions such as pg_describe_object or pg_identify_object. classid is the OID of the system catalog containing the object; objid is the OID of the object itself, and objsubid is the sub-object ID, or zero if none. This function is the inverse of pg_identify_object_as_address. Undefined objects are identified with NULL values.

The functions shown in Table 9.75 extract comments previously stored with the COMMENT command. A null value is returned if no comment could be found for the specified parameters.

Table 9.75. Comment Information Functions

Function

Description

col_description ( table oid, column integer ) → text

Returns the comment for a table column, which is specified by the OID of its table and its column number. (obj_description cannot be used for table columns, since columns do not have OIDs of their own.)

obj_description ( object oid, catalog name ) → text

Returns the comment for a database object specified by its OID and the name of the containing system catalog. For example, obj_description(123456, 'pg_class') would retrieve the comment for the table with OID 123456.

obj_description ( object oid ) → text

Returns the comment for a database object specified by its OID alone. This is deprecated since there is no guarantee that OIDs are unique across different system catalogs; therefore, the wrong comment might be returned.

shobj_description ( object oid, catalog name ) → text

Returns the comment for a shared database object specified by its OID and the name of the containing system catalog. This is just like obj_description except that it is used for retrieving comments on shared objects (that is, databases, roles, and tablespaces). Some system catalogs are global to all databases within each cluster, and the descriptions for objects in them are stored globally as well.

The functions shown in Table 9.76 provide server transaction information in an exportable form. The main use of these functions is to determine which transactions were committed between two snapshots.

Table 9.76. Transaction ID and Snapshot Information Functions

Function

Description

pg_current_xact_id () → xid8

Returns the current transaction's ID. It will assign a new one if the current transaction does not have one already (because it has not performed any database updates).

pg_current_xact_id_if_assigned () → xid8

Returns the current transaction's ID, or NULL if no ID is assigned yet. (It's best to use this variant if the transaction might otherwise be read-only, to avoid unnecessary consumption of an XID.)

pg_xact_status ( xid8 ) → text

Reports the commit status of a recent transaction. The result is one of in progress, committed, or aborted, provided that the transaction is recent enough that the system retains the commit status of that transaction. If it is old enough that no references to the transaction survive in the system and the commit status information has been discarded, the result is NULL. Applications might use this function, for example, to determine whether their transaction committed or aborted after the application and database server become disconnected while a COMMIT is in progress. Note that prepared transactions are reported as in progress; applications must check pg_prepared_xacts if they need to determine whether a transaction ID belongs to a prepared transaction.

pg_current_snapshot () → pg_snapshot

Returns a current snapshot, a data structure showing which transaction IDs are now in-progress.

pg_snapshot_xip ( pg_snapshot ) → setof xid8

Returns the set of in-progress transaction IDs contained in a snapshot.

pg_snapshot_xmax ( pg_snapshot ) → xid8

Returns the xmax of a snapshot.

pg_snapshot_xmin ( pg_snapshot ) → xid8

Returns the xmin of a snapshot.

pg_visible_in_snapshot ( xid8, pg_snapshot ) → boolean

Is the given transaction ID visible according to this snapshot (that is, was it completed before the snapshot was taken)? Note that this function will not give the correct answer for a subtransaction ID.

The internal transaction ID type xid is 32 bits wide and wraps around every 4 billion transactions. However, the functions shown in Table 9.76 use a 64-bit type xid8 that does not wrap around during the life of an installation, and can be converted to xid by casting if required. The data type pg_snapshot stores information about transaction ID visibility at a particular moment in time. Its components are described in Table 9.77. pg_snapshot's textual representation is xmin:xmax:xip_list. For example 10:20:10,14,15 means xmin=10, xmax=20, xip_list=10, 14, 15.

Table 9.77. Snapshot Components

Name | Description
xmin | Lowest transaction ID that was still active. All transaction IDs less than xmin are either committed and visible, or rolled back and dead.
xmax | One past the highest completed transaction ID. All transaction IDs greater than or equal to xmax had not yet completed as of the time of the snapshot, and thus are invisible.
xip_list | Transactions in progress at the time of the snapshot. A transaction ID that is xmin <= X < xmax and not in this list was already completed at the time of the snapshot, and thus is either visible or dead according to its commit status. This list does not include the transaction IDs of subtransactions.

In releases of PostgreSQL before 13 there was no xid8 type, so variants of these functions were provided that used bigint to represent a 64-bit XID, with a correspondingly distinct snapshot data type txid_snapshot. These older functions have txid in their names. They are still supported for backward compatibility, but may be removed from a future release. See Table 9.78.

Table 9.78. Deprecated Transaction ID and Snapshot Information Functions

Function

Description

txid_current () → bigint

See pg_current_xact_id().

txid_current_if_assigned () → bigint

See pg_current_xact_id_if_assigned().

txid_current_snapshot () → txid_snapshot

See pg_current_snapshot().

txid_snapshot_xip ( txid_snapshot ) → setof bigint

See pg_snapshot_xip().

txid_snapshot_xmax ( txid_snapshot ) → bigint

See pg_snapshot_xmax().

txid_snapshot_xmin ( txid_snapshot ) → bigint

See pg_snapshot_xmin().

txid_visible_in_snapshot ( bigint, txid_snapshot ) → boolean

See pg_visible_in_snapshot().

txid_status ( bigint ) → text

See pg_xact_status().

The functions shown in Table 9.79 provide information about when past transactions were committed. They only provide useful data when the track_commit_timestamp configuration option is enabled, and only for transactions that were committed after it was enabled.

Table 9.79. Committed Transaction Information Functions

Function

Description

pg_xact_commit_timestamp ( xid ) → timestamp with time zone

Returns the commit timestamp of a transaction.

pg_xact_commit_timestamp_origin ( xid ) → record ( timestamp timestamp with time zone, roident oid)

Returns the commit timestamp and replication origin of a transaction.

pg_last_committed_xact () → record ( xid xid, timestamp timestamp with time zone, roident oid )

Returns the transaction ID, commit timestamp and replication origin of the latest committed transaction.

The functions shown in Table 9.80 print information initialized during initdb, such as the catalog version. They also show information about write-ahead logging and checkpoint processing. This information is cluster-wide, not specific to any one database. These functions provide most of the same information, from the same source, as the pg_controldata application.

Table 9.80. Control Data Functions

Function

Description

pg_control_checkpoint () → record

Returns information about current checkpoint state, as shown in Table 9.81.

pg_control_system () → record

Returns information about current control file state, as shown in Table 9.82.

pg_control_init () → record

Returns information about cluster initialization state, as shown in Table 9.83.

pg_control_recovery () → record

Returns information about recovery state, as shown in Table 9.84.

Table 9.81. pg_control_checkpoint Output Columns

Column Name | Data Type
checkpoint_lsn | pg_lsn
redo_lsn | pg_lsn
redo_wal_file | text
timeline_id | integer
prev_timeline_id | integer
full_page_writes | boolean
next_xid | text
next_oid | oid
next_multixact_id | xid
next_multi_offset | xid
oldest_xid | xid
oldest_xid_dbid | oid
oldest_active_xid | xid
oldest_multi_xid | xid
oldest_multi_dbid | oid
oldest_commit_ts_xid | xid
newest_commit_ts_xid | xid
checkpoint_time | timestamp with time zone

Table 9.82. pg_control_system Output Columns

Column Name | Data Type
pg_control_version | integer
catalog_version_no | integer
system_identifier | bigint
pg_control_last_modified | timestamp with time zone

Table 9.83. pg_control_init Output Columns

Column Name | Data Type
max_data_alignment | integer
database_block_size | integer
blocks_per_segment | integer
wal_block_size | integer
bytes_per_wal_segment | integer
max_identifier_length | integer
max_index_columns | integer
max_toast_chunk_size | integer
large_object_chunk_size | integer
float8_pass_by_value | boolean
data_page_checksum_version | integer

Table 9.84. pg_control_recovery Output Columns

Column Name | Data Type
min_recovery_end_lsn | pg_lsn
min_recovery_end_timeline | integer
backup_start_lsn | pg_lsn
backup_end_lsn | pg_lsn
end_of_backup_record_required | boolean

Prev | Up | Next
9.25. Set Returning Functions | Home | 9.27. System Administration Functions


9.27. System Administration Functions
Prev Up Chapter 9. Functions and Operators Home Next

9.27. System Administration Functions

9.27.1. Configuration Settings Functions
9.27.2. Server Signaling Functions
9.27.3. Backup Control Functions
9.27.4. Recovery Control Functions
9.27.5. Snapshot Synchronization Functions
9.27.6. Replication Management Functions
9.27.7. Database Object Management Functions
9.27.8. Index Maintenance Functions
9.27.9. Generic File Access Functions
9.27.10. Advisory Lock Functions

The functions described in this section are used to control and monitor a PostgreSQL installation.

9.27.1. Configuration Settings Functions

Table 9.85 shows the functions available to query and alter run-time configuration parameters.

Table 9.85. Configuration Settings Functions

Function

Description

Example(s)

current_setting ( setting_name text [, missing_ok boolean ] ) → text

Returns the current value of the setting setting_name. If there is no such setting, current_setting throws an error unless missing_ok is supplied and is true (in which case NULL is returned). This function corresponds to the SQL command SHOW.

current_setting('datestyle') → ISO, MDY

set_config ( setting_name text, new_value text, is_local boolean ) → text

Sets the parameter setting_name to new_value, and returns that value. If is_local is true, the new value will only apply during the current transaction. If you want the new value to apply for the rest of the current session, use false instead. This function corresponds to the SQL command SET.

set_config('log_statement_stats', 'off', false) → off

9.27.2. Server Signaling Functions

The functions shown in Table 9.86 send control signals to other server processes. Use of these functions is restricted to superusers by default but access may be granted to others using GRANT, with noted exceptions.

Each of these functions returns true if the signal was successfully sent and false if sending the signal failed.

Table 9.86. Server Signaling Functions

Function

Description

pg_cancel_backend ( pid integer ) → boolean

Cancels the current query of the session whose backend process has the specified process ID. This is also allowed if the calling role is a member of the role whose backend is being canceled or the calling role has been granted pg_signal_backend, however only superusers can cancel superuser backends.

pg_log_backend_memory_contexts ( pid integer ) → boolean

Requests to log the memory contexts of the backend with the specified process ID. These memory contexts will be logged at LOG message level. They will appear in the server log based on the log configuration set (See Section 20.8 for more information), but will not be sent to the client regardless of client_min_messages. Only superusers can request to log the memory contexts.

pg_reload_conf () → boolean

Causes all processes of the PostgreSQL server to reload their configuration files. (This is initiated by sending a SIGHUP signal to the postmaster process, which in turn sends SIGHUP to each of its children.) You can use the pg_file_settings and pg_hba_file_rules views to check the configuration files for possible errors, before reloading.

pg_rotate_logfile () → boolean

Signals the log-file manager to switch to a new output file immediately. This works only when the built-in log collector is running, since otherwise there is no log-file manager subprocess.

pg_terminate_backend ( pid integer, timeout bigint DEFAULT 0 ) → boolean

Terminates the session whose backend process has the specified process ID. This is also allowed if the calling role is a member of the role whose backend is being terminated or the calling role has been granted pg_signal_backend, however only superusers can terminate superuser backends.

If timeout is not specified or zero, this function returns true whether the process actually terminates or not, indicating only that the sending of the signal was successful. If the timeout is specified (in milliseconds) and greater than zero, the function waits until the process is actually terminated or until the given time has passed. If the process is terminated, the function returns true. On timeout, a warning is emitted and false is returned.

pg_cancel_backend and pg_terminate_backend send signals (SIGINT or SIGTERM respectively) to backend processes identified by process ID. The process ID of an active backend can be found from the pid column of the pg_stat_activity view, or by listing the postgres processes on the server (using ps on Unix or the Task Manager on Windows). The role of an active backend can be found from the usename column of the pg_stat_activity view.

pg_log_backend_memory_contexts can be used to log the memory contexts of a backend process. For example:

postgres=# SELECT pg_log_backend_memory_contexts(pg_backend_pid());
 pg_log_backend_memory_contexts
--------------------------------
 t
(1 row)

One message for each memory context will be logged. For example:

LOG:  logging memory contexts of PID 10377
STATEMENT:  SELECT pg_log_backend_memory_contexts(pg_backend_pid());
LOG:  level: 0; TopMemoryContext: 80800 total in 6 blocks; 14432 free (5 chunks); 66368 used
LOG:  level: 1; pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1408 free (0 chunks); 6784 used
LOG:  level: 1; TopTransactionContext: 8192 total in 1 blocks; 7720 free (1 chunks); 472 used
LOG:  level: 1; RowDescriptionContext: 8192 total in 1 blocks; 6880 free (0 chunks); 1312 used
LOG:  level: 1; MessageContext: 16384 total in 2 blocks; 5152 free (0 chunks); 11232 used
LOG:  level: 1; Operator class cache: 8192 total in 1 blocks; 512 free (0 chunks); 7680 used
LOG:  level: 1; smgr relation table: 16384 total in 2 blocks; 4544 free (3 chunks); 11840 used
LOG:  level: 1; TransactionAbortContext: 32768 total in 1 blocks; 32504 free (0 chunks); 264 used
...
LOG:  level: 1; ErrorContext: 8192 total in 1 blocks; 7928 free (3 chunks); 264 used
LOG:  Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560 used

If there are more than 100 child contexts under the same parent, the first 100 child contexts are logged, along with a summary of the remaining contexts. Note that frequent calls to this function could incur significant overhead, because it may generate a large number of log messages.

9.27.3. Backup Control Functions

The functions shown in Table 9.87 assist in making on-line backups. These functions cannot be executed during recovery (except non-exclusive pg_start_backup, non-exclusive pg_stop_backup, pg_is_in_backup, pg_backup_start_time and pg_wal_lsn_diff).

For details about proper usage of these functions, see Section 26.3.

Table 9.87. Backup Control Functions

Function

Description

pg_create_restore_point ( name text ) → pg_lsn

Creates a named marker record in the write-ahead log that can later be used as a recovery target, and returns the corresponding write-ahead log location. The given name can then be used with recovery_target_name to specify the point up to which recovery will proceed. Avoid creating multiple restore points with the same name, since recovery will stop at the first one whose name matches the recovery target.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

pg_current_wal_flush_lsn () → pg_lsn

Returns the current write-ahead log flush location (see notes below).

pg_current_wal_insert_lsn () → pg_lsn

Returns the current write-ahead log insert location (see notes below).

pg_current_wal_lsn () → pg_lsn

Returns the current write-ahead log write location (see notes below).

pg_start_backup ( label text [, fast boolean [, exclusive boolean ]] ) → pg_lsn

Prepares the server to begin an on-line backup. The only required parameter is an arbitrary user-defined label for the backup. (Typically this would be the name under which the backup dump file will be stored.) If the optional second parameter is given as true, it specifies executing pg_start_backup as quickly as possible. This forces an immediate checkpoint which will cause a spike in I/O operations, slowing any concurrently executing queries. The optional third parameter specifies whether to perform an exclusive or non-exclusive backup (default is exclusive).

When used in exclusive mode, this function writes a backup label file (backup_label) and, if there are any links in the pg_tblspc/ directory, a tablespace map file (tablespace_map) into the database cluster's data directory, then performs a checkpoint, and then returns the backup's starting write-ahead log location. (The user can ignore this result value, but it is provided in case it is useful.) When used in non-exclusive mode, the contents of these files are instead returned by the pg_stop_backup function, and should be copied to the backup area by the user.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

pg_stop_backup ( exclusive boolean [, wait_for_archive boolean ] ) → setof record ( lsn pg_lsn, labelfile text, spcmapfile text )

Finishes performing an exclusive or non-exclusive on-line backup. The exclusive parameter must match the previous pg_start_backup call. In an exclusive backup, pg_stop_backup removes the backup label file and, if it exists, the tablespace map file created by pg_start_backup. In a non-exclusive backup, the desired contents of these files are returned as part of the result of the function, and should be written to files in the backup area (not in the data directory).

There is an optional second parameter of type boolean. If false, the function will return immediately after the backup is completed, without waiting for WAL to be archived. This behavior is only useful with backup software that independently monitors WAL archiving. Otherwise, WAL required to make the backup consistent might be missing and make the backup useless. By default or when this parameter is true, pg_stop_backup will wait for WAL to be archived when archiving is enabled. (On a standby, this means that it will wait only when archive_mode = always. If write activity on the primary is low, it may be useful to run pg_switch_wal on the primary in order to trigger an immediate segment switch.)

When executed on a primary, this function also creates a backup history file in the write-ahead log archive area. The history file includes the label given to pg_start_backup, the starting and ending write-ahead log locations for the backup, and the starting and ending times of the backup. After recording the ending location, the current write-ahead log insertion point is automatically advanced to the next write-ahead log file, so that the ending write-ahead log file can be archived immediately to complete the backup.

The result of the function is a single record. The lsn column holds the backup's ending write-ahead log location (which again can be ignored). The second and third columns are NULL when ending an exclusive backup; after a non-exclusive backup they hold the desired contents of the label and tablespace map files.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

pg_stop_backup () → pg_lsn

Finishes performing an exclusive on-line backup. This simplified version is equivalent to pg_stop_backup(true, true), except that it only returns the pg_lsn result.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

pg_is_in_backup () → boolean

Returns true if an on-line exclusive backup is in progress.

pg_backup_start_time () → timestamp with time zone

Returns the start time of the current on-line exclusive backup if one is in progress, otherwise NULL.

pg_switch_wal () → pg_lsn

Forces the server to switch to a new write-ahead log file, which allows the current file to be archived (assuming you are using continuous archiving). The result is the ending write-ahead log location plus 1 within the just-completed write-ahead log file. If there has been no write-ahead log activity since the last write-ahead log switch, pg_switch_wal does nothing and returns the start location of the write-ahead log file currently in use.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

pg_walfile_name ( lsn pg_lsn ) → text

Converts a write-ahead log location to the name of the WAL file holding that location.

pg_walfile_name_offset ( lsn pg_lsn ) → record ( file_name text, file_offset integer )

Converts a write-ahead log location to a WAL file name and byte offset within that file.

pg_wal_lsn_diff ( lsn1 pg_lsn, lsn2 pg_lsn ) → numeric

Calculates the difference in bytes (lsn1 - lsn2) between two write-ahead log locations. This can be used with pg_stat_replication or some of the functions shown in Table 9.87 to get the replication lag.

pg_current_wal_lsn displays the current write-ahead log write location in the same format used by the above functions. Similarly, pg_current_wal_insert_lsn displays the current write-ahead log insertion location and pg_current_wal_flush_lsn displays the current write-ahead log flush location. The insertion location is the “logical” end of the write-ahead log at any instant, while the write location is the end of what has actually been written out from the server's internal buffers, and the flush location is the last location known to be written to durable storage. The write location is the end of what can be examined from outside the server, and is usually what you want if you are interested in archiving partially-complete write-ahead log files. The insertion and flush locations are made available primarily for server debugging purposes. These are all read-only operations and do not require superuser permissions.

You can use pg_walfile_name_offset to extract the corresponding write-ahead log file name and byte offset from a pg_lsn value. For example:

postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
        file_name         | file_offset
--------------------------+-------------
 00000001000000000000000D |     4039624
(1 row)

Similarly, pg_walfile_name extracts just the write-ahead log file name. When the given write-ahead log location is exactly at a write-ahead log file boundary, both these functions return the name of the preceding write-ahead log file. This is usually the desired behavior for managing write-ahead log archiving behavior, since the preceding file is the last one that currently needs to be archived.

9.27.4. Recovery Control Functions

The functions shown in Table 9.88 provide information about the current status of a standby server. These functions may be executed both during recovery and in normal running.

Table 9.88. Recovery Information Functions

Function

Description

pg_is_in_recovery () → boolean

Returns true if recovery is still in progress.

pg_last_wal_receive_lsn () → pg_lsn

Returns the last write-ahead log location that has been received and synced to disk by streaming replication. While streaming replication is in progress this will increase monotonically. If recovery has completed then this will remain static at the location of the last WAL record received and synced to disk during recovery. If streaming replication is disabled, or if it has not yet started, the function returns NULL.

pg_last_wal_replay_lsn () → pg_lsn

Returns the last write-ahead log location that has been replayed during recovery. If recovery is still in progress this will increase monotonically. If recovery has completed then this will remain static at the location of the last WAL record applied during recovery. When the server has been started normally without recovery, the function returns NULL.

pg_last_xact_replay_timestamp () → timestamp with time zone

Returns the time stamp of the last transaction replayed during recovery. This is the time at which the commit or abort WAL record for that transaction was generated on the primary. If no transactions have been replayed during recovery, the function returns NULL. Otherwise, if recovery is still in progress this will increase monotonically. If recovery has completed then this will remain static at the time of the last transaction applied during recovery. When the server has been started normally without recovery, the function returns NULL.

The functions shown in Table 9.89 control the progress of recovery. These functions may be executed only during recovery.

Table 9.89. Recovery Control Functions

Function

Description

pg_is_wal_replay_paused () → boolean

Returns true if recovery pause is requested.

pg_get_wal_replay_pause_state () → text

Returns recovery pause state. The return values are not paused if pause is not requested, pause requested if pause is requested but recovery is not yet paused, and paused if the recovery is actually paused.

pg_promote ( wait boolean DEFAULT true, wait_seconds integer DEFAULT 60 ) → boolean

Promotes a standby server to primary status. With wait set to true (the default), the function waits until promotion is completed or wait_seconds seconds have passed, and returns true if promotion is successful and false otherwise. If wait is set to false, the function returns true immediately after sending a SIGUSR1 signal to the postmaster to trigger promotion.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

pg_wal_replay_pause () → void

Request to pause recovery. A request doesn't mean that recovery stops right away. If you want a guarantee that recovery is actually paused, you need to check for the recovery pause state returned by pg_get_wal_replay_pause_state(). Note that pg_is_wal_replay_paused() returns whether a request is made. While recovery is paused, no further database changes are applied. If hot standby is active, all new queries will see the same consistent snapshot of the database, and no further query conflicts will be generated until recovery is resumed.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

pg_wal_replay_resume () → void

Restarts recovery if it was paused.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

pg_wal_replay_pause and pg_wal_replay_resume cannot be executed while a promotion is ongoing. If a promotion is triggered while recovery is paused, the paused state ends and promotion continues.

If streaming replication is disabled, the paused state may continue indefinitely without a problem. If streaming replication is in progress then WAL records will continue to be received, which will eventually fill available disk space, depending upon the duration of the pause, the rate of WAL generation and available disk space.

9.27.5. Snapshot Synchronization Functions

PostgreSQL allows database sessions to synchronize their snapshots. A snapshot determines which data is visible to the transaction that is using the snapshot. Synchronized snapshots are necessary when two or more sessions need to see identical content in the database. If two sessions just start their transactions independently, there is always a possibility that some third transaction commits between the executions of the two START TRANSACTION commands, so that one session sees the effects of that transaction and the other does not.

To solve this problem, PostgreSQL allows a transaction to export the snapshot it is using. As long as the exporting transaction remains open, other transactions can import its snapshot, and thereby be guaranteed that they see exactly the same view of the database that the first transaction sees. But note that any database changes made by any one of these transactions remain invisible to the other transactions, as is usual for changes made by uncommitted transactions. So the transactions are synchronized with respect to pre-existing data, but act normally for changes they make themselves.

Snapshots are exported with the pg_export_snapshot function, shown in Table 9.90, and imported with the SET TRANSACTION command.

Table 9.90. Snapshot Synchronization Functions

Function

Description

pg_export_snapshot () → text

Saves the transaction's current snapshot and returns a text string identifying the snapshot. This string must be passed (outside the database) to clients that want to import the snapshot. The snapshot is available for import only until the end of the transaction that exported it.

A transaction can export more than one snapshot, if needed. Note that doing so is only useful in READ COMMITTED transactions, since in REPEATABLE READ and higher isolation levels, transactions use the same snapshot throughout their lifetime. Once a transaction has exported any snapshots, it cannot be prepared with PREPARE TRANSACTION.

9.27.6. Replication Management Functions

The functions shown in Table 9.91 are for controlling and interacting with replication features. See Section 27.2.5, Section 27.2.6, and Chapter 50 for information about the underlying features. Use of functions for replication origin is only allowed to the superuser by default, but may be allowed to other users by using the GRANT command. Use of functions for replication slots is restricted to superusers and users having REPLICATION privilege.

Many of these functions have equivalent commands in the replication protocol; see Section 53.4.

The functions described in Section 9.27.3, Section 9.27.4, and Section 9.27.5 are also relevant for replication.

Table 9.91. Replication Management Functions

Function

Description

pg_create_physical_replication_slot ( slot_name name [, immediately_reserve boolean, temporary boolean ] ) → record ( slot_name name, lsn pg_lsn )

Creates a new physical replication slot named slot_name. The optional second parameter, when true, specifies that the LSN for this replication slot be reserved immediately; otherwise the LSN is reserved on first connection from a streaming replication client. Streaming changes from a physical slot is only possible with the streaming-replication protocol — see Section 53.4. The optional third parameter, temporary, when set to true, specifies that the slot should not be permanently stored to disk and is only meant for use by the current session. Temporary slots are also released upon any error. This function corresponds to the replication protocol command CREATE_REPLICATION_SLOT ... PHYSICAL.

pg_drop_replication_slot ( slot_name name ) → void

Drops the physical or logical replication slot named slot_name. Same as replication protocol command DROP_REPLICATION_SLOT. For logical slots, this must be called while connected to the same database the slot was created on.

pg_create_logical_replication_slot ( slot_name name, plugin name [, temporary boolean, two_phase boolean ] ) → record ( slot_name name, lsn pg_lsn )

Creates a new logical (decoding) replication slot named slot_name using the output plugin plugin. The optional third parameter, temporary, when set to true, specifies that the slot should not be permanently stored to disk and is only meant for use by the current session. Temporary slots are also released upon any error. The optional fourth parameter, two_phase, when set to true, specifies that the decoding of prepared transactions is enabled for this slot. A call to this function has the same effect as the replication protocol command CREATE_REPLICATION_SLOT ... LOGICAL.

pg_copy_physical_replication_slot ( src_slot_name name, dst_slot_name name [, temporary boolean ] ) → record ( slot_name name, lsn pg_lsn )

Copies an existing physical replication slot named src_slot_name to a physical replication slot named dst_slot_name. The copied physical slot starts to reserve WAL from the same LSN as the source slot. temporary is optional. If temporary is omitted, the same value as the source slot is used.

pg_copy_logical_replication_slot ( src_slot_name name, dst_slot_name name [, temporary boolean [, plugin name ]] ) → record ( slot_name name, lsn pg_lsn )

Copies an existing logical replication slot named src_slot_name to a logical replication slot named dst_slot_name, optionally changing the output plugin and persistence. The copied logical slot starts from the same LSN as the source logical slot. Both temporary and plugin are optional; if they are omitted, the values of the source slot are used.

pg_logical_slot_get_changes ( slot_name name, upto_lsn pg_lsn, upto_nchanges integer, VARIADIC options text[] ) → setof record ( lsn pg_lsn, xid xid, data text )

Returns changes in the slot slot_name, starting from the point from which changes have been consumed last. If upto_lsn and upto_nchanges are NULL, logical decoding will continue until end of WAL. If upto_lsn is non-NULL, decoding will include only those transactions which commit prior to the specified LSN. If upto_nchanges is non-NULL, decoding will stop when the number of rows produced by decoding exceeds the specified value. Note, however, that the actual number of rows returned may be larger, since this limit is only checked after adding the rows produced when decoding each new transaction commit.

pg_logical_slot_peek_changes ( slot_name name, upto_lsn pg_lsn, upto_nchanges integer, VARIADIC options text[] ) → setof record ( lsn pg_lsn, xid xid, data text )

Behaves just like the pg_logical_slot_get_changes() function, except that changes are not consumed; that is, they will be returned again on future calls.

pg_logical_slot_get_binary_changes ( slot_name name, upto_lsn pg_lsn, upto_nchanges integer, VARIADIC options text[] ) → setof record ( lsn pg_lsn, xid xid, data bytea )

Behaves just like the pg_logical_slot_get_changes() function, except that changes are returned as bytea.

pg_logical_slot_peek_binary_changes ( slot_name name, upto_lsn pg_lsn, upto_nchanges integer, VARIADIC options text[] ) → setof record ( lsn pg_lsn, xid xid, data bytea )

Behaves just like the pg_logical_slot_peek_changes() function, except that changes are returned as bytea.

pg_replication_slot_advance ( slot_name name, upto_lsn pg_lsn ) → record ( slot_name name, end_lsn pg_lsn )

Advances the current confirmed position of a replication slot named slot_name. The slot will not be moved backwards, and it will not be moved beyond the current insert location. Returns the name of the slot and the actual position that it was advanced to. The updated slot position information is written out at the next checkpoint if any advancing is done. So in the event of a crash, the slot may return to an earlier position.

pg_replication_origin_create ( node_name text ) → oid

Creates a replication origin with the given external name, and returns the internal ID assigned to it.

pg_replication_origin_drop ( node_name text ) → void

Deletes a previously-created replication origin, including any associated replay progress.

pg_replication_origin_oid ( node_name text ) → oid

Looks up a replication origin by name and returns the internal ID. If no such replication origin is found, NULL is returned.

pg_replication_origin_session_setup ( node_name text ) → void

Marks the current session as replaying from the given origin, allowing replay progress to be tracked. Can only be used if no origin is currently selected. Use pg_replication_origin_session_reset to undo.

pg_replication_origin_session_reset () → void

Cancels the effects of pg_replication_origin_session_setup().

pg_replication_origin_session_is_setup () → boolean

Returns true if a replication origin has been selected in the current session.

pg_replication_origin_session_progress ( flush boolean ) → pg_lsn

Returns the replay location for the replication origin selected in the current session. The parameter flush determines whether the corresponding local transaction will be guaranteed to have been flushed to disk or not.

pg_replication_origin_xact_setup ( origin_lsn pg_lsn, origin_timestamp timestamp with time zone ) → void

Marks the current transaction as replaying a transaction that has committed at the given LSN and timestamp. Can only be called when a replication origin has been selected using pg_replication_origin_session_setup.

pg_replication_origin_xact_reset () → void

Cancels the effects of pg_replication_origin_xact_setup().

pg_replication_origin_advance ( node_name text, lsn pg_lsn ) → void

Sets replication progress for the given node to the given location. This is primarily useful for setting up the initial location, or setting a new location after configuration changes and similar. Be aware that careless use of this function can lead to inconsistently replicated data.

pg_replication_origin_progress ( node_name text, flush boolean ) → pg_lsn

Returns the replay location for the given replication origin. The parameter flush determines whether the corresponding local transaction will be guaranteed to have been flushed to disk or not.

pg_logical_emit_message ( transactional boolean, prefix text, content text ) → pg_lsn

pg_logical_emit_message ( transactional boolean, prefix text, content bytea ) → pg_lsn

Emits a logical decoding message. This can be used to pass generic messages to logical decoding plugins through WAL. The transactional parameter specifies if the message should be part of the current transaction, or if it should be written immediately and decoded as soon as the logical decoder reads the record. The prefix parameter is a textual prefix that can be used by logical decoding plugins to easily recognize messages that are interesting for them. The content parameter is the content of the message, given either in text or binary form.

9.27.7. Database Object Management Functions

The functions shown in Table 9.92 calculate the disk space usage of database objects, or assist in presentation or understanding of usage results. bigint results are measured in bytes. If an OID that does not represent an existing object is passed to one of these functions, NULL is returned.

Table 9.92. Database Object Size Functions

Function

Description

pg_column_size ( "any" ) → integer

Shows the number of bytes used to store any individual data value. If applied directly to a table column value, this reflects any compression that was done.

pg_column_compression ( "any" ) → text

Shows the compression algorithm that was used to compress an individual variable-length value. Returns NULL if the value is not compressed.

pg_database_size ( name ) → bigint

pg_database_size ( oid ) → bigint

Computes the total disk space used by the database with the specified name or OID. To use this function, you must have CONNECT privilege on the specified database (which is granted by default) or be a member of the pg_read_all_stats role.

pg_indexes_size ( regclass ) → bigint

Computes the total disk space used by indexes attached to the specified table.

pg_relation_size ( relation regclass [, fork text ] ) → bigint

Computes the disk space used by one “fork” of the specified relation. (Note that for most purposes it is more convenient to use the higher-level functions pg_total_relation_size or pg_table_size, which sum the sizes of all forks.) With one argument, this returns the size of the main data fork of the relation. The second argument can be provided to specify which fork to examine:

  • main returns the size of the main data fork of the relation.

  • fsm returns the size of the Free Space Map (see Section 70.3) associated with the relation.

  • vm returns the size of the Visibility Map (see Section 70.4) associated with the relation.

  • init returns the size of the initialization fork, if any, associated with the relation.

pg_size_bytes ( text ) → bigint

Converts a size in human-readable format (as returned by pg_size_pretty) into bytes.

pg_size_pretty ( bigint ) → text

pg_size_pretty ( numeric ) → text

Converts a size in bytes into a more easily human-readable format with size units (bytes, kB, MB, GB or TB as appropriate). Note that the units are powers of 2 rather than powers of 10, so 1kB is 1024 bytes, 1MB is 10242 = 1048576 bytes, and so on.

pg_table_size ( regclass ) → bigint

Computes the disk space used by the specified table, excluding indexes (but including its TOAST table if any, free space map, and visibility map).

pg_tablespace_size ( name ) → bigint

pg_tablespace_size ( oid ) → bigint

Computes the total disk space used in the tablespace with the specified name or OID. To use this function, you must have CREATE privilege on the specified tablespace or be a member of the pg_read_all_stats role, unless it is the default tablespace for the current database.

pg_total_relation_size ( regclass ) → bigint

Computes the total disk space used by the specified table, including all indexes and TOAST data. The result is equivalent to pg_table_size + pg_indexes_size.

The functions above that operate on tables or indexes accept a regclass argument, which is simply the OID of the table or index in the pg_class system catalog. You do not have to look up the OID by hand, however, since the regclass data type's input converter will do the work for you. See Section 8.19 for details.

The functions shown in Table 9.93 assist in identifying the specific disk files associated with database objects.

Table 9.93. Database Object Location Functions

Function

Description

pg_relation_filenode ( relation regclass ) → oid

Returns the “filenode” number currently assigned to the specified relation. The filenode is the base component of the file name(s) used for the relation (see Section 70.1 for more information). For most relations the result is the same as pg_class.relfilenode, but for certain system catalogs relfilenode is zero and this function must be used to get the correct value. The function returns NULL if passed a relation that does not have storage, such as a view.

pg_relation_filepath ( relation regclass ) → text

Returns the entire file path name (relative to the database cluster's data directory, PGDATA) of the relation.

pg_filenode_relation ( tablespace oid, filenode oid ) → regclass

Returns a relation's OID given the tablespace OID and filenode it is stored under. This is essentially the inverse mapping of pg_relation_filepath. For a relation in the database's default tablespace, the tablespace can be specified as zero. Returns NULL if no relation in the current database is associated with the given values.

Table 9.94 lists functions used to manage collations.

Table 9.94. Collation Management Functions

Function

Description

pg_collation_actual_version ( oid ) → text

Returns the actual version of the collation object as it is currently installed in the operating system. If this is different from the value in pg_collation.collversion, then objects depending on the collation might need to be rebuilt. See also ALTER COLLATION.

pg_import_system_collations ( schema regnamespace ) → integer

Adds collations to the system catalog pg_collation based on all the locales it finds in the operating system. This is what initdb uses; see Section 24.2.2 for more details. If additional locales are installed into the operating system later on, this function can be run again to add collations for the new locales. Locales that match existing entries in pg_collation will be skipped. (But collation objects based on locales that are no longer present in the operating system are not removed by this function.) The schema parameter would typically be pg_catalog, but that is not a requirement; the collations could be installed into some other schema as well. The function returns the number of new collation objects it created. Use of this function is restricted to superusers.

Table 9.95 lists functions that provide information about the structure of partitioned tables.

Table 9.95. Partitioning Information Functions

Function

Description

pg_partition_tree ( regclass ) → setof record ( relid regclass, parentrelid regclass, isleaf boolean, level integer )

Lists the tables or indexes in the partition tree of the given partitioned table or partitioned index, with one row for each partition. Information provided includes the OID of the partition, the OID of its immediate parent, a boolean value telling if the partition is a leaf, and an integer telling its level in the hierarchy. The level value is 0 for the input table or index, 1 for its immediate child partitions, 2 for their partitions, and so on. Returns no rows if the relation does not exist or is not a partition or partitioned table.

pg_partition_ancestors ( regclass ) → setof regclass

Lists the ancestor relations of the given partition, including the relation itself. Returns no rows if the relation does not exist or is not a partition or partitioned table.

pg_partition_root ( regclass ) → regclass

Returns the top-most parent of the partition tree to which the given relation belongs. Returns NULL if the relation does not exist or is not a partition or partitioned table.

For example, to check the total size of the data contained in a partitioned table measurement, one could use the following query:

SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS total_size
  FROM pg_partition_tree('measurement');

9.27.8. Index Maintenance Functions

Table 9.96 shows the functions available for index maintenance tasks. (Note that these maintenance tasks are normally done automatically by autovacuum; use of these functions is only required in special cases.) These functions cannot be executed during recovery. Use of these functions is restricted to superusers and the owner of the given index.

Table 9.96. Index Maintenance Functions

Function

Description

brin_summarize_new_values ( index regclass ) → integer

Scans the specified BRIN index to find page ranges in the base table that are not currently summarized by the index; for any such range it creates a new summary index tuple by scanning those table pages. Returns the number of new page range summaries that were inserted into the index.

brin_summarize_range ( index regclass, blockNumber bigint ) → integer

Summarizes the page range covering the given block, if not already summarized. This is like brin_summarize_new_values except that it only processes the page range that covers the given table block number.

brin_desummarize_range ( index regclass, blockNumber bigint ) → void

Removes the BRIN index tuple that summarizes the page range covering the given table block, if there is one.

gin_clean_pending_list ( index regclass ) → bigint

Cleans up the “pending” list of the specified GIN index by moving entries in it, in bulk, to the main GIN data structure. Returns the number of pages removed from the pending list. If the argument is a GIN index built with the fastupdate option disabled, no cleanup happens and the result is zero, because the index doesn't have a pending list. See Section 67.4.1 and Section 67.5 for details about the pending list and fastupdate option.

9.27.9. Generic File Access Functions

The functions shown in Table 9.97 provide native access to files on the machine hosting the server. Only files within the database cluster directory and the log_directory can be accessed, unless the user is a superuser or is granted the role pg_read_server_files. Use a relative path for files in the cluster directory, and a path matching the log_directory configuration setting for log files.

Note that granting users the EXECUTE privilege on pg_read_file(), or related functions, allows them the ability to read any file on the server that the database server process can read; these functions bypass all in-database privilege checks. This means that, for example, a user with such access is able to read the contents of the pg_authid table where authentication information is stored, as well as read any table data in the database. Therefore, granting access to these functions should be carefully considered.

Some of these functions take an optional missing_ok parameter, which specifies the behavior when the file or directory does not exist. If true, the function returns NULL or an empty result set, as appropriate. If false, an error is raised. The default is false.

Table 9.97. Generic File Access Functions

Function

Description

pg_ls_dir ( dirname text [, missing_ok boolean, include_dot_dirs boolean ] ) → setof text

Returns the names of all files (and directories and other special files) in the specified directory. The include_dot_dirs parameter indicates whether “.” and “..” are to be included in the result set; the default is to exclude them. Including them can be useful when missing_ok is true, to distinguish an empty directory from a non-existent directory.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

pg_ls_logdir () → setof record ( name text, size bigint, modification timestamp with time zone )

Returns the name, size, and last modification time (mtime) of each ordinary file in the server's log directory. Filenames beginning with a dot, directories, and other special files are excluded.

This function is restricted to superusers and members of the pg_monitor role by default, but other users can be granted EXECUTE to run the function.

pg_ls_waldir () → setof record ( name text, size bigint, modification timestamp with time zone )

Returns the name, size, and last modification time (mtime) of each ordinary file in the server's write-ahead log (WAL) directory. Filenames beginning with a dot, directories, and other special files are excluded.

This function is restricted to superusers and members of the pg_monitor role by default, but other users can be granted EXECUTE to run the function.

pg_ls_archive_statusdir () → setof record ( name text, size bigint, modification timestamp with time zone )

Returns the name, size, and last modification time (mtime) of each ordinary file in the server's WAL archive status directory (pg_wal/archive_status). Filenames beginning with a dot, directories, and other special files are excluded.

This function is restricted to superusers and members of the pg_monitor role by default, but other users can be granted EXECUTE to run the function.

pg_ls_tmpdir ( [ tablespace oid ] ) → setof record ( name text, size bigint, modification timestamp with time zone )

Returns the name, size, and last modification time (mtime) of each ordinary file in the temporary file directory for the specified tablespace. If tablespace is not provided, the pg_default tablespace is examined. Filenames beginning with a dot, directories, and other special files are excluded.

This function is restricted to superusers and members of the pg_monitor role by default, but other users can be granted EXECUTE to run the function.

pg_read_file ( filename text [, offset bigint, length bigint [, missing_ok boolean ]] ) → text

Returns all or part of a text file, starting at the given byte offset, returning at most length bytes (less if the end of file is reached first). If offset is negative, it is relative to the end of the file. If offset and length are omitted, the entire file is returned. The bytes read from the file are interpreted as a string in the database's encoding; an error is thrown if they are not valid in that encoding.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

pg_read_binary_file ( filename text [, offset bigint, length bigint [, missing_ok boolean ]] ) → bytea

Returns all or part of a file. This function is identical to pg_read_file except that it can read arbitrary binary data, returning the result as bytea not text; accordingly, no encoding checks are performed.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

In combination with the convert_from function, this function can be used to read a text file in a specified encoding and convert to the database's encoding:

SELECT convert_from(pg_read_binary_file('file_in_utf8.txt'), 'UTF8');

pg_stat_file ( filename text [, missing_ok boolean ] ) → record ( size bigint, access timestamp with time zone, modification timestamp with time zone, change timestamp with time zone, creation timestamp with time zone, isdir boolean )

Returns a record containing the file's size, last access time stamp, last modification time stamp, last file status change time stamp (Unix platforms only), file creation time stamp (Windows only), and a flag indicating if it is a directory.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

9.27.10. Advisory Lock Functions

The functions shown in Table 9.98 manage advisory locks. For details about proper use of these functions, see Section 13.3.5.

All these functions are intended to be used to lock application-defined resources, which can be identified either by a single 64-bit key value or two 32-bit key values (note that these two key spaces do not overlap). If another session already holds a conflicting lock on the same resource identifier, the functions will either wait until the resource becomes available, or return a false result, as appropriate for the function. Locks can be either shared or exclusive: a shared lock does not conflict with other shared locks on the same resource, only with exclusive locks. Locks can be taken at session level (so that they are held until released or the session ends) or at transaction level (so that they are held until the current transaction ends; there is no provision for manual release). Multiple session-level lock requests stack, so that if the same resource identifier is locked three times there must then be three unlock requests to release the resource in advance of session end.

Table 9.98. Advisory Lock Functions

Function

Description

pg_advisory_lock ( key bigint ) → void

pg_advisory_lock ( key1 integer, key2 integer ) → void

Obtains an exclusive session-level advisory lock, waiting if necessary.

pg_advisory_lock_shared ( key bigint ) → void

pg_advisory_lock_shared ( key1 integer, key2 integer ) → void

Obtains a shared session-level advisory lock, waiting if necessary.

pg_advisory_unlock ( key bigint ) → boolean

pg_advisory_unlock ( key1 integer, key2 integer ) → boolean

Releases a previously-acquired exclusive session-level advisory lock. Returns true if the lock is successfully released. If the lock was not held, false is returned, and in addition, an SQL warning will be reported by the server.

pg_advisory_unlock_all () → void

Releases all session-level advisory locks held by the current session. (This function is implicitly invoked at session end, even if the client disconnects ungracefully.)

pg_advisory_unlock_shared ( key bigint ) → boolean

pg_advisory_unlock_shared ( key1 integer, key2 integer ) → boolean

Releases a previously-acquired shared session-level advisory lock. Returns true if the lock is successfully released. If the lock was not held, false is returned, and in addition, an SQL warning will be reported by the server.

pg_advisory_xact_lock ( key bigint ) → void

pg_advisory_xact_lock ( key1 integer, key2 integer ) → void

Obtains an exclusive transaction-level advisory lock, waiting if necessary.

pg_advisory_xact_lock_shared ( key bigint ) → void

pg_advisory_xact_lock_shared ( key1 integer, key2 integer ) → void

Obtains a shared transaction-level advisory lock, waiting if necessary.

pg_try_advisory_lock ( key bigint ) → boolean

pg_try_advisory_lock ( key1 integer, key2 integer ) → boolean

Obtains an exclusive session-level advisory lock if available. This will either obtain the lock immediately and return true, or return false without waiting if the lock cannot be acquired immediately.

pg_try_advisory_lock_shared ( key bigint ) → boolean

pg_try_advisory_lock_shared ( key1 integer, key2 integer ) → boolean

Obtains a shared session-level advisory lock if available. This will either obtain the lock immediately and return true, or return false without waiting if the lock cannot be acquired immediately.

pg_try_advisory_xact_lock ( key bigint ) → boolean

pg_try_advisory_xact_lock ( key1 integer, key2 integer ) → boolean

Obtains an exclusive transaction-level advisory lock if available. This will either obtain the lock immediately and return true, or return false without waiting if the lock cannot be acquired immediately.

pg_try_advisory_xact_lock_shared ( key bigint ) → boolean

pg_try_advisory_xact_lock_shared ( key1 integer, key2 integer ) → boolean

Obtains a shared transaction-level advisory lock if available. This will either obtain the lock immediately and return true, or return false without waiting if the lock cannot be acquired immediately.

Prev | Up | Next
9.26. System Information Functions and Operators | Home | 9.28. Trigger Functions


9.28. Trigger Functions
Prev Up Chapter 9. Functions and Operators Home Next

9.28. Trigger Functions

While many uses of triggers involve user-written trigger functions, PostgreSQL provides a few built-in trigger functions that can be used directly in user-defined triggers. These are summarized in Table 9.99. (Additional built-in trigger functions exist, which implement foreign key constraints and deferred index constraints. Those are not documented here since users need not use them directly.)

For more information about creating triggers, see CREATE TRIGGER.

Table 9.99. Built-In Trigger Functions

Function

Description

Example Usage

suppress_redundant_updates_trigger ( ) → trigger

Suppresses do-nothing update operations. See below for details.

CREATE TRIGGER ... suppress_redundant_updates_trigger()

tsvector_update_trigger ( ) → trigger

Automatically updates a tsvector column from associated plain-text document column(s). The text search configuration to use is specified by name as a trigger argument. See Section 12.4.3 for details.

CREATE TRIGGER ... tsvector_update_trigger(tsvcol, 'pg_catalog.swedish', title, body)

tsvector_update_trigger_column ( ) → trigger

Automatically updates a tsvector column from associated plain-text document column(s). The text search configuration to use is taken from a regconfig column of the table. See Section 12.4.3 for details.

CREATE TRIGGER ... tsvector_update_trigger_column(tsvcol, tsconfigcol, title, body)

The suppress_redundant_updates_trigger function, when applied as a row-level BEFORE UPDATE trigger, will prevent any update that does not actually change the data in the row from taking place. This overrides the normal behavior which always performs a physical row update regardless of whether or not the data has changed. (This normal behavior makes updates run faster, since no checking is required, and is also useful in certain cases.)

Ideally, you should avoid running updates that don't actually change the data in the record. Redundant updates can cost considerable unnecessary time, especially if there are lots of indexes to alter, and space in dead rows that will eventually have to be vacuumed. However, detecting such situations in client code is not always easy, or even possible, and writing expressions to detect them can be error-prone. An alternative is to use suppress_redundant_updates_trigger, which will skip updates that don't change the data. You should use this with care, however. The trigger takes a small but non-trivial time for each record, so if most of the records affected by updates do actually change, use of this trigger will make updates run slower on average.

The suppress_redundant_updates_trigger function can be added to a table like this:

CREATE TRIGGER z_min_update
BEFORE UPDATE ON tablename
FOR EACH ROW EXECUTE FUNCTION suppress_redundant_updates_trigger();

In most cases, you need to fire this trigger last for each row, so that it does not override other triggers that might wish to alter the row. Bearing in mind that triggers fire in name order, you would therefore choose a trigger name that comes after the name of any other trigger you might have on the table. (Hence the “z” prefix in the example.)

Prev | Up | Next
9.27. System Administration Functions | Home | 9.29. Event Trigger Functions


9.29. Event Trigger Functions
Prev Up Chapter 9. Functions and Operators Home Next

9.29. Event Trigger Functions

9.29.1. Capturing Changes at Command End
9.29.2. Processing Objects Dropped by a DDL Command
9.29.3. Handling a Table Rewrite Event

PostgreSQL provides these helper functions to retrieve information from event triggers.

For more information about event triggers, see Chapter 40.

9.29.1. Capturing Changes at Command End

pg_event_trigger_ddl_commands () → setof record

pg_event_trigger_ddl_commands returns a list of DDL commands executed by each user action, when invoked in a function attached to a ddl_command_end event trigger. If called in any other context, an error is raised. pg_event_trigger_ddl_commands returns one row for each base command executed; some commands that are a single SQL sentence may return more than one row. This function returns the following columns:

Name | Type | Description
classid | oid | OID of catalog the object belongs in
objid | oid | OID of the object itself
objsubid | integer | Sub-object ID (e.g., attribute number for a column)
command_tag | text | Command tag
object_type | text | Type of the object
schema_name | text | Name of the schema the object belongs in, if any; otherwise NULL. No quoting is applied.
object_identity | text | Text rendering of the object identity, schema-qualified. Each identifier included in the identity is quoted if necessary.
in_extension | boolean | True if the command is part of an extension script
command | pg_ddl_command | A complete representation of the command, in internal format. This cannot be output directly, but it can be passed to other functions to obtain different pieces of information about the command.

9.29.2. Processing Objects Dropped by a DDL Command

pg_event_trigger_dropped_objects () → setof record

pg_event_trigger_dropped_objects returns a list of all objects dropped by the command in whose sql_drop event it is called. If called in any other context, an error is raised. This function returns the following columns:

Name | Type | Description
classid | oid | OID of catalog the object belonged in
objid | oid | OID of the object itself
objsubid | integer | Sub-object ID (e.g., attribute number for a column)
original | boolean | True if this was one of the root object(s) of the deletion
normal | boolean | True if there was a normal dependency relationship in the dependency graph leading to this object
is_temporary | boolean | True if this was a temporary object
object_type | text | Type of the object
schema_name | text | Name of the schema the object belonged in, if any; otherwise NULL. No quoting is applied.
object_name | text | Name of the object, if the combination of schema and name can be used as a unique identifier for the object; otherwise NULL. No quoting is applied, and name is never schema-qualified.
object_identity | text | Text rendering of the object identity, schema-qualified. Each identifier included in the identity is quoted if necessary.
address_names | text[] | An array that, together with object_type and address_args, can be used by the pg_get_object_address function to recreate the object address in a remote server containing an identically named object of the same kind.
address_args | text[] | Complement for address_names

The pg_event_trigger_dropped_objects function can be used in an event trigger like this:

CREATE FUNCTION test_event_trigger_for_drops()
        RETURNS event_trigger LANGUAGE plpgsql AS $$
DECLARE
    obj record;
BEGIN
    FOR obj IN SELECT * FROM pg_event_trigger_dropped_objects()
    LOOP
        RAISE NOTICE '% dropped object: % %.% %',
                     tg_tag,
                     obj.object_type,
                     obj.schema_name,
                     obj.object_name,
                     obj.object_identity;
    END LOOP;
END;
$$;
CREATE EVENT TRIGGER test_event_trigger_for_drops
   ON sql_drop
   EXECUTE FUNCTION test_event_trigger_for_drops();

9.29.3. Handling a Table Rewrite Event

The functions shown in Table 9.100 provide information about a table for which a table_rewrite event has just been called. If called in any other context, an error is raised.

Table 9.100. Table Rewrite Information Functions

Function

Description

pg_event_trigger_table_rewrite_oid () → oid

Returns the OID of the table about to be rewritten.

pg_event_trigger_table_rewrite_reason () → integer

Returns a code explaining the reason(s) for rewriting. The exact meaning of the codes is release dependent.

These functions can be used in an event trigger like this:

CREATE FUNCTION test_event_trigger_table_rewrite_oid()
 RETURNS event_trigger
 LANGUAGE plpgsql AS
$$
BEGIN
  RAISE NOTICE 'rewriting table % for reason %',
                pg_event_trigger_table_rewrite_oid()::regclass,
                pg_event_trigger_table_rewrite_reason();
END;
$$;

CREATE EVENT TRIGGER test_table_rewrite_oid
                  ON table_rewrite
   EXECUTE FUNCTION test_event_trigger_table_rewrite_oid();

Prev | Up | Next
9.28. Trigger Functions | Home | 9.30. Statistics Information Functions


9.30. Statistics Information Functions
Prev Up Chapter 9. Functions and Operators Home Next

9.30. Statistics Information Functions

9.30.1. Inspecting MCV Lists

PostgreSQL provides a function to inspect complex statistics defined using the CREATE STATISTICS command.

9.30.1. Inspecting MCV Lists

pg_mcv_list_items ( pg_mcv_list ) → setof record

pg_mcv_list_items returns a set of records describing all items stored in a multi-column MCV list. It returns the following columns:

Name | Type | Description
index | integer | index of the item in the MCV list
values | text[] | values stored in the MCV item
nulls | boolean[] | flags identifying NULL values
frequency | double precision | frequency of this MCV item
base_frequency | double precision | base frequency of this MCV item

The pg_mcv_list_items function can be used like this:

SELECT m.* FROM pg_statistic_ext join pg_statistic_ext_data on (oid = stxoid),
                pg_mcv_list_items(stxdmcv) m WHERE stxname = 'stts';

Values of the pg_mcv_list type can be obtained only from the pg_statistic_ext_data.stxdmcv column.

Prev | Up | Next
9.29. Event Trigger Functions | Home | Chapter 10. Type Conversion


Chapter 11. Indexes
Prev Up Part II. The SQL Language Home Next

Chapter 11. Indexes

Table of Contents

11.1. Introduction
11.2. Index Types
11.2.1. B-Tree
11.2.2. Hash
11.2.3. GiST
11.2.4. SP-GiST
11.2.5. GIN
11.2.6. BRIN
11.3. Multicolumn Indexes
11.4. Indexes and ORDER BY
11.5. Combining Multiple Indexes
11.6. Unique Indexes
11.7. Indexes on Expressions
11.8. Partial Indexes
11.9. Index-Only Scans and Covering Indexes
11.10. Operator Classes and Operator Families
11.11. Indexes and Collations
11.12. Examining Index Usage

Indexes are a common way to enhance database performance. An index allows the database server to find and retrieve specific rows much faster than it could do without an index. But indexes also add overhead to the database system as a whole, so they should be used sensibly.

Prev | Up | Next
10.6. SELECT Output Columns | Home | 11.1. Introduction


11.1. Introduction
Prev Up Chapter 11. Indexes Home Next

11.1. Introduction

Suppose we have a table similar to this:

CREATE TABLE test1 (
    id integer,
    content varchar
);

and the application issues many queries of the form:

SELECT content FROM test1 WHERE id = constant;

With no advance preparation, the system would have to scan the entire test1 table, row by row, to find all matching entries. If there are many rows in test1 and only a few rows (perhaps zero or one) that would be returned by such a query, this is clearly an inefficient method. But if the system has been instructed to maintain an index on the id column, it can use a more efficient method for locating matching rows. For instance, it might only have to walk a few levels deep into a search tree.

A similar approach is used in most non-fiction books: terms and concepts that are frequently looked up by readers are collected in an alphabetic index at the end of the book. The interested reader can scan the index relatively quickly and flip to the appropriate page(s), rather than having to read the entire book to find the material of interest. Just as it is the task of the author to anticipate the items that readers are likely to look up, it is the task of the database programmer to foresee which indexes will be useful.

The following command can be used to create an index on the id column, as discussed:

CREATE INDEX test1_id_index ON test1 (id);

The name test1_id_index can be chosen freely, but you should pick something that enables you to remember later what the index was for.

To remove an index, use the DROP INDEX command. Indexes can be added to and removed from tables at any time.

Once an index is created, no further intervention is required: the system will update the index when the table is modified, and it will use the index in queries when it thinks doing so would be more efficient than a sequential table scan. But you might have to run the ANALYZE command regularly to update statistics to allow the query planner to make educated decisions. See Chapter 14 for information about how to find out whether an index is used and when and why the planner might choose not to use an index.

Indexes can also benefit UPDATE and DELETE commands with search conditions. Indexes can moreover be used in join searches. Thus, an index defined on a column that is part of a join condition can also significantly speed up queries with joins.

Creating an index on a large table can take a long time. By default, PostgreSQL allows reads (SELECT statements) to occur on the table in parallel with index creation, but writes (INSERT, UPDATE, DELETE) are blocked until the index build is finished. In production environments this is often unacceptable. It is possible to allow writes to occur in parallel with index creation, but there are several caveats to be aware of — for more information see Building Indexes Concurrently.

After an index is created, the system has to keep it synchronized with the table. This adds overhead to data manipulation operations. Therefore indexes that are seldom or never used in queries should be removed.

Prev | Up | Next
Chapter 11. Indexes | Home | 11.2. Index Types


11.2. Index Types
Prev Up Chapter 11. Indexes Home Next

11.2. Index Types

11.2.1. B-Tree
11.2.2. Hash
11.2.3. GiST
11.2.4. SP-GiST
11.2.5. GIN
11.2.6. BRIN

PostgreSQL provides several index types: B-tree, Hash, GiST, SP-GiST, GIN and BRIN. Each index type uses a different algorithm that is best suited to different types of queries. By default, the CREATE INDEX command creates B-tree indexes, which fit the most common situations. The other index types are selected by writing the keyword USING followed by the index type name. For example, to create a Hash index:

CREATE INDEX name ON table USING HASH (column);

11.2.1. B-Tree

B-trees can handle equality and range queries on data that can be sorted into some ordering. In particular, the PostgreSQL query planner will consider using a B-tree index whenever an indexed column is involved in a comparison using one of these operators:

<   <=   =   >=   >

Constructs equivalent to combinations of these operators, such as BETWEEN and IN, can also be implemented with a B-tree index search. Also, an IS NULL or IS NOT NULL condition on an index column can be used with a B-tree index.

The optimizer can also use a B-tree index for queries involving the pattern matching operators LIKE and ~ if the pattern is a constant and is anchored to the beginning of the string — for example, col LIKE 'foo%' or col ~ '^foo', but not col LIKE '%bar'. However, if your database does not use the C locale you will need to create the index with a special operator class to support indexing of pattern-matching queries; see Section 11.10 below. It is also possible to use B-tree indexes for ILIKE and ~*, but only if the pattern starts with non-alphabetic characters, i.e., characters that are not affected by upper/lower case conversion.

B-tree indexes can also be used to retrieve data in sorted order. This is not always faster than a simple scan and sort, but it is often helpful.

11.2.2. Hash

Hash indexes store a 32-bit hash code derived from the value of the indexed column. Hence, such indexes can only handle simple equality comparisons. The query planner will consider using a hash index whenever an indexed column is involved in a comparison using the equal operator:

=

11.2.3. GiST

GiST indexes are not a single kind of index, but rather an infrastructure within which many different indexing strategies can be implemented. Accordingly, the particular operators with which a GiST index can be used vary depending on the indexing strategy (the operator class). As an example, the standard distribution of PostgreSQL includes GiST operator classes for several two-dimensional geometric data types, which support indexed queries using these operators:

<<   &<   &>   >>   <<|   &<|   |&>   |>>   @>   <@   ~=   &&

(See Section 9.11 for the meaning of these operators.) The GiST operator classes included in the standard distribution are documented in Table 65.1. Many other GiST operator classes are available in the contrib collection or as separate projects. For more information see Chapter 65.

GiST indexes are also capable of optimizing “nearest-neighbor” searches, such as

SELECT * FROM places ORDER BY location <-> point '(101,456)' LIMIT 10;

which finds the ten places closest to a given target point. The ability to do this is again dependent on the particular operator class being used. In Table 65.1, operators that can be used in this way are listed in the column “Ordering Operators”.

11.2.4. SP-GiST

SP-GiST indexes, like GiST indexes, offer an infrastructure that supports various kinds of searches. SP-GiST permits implementation of a wide range of different non-balanced disk-based data structures, such as quadtrees, k-d trees, and radix trees (tries). As an example, the standard distribution of PostgreSQL includes SP-GiST operator classes for two-dimensional points, which support indexed queries using these operators:

<<   >>   ~=   <@   <<|   |>>

(See Section 9.11 for the meaning of these operators.) The SP-GiST operator classes included in the standard distribution are documented in Table 66.1. For more information see Chapter 66.

Like GiST, SP-GiST supports “nearest-neighbor” searches. For SP-GiST operator classes that support distance ordering, the corresponding operator is listed in the “Ordering Operators” column in Table 66.1.

11.2.5. GIN

GIN indexes are “inverted indexes” which are appropriate for data values that contain multiple component values, such as arrays. An inverted index contains a separate entry for each component value, and can efficiently handle queries that test for the presence of specific component values.

Like GiST and SP-GiST, GIN can support many different user-defined indexing strategies, and the particular operators with which a GIN index can be used vary depending on the indexing strategy. As an example, the standard distribution of PostgreSQL includes a GIN operator class for arrays, which supports indexed queries using these operators:

<@   @>   =   &&

(See Section 9.19 for the meaning of these operators.) The GIN operator classes included in the standard distribution are documented in Table 67.1. Many other GIN operator classes are available in the contrib collection or as separate projects. For more information see Chapter 67.

11.2.6. BRIN

BRIN indexes (a shorthand for Block Range INdexes) store summaries about the values stored in consecutive physical block ranges of a table. Thus, they are most effective for columns whose values are well-correlated with the physical order of the table rows. Like GiST, SP-GiST and GIN, BRIN can support many different indexing strategies, and the particular operators with which a BRIN index can be used vary depending on the indexing strategy. For data types that have a linear sort order, the indexed data corresponds to the minimum and maximum values of the values in the column for each block range. This supports indexed queries using these operators:

<   <=   =   >=   >

The BRIN operator classes included in the standard distribution are documented in Table 68.1. For more information see Chapter 68.

Prev | Up | Next
11.1. Introduction | Home | 11.3. Multicolumn Indexes


11.3. Multicolumn Indexes
Prev Up Chapter 11. Indexes Home Next

11.3. Multicolumn Indexes

An index can be defined on more than one column of a table. For example, if you have a table of this form:

CREATE TABLE test2 (
  major int,
  minor int,
  name varchar
);

(say, you keep your /dev directory in a database...) and you frequently issue queries like:

SELECT name FROM test2 WHERE major = constant AND minor = constant;

then it might be appropriate to define an index on the columns major and minor together, e.g.:

CREATE INDEX test2_mm_idx ON test2 (major, minor);

Currently, only the B-tree, GiST, GIN, and BRIN index types support multiple-key-column indexes. Whether there can be multiple key columns is independent of whether INCLUDE columns can be added to the index. Indexes can have up to 32 columns, including INCLUDE columns. (This limit can be altered when building PostgreSQL; see the file pg_config_manual.h.)

A multicolumn B-tree index can be used with query conditions that involve any subset of the index's columns, but the index is most efficient when there are constraints on the leading (leftmost) columns. The exact rule is that equality constraints on leading columns, plus any inequality constraints on the first column that does not have an equality constraint, will be used to limit the portion of the index that is scanned. Constraints on columns to the right of these columns are checked in the index, so they save visits to the table proper, but they do not reduce the portion of the index that has to be scanned. For example, given an index on (a, b, c) and a query condition WHERE a = 5 AND b >= 42 AND c < 77, the index would have to be scanned from the first entry with a = 5 and b = 42 up through the last entry with a = 5. Index entries with c >= 77 would be skipped, but they'd still have to be scanned through. This index could in principle be used for queries that have constraints on b and/or c with no constraint on a — but the entire index would have to be scanned, so in most cases the planner would prefer a sequential table scan over using the index.

A multicolumn GiST index can be used with query conditions that involve any subset of the index's columns. Conditions on additional columns restrict the entries returned by the index, but the condition on the first column is the most important one for determining how much of the index needs to be scanned. A GiST index will be relatively ineffective if its first column has only a few distinct values, even if there are many distinct values in additional columns.

A multicolumn GIN index can be used with query conditions that involve any subset of the index's columns. Unlike B-tree or GiST, index search effectiveness is the same regardless of which index column(s) the query conditions use.

A multicolumn BRIN index can be used with query conditions that involve any subset of the index's columns. Like GIN and unlike B-tree or GiST, index search effectiveness is the same regardless of which index column(s) the query conditions use. The only reason to have multiple BRIN indexes instead of one multicolumn BRIN index on a single table is to have a different pages_per_range storage parameter.

Of course, each column must be used with operators appropriate to the index type; clauses that involve other operators will not be considered.

Multicolumn indexes should be used sparingly. In most situations, an index on a single column is sufficient and saves space and time. Indexes with more than three columns are unlikely to be helpful unless the usage of the table is extremely stylized. See also Section 11.5 and Section 11.9 for some discussion of the merits of different index configurations.

Prev | Up | Next
11.2. Index Types | Home | 11.4. Indexes and ORDER BY


11.4. Indexes and ORDER BY
Prev Up Chapter 11. Indexes Home Next

11.4. Indexes and ORDER BY

In addition to simply finding the rows to be returned by a query, an index may be able to deliver them in a specific sorted order. This allows a query's ORDER BY specification to be honored without a separate sorting step. Of the index types currently supported by PostgreSQL, only B-tree can produce sorted output — the other index types return matching rows in an unspecified, implementation-dependent order.

The planner will consider satisfying an ORDER BY specification either by scanning an available index that matches the specification, or by scanning the table in physical order and doing an explicit sort. For a query that requires scanning a large fraction of the table, an explicit sort is likely to be faster than using an index because it requires less disk I/O due to following a sequential access pattern. Indexes are more useful when only a few rows need be fetched. An important special case is ORDER BY in combination with LIMIT n: an explicit sort will have to process all the data to identify the first n rows, but if there is an index matching the ORDER BY, the first n rows can be retrieved directly, without scanning the remainder at all.

By default, B-tree indexes store their entries in ascending order with nulls last (table TID is treated as a tiebreaker column among otherwise equal entries). This means that a forward scan of an index on column x produces output satisfying ORDER BY x (or more verbosely, ORDER BY x ASC NULLS LAST). The index can also be scanned backward, producing output satisfying ORDER BY x DESC (or more verbosely, ORDER BY x DESC NULLS FIRST, since NULLS FIRST is the default for ORDER BY DESC).

You can adjust the ordering of a B-tree index by including the options ASC, DESC, NULLS FIRST, and/or NULLS LAST when creating the index; for example:

CREATE INDEX test2_info_nulls_low ON test2 (info NULLS FIRST);
CREATE INDEX test3_desc_index ON test3 (id DESC NULLS LAST);

An index stored in ascending order with nulls first can satisfy either ORDER BY x ASC NULLS FIRST or ORDER BY x DESC NULLS LAST depending on which direction it is scanned in.

You might wonder why bother providing all four options, when two options together with the possibility of backward scan would cover all the variants of ORDER BY. In single-column indexes the options are indeed redundant, but in multicolumn indexes they can be useful. Consider a two-column index on (x, y): this can satisfy ORDER BY x, y if we scan forward, or ORDER BY x DESC, y DESC if we scan backward. But it might be that the application frequently needs to use ORDER BY x ASC, y DESC. There is no way to get that ordering from a plain index, but it is possible if the index is defined as (x ASC, y DESC) or (x DESC, y ASC).

Obviously, indexes with non-default sort orderings are a fairly specialized feature, but sometimes they can produce tremendous speedups for certain queries. Whether it's worth maintaining such an index depends on how often you use queries that require a special sort ordering.

Prev | Up | Next
11.3. Multicolumn Indexes | Home | 11.5. Combining Multiple Indexes


11.5. Combining Multiple Indexes
Prev Up Chapter 11. Indexes Home Next

11.5. Combining Multiple Indexes

A single index scan can only use query clauses that use the index's columns with operators of its operator class and are joined with AND. For example, given an index on (a, b) a query condition like WHERE a = 5 AND b = 6 could use the index, but a query like WHERE a = 5 OR b = 6 could not directly use the index.

Fortunately, PostgreSQL has the ability to combine multiple indexes (including multiple uses of the same index) to handle cases that cannot be implemented by single index scans. The system can form AND and OR conditions across several index scans. For example, a query like WHERE x = 42 OR x = 47 OR x = 53 OR x = 99 could be broken down into four separate scans of an index on x, each scan using one of the query clauses. The results of these scans are then ORed together to produce the result. Another example is that if we have separate indexes on x and y, one possible implementation of a query like WHERE x = 5 AND y = 6 is to use each index with the appropriate query clause and then AND together the index results to identify the result rows.

To combine multiple indexes, the system scans each needed index and prepares a bitmap in memory giving the locations of table rows that are reported as matching that index's conditions. The bitmaps are then ANDed and ORed together as needed by the query. Finally, the actual table rows are visited and returned. The table rows are visited in physical order, because that is how the bitmap is laid out; this means that any ordering of the original indexes is lost, and so a separate sort step will be needed if the query has an ORDER BY clause. For this reason, and because each additional index scan adds extra time, the planner will sometimes choose to use a simple index scan even though additional indexes are available that could have been used as well.

In all but the simplest applications, there are various combinations of indexes that might be useful, and the database developer must make trade-offs to decide which indexes to provide. Sometimes multicolumn indexes are best, but sometimes it's better to create separate indexes and rely on the index-combination feature. For example, if your workload includes a mix of queries that sometimes involve only column x, sometimes only column y, and sometimes both columns, you might choose to create two separate indexes on x and y, relying on index combination to process the queries that use both columns. You could also create a multicolumn index on (x, y). This index would typically be more efficient than index combination for queries involving both columns, but as discussed in Section 11.3, it would be almost useless for queries involving only y, so it should not be the only index. A combination of the multicolumn index and a separate index on y would serve reasonably well. For queries involving only x, the multicolumn index could be used, though it would be larger and hence slower than an index on x alone. The last alternative is to create all three indexes, but this is probably only reasonable if the table is searched much more often than it is updated and all three types of query are common. If one of the types of query is much less common than the others, you'd probably settle for creating just the two indexes that best match the common types.

Prev | Up | Next
11.4. Indexes and ORDER BY | Home | 11.6. Unique Indexes


11.6. Unique Indexes
Prev Up Chapter 11. Indexes Home Next

11.6. Unique Indexes

Indexes can also be used to enforce uniqueness of a column's value, or the uniqueness of the combined values of more than one column.

CREATE UNIQUE INDEX name ON table (column [, ...]);

Currently, only B-tree indexes can be declared unique.

When an index is declared unique, multiple table rows with equal indexed values are not allowed. Null values are not considered equal. A multicolumn unique index will only reject cases where all indexed columns are equal in multiple rows.

PostgreSQL automatically creates a unique index when a unique constraint or primary key is defined for a table. The index covers the columns that make up the primary key or unique constraint (a multicolumn index, if appropriate), and is the mechanism that enforces the constraint.

Note

There's no need to manually create indexes on unique columns; doing so would just duplicate the automatically-created index.

Prev | Up | Next
11.5. Combining Multiple Indexes | Home | 11.7. Indexes on Expressions


11.7. Indexes on Expressions
Prev Up Chapter 11. Indexes Home Next

11.7. Indexes on Expressions

An index column need not be just a column of the underlying table, but can be a function or scalar expression computed from one or more columns of the table. This feature is useful to obtain fast access to tables based on the results of computations.

For example, a common way to do case-insensitive comparisons is to use the lower function:

SELECT * FROM test1 WHERE lower(col1) = 'value';

This query can use an index if one has been defined on the result of the lower(col1) function:

CREATE INDEX test1_lower_col1_idx ON test1 (lower(col1));

If we were to declare this index UNIQUE, it would prevent creation of rows whose col1 values differ only in case, as well as rows whose col1 values are actually identical. Thus, indexes on expressions can be used to enforce constraints that are not definable as simple unique constraints.

As another example, if one often does queries like:

SELECT * FROM people WHERE (first_name || ' ' || last_name) = 'John Smith';

then it might be worth creating an index like this:

CREATE INDEX people_names ON people ((first_name || ' ' || last_name));

The syntax of the CREATE INDEX command normally requires writing parentheses around index expressions, as shown in the second example. The parentheses can be omitted when the expression is just a function call, as in the first example.

Index expressions are relatively expensive to maintain, because the derived expression(s) must be computed for each row insertion and non-HOT update. However, the index expressions are not recomputed during an indexed search, since they are already stored in the index. In both examples above, the system sees the query as just WHERE indexedcolumn = 'constant' and so the speed of the search is equivalent to any other simple index query. Thus, indexes on expressions are useful when retrieval speed is more important than insertion and update speed.

Prev | Up | Next
11.6. Unique Indexes | Home | 11.8. Partial Indexes


11.8. Partial Indexes
Prev Up Chapter 11. Indexes Home Next

11.8. Partial Indexes

A partial index is an index built over a subset of a table; the subset is defined by a conditional expression (called the predicate of the partial index). The index contains entries only for those table rows that satisfy the predicate. Partial indexes are a specialized feature, but there are several situations in which they are useful.

One major reason for using a partial index is to avoid indexing common values. Since a query searching for a common value (one that accounts for more than a few percent of all the table rows) will not use the index anyway, there is no point in keeping those rows in the index at all. This reduces the size of the index, which will speed up those queries that do use the index. It will also speed up many table update operations because the index does not need to be updated in all cases. Example 11.1 shows a possible application of this idea.

Example 11.1. Setting up a Partial Index to Exclude Common Values

Suppose you are storing web server access logs in a database. Most accesses originate from the IP address range of your organization but some are from elsewhere (say, employees on dial-up connections). If your searches by IP are primarily for outside accesses, you probably do not need to index the IP range that corresponds to your organization's subnet.

Assume a table like this:

CREATE TABLE access_log (
    url varchar,
    client_ip inet,
    ...
);

To create a partial index that suits our example, use a command such as this:

CREATE INDEX access_log_client_ip_ix ON access_log (client_ip)
WHERE NOT (client_ip > inet '192.168.100.0' AND
           client_ip < inet '192.168.100.255');

A typical query that can use this index would be:

SELECT *
FROM access_log
WHERE url = '/index.html' AND client_ip = inet '212.78.10.32';

Here the query's IP address is covered by the partial index. The following query cannot use the partial index, as it uses an IP address that is excluded from the index:

SELECT *
FROM access_log
WHERE url = '/index.html' AND client_ip = inet '192.168.100.23';

Observe that this kind of partial index requires that the common values be predetermined, so such partial indexes are best used for data distributions that do not change. Such indexes can be recreated occasionally to adjust for new data distributions, but this adds maintenance effort.

Another possible use for a partial index is to exclude values from the index that the typical query workload is not interested in; this is shown in Example 11.2. This results in the same advantages as listed above, but it prevents the “uninteresting” values from being accessed via that index, even if an index scan might be profitable in that case. Obviously, setting up partial indexes for this kind of scenario will require a lot of care and experimentation.

Example 11.2. Setting up a Partial Index to Exclude Uninteresting Values

If you have a table that contains both billed and unbilled orders, where the unbilled orders take up a small fraction of the total table and yet those are the most-accessed rows, you can improve performance by creating an index on just the unbilled rows. The command to create the index would look like this:

CREATE INDEX orders_unbilled_index ON orders (order_nr)
    WHERE billed is not true;

A possible query to use this index would be:

SELECT * FROM orders WHERE billed is not true AND order_nr < 10000;

However, the index can also be used in queries that do not involve order_nr at all, e.g.:

SELECT * FROM orders WHERE billed is not true AND amount > 5000.00;

This is not as efficient as a partial index on the amount column would be, since the system has to scan the entire index. Yet, if there are relatively few unbilled orders, using this partial index just to find the unbilled orders could be a win.

Note that this query cannot use this index:

SELECT * FROM orders WHERE order_nr = 3501;

The order 3501 might be among the billed or unbilled orders.

Example 11.2 also illustrates that the indexed column and the column used in the predicate do not need to match. PostgreSQL supports partial indexes with arbitrary predicates, so long as only columns of the table being indexed are involved. However, keep in mind that the predicate must match the conditions used in the queries that are supposed to benefit from the index. To be precise, a partial index can be used in a query only if the system can recognize that the WHERE condition of the query mathematically implies the predicate of the index. PostgreSQL does not have a sophisticated theorem prover that can recognize mathematically equivalent expressions that are written in different forms. (Not only is such a general theorem prover extremely difficult to create, it would probably be too slow to be of any real use.) The system can recognize simple inequality implications, for example “x < 1” implies “x < 2”; otherwise the predicate condition must exactly match part of the query's WHERE condition or the index will not be recognized as usable. Matching takes place at query planning time, not at run time. As a result, parameterized query clauses do not work with a partial index. For example a prepared query with a parameter might specify “x < ?” which will never imply “x < 2” for all possible values of the parameter.

A third possible use for partial indexes does not require the index to be used in queries at all. The idea here is to create a unique index over a subset of a table, as in Example 11.3. This enforces uniqueness among the rows that satisfy the index predicate, without constraining those that do not.

Example 11.3. Setting up a Partial Unique Index

Suppose that we have a table describing test outcomes. We wish to ensure that there is only one “successful” entry for a given subject and target combination, but there might be any number of “unsuccessful” entries. Here is one way to do it:

CREATE TABLE tests (
    subject text,
    target text,
    success boolean,
    ...
);

CREATE UNIQUE INDEX tests_success_constraint ON tests (subject, target)
    WHERE success;

This is a particularly efficient approach when there are few successful tests and many unsuccessful ones. It is also possible to allow only one null in a column by creating a unique partial index with an IS NULL restriction.

Finally, a partial index can also be used to override the system's query plan choices. Also, data sets with peculiar distributions might cause the system to use an index when it really should not. In that case the index can be set up so that it is not available for the offending query. Normally, PostgreSQL makes reasonable choices about index usage (e.g., it avoids them when retrieving common values, so the earlier example really only saves index size, it is not required to avoid index usage), and grossly incorrect plan choices are cause for a bug report.

Keep in mind that setting up a partial index indicates that you know at least as much as the query planner knows, in particular you know when an index might be profitable. Forming this knowledge requires experience and understanding of how indexes in PostgreSQL work. In most cases, the advantage of a partial index over a regular index will be minimal. There are cases where they are quite counterproductive, as in Example 11.4.

Example 11.4. Do Not Use Partial Indexes as a Substitute for Partitioning

You might be tempted to create a large set of non-overlapping partial indexes, for example

CREATE INDEX mytable_cat_1 ON mytable (data) WHERE category = 1;
CREATE INDEX mytable_cat_2 ON mytable (data) WHERE category = 2;
CREATE INDEX mytable_cat_3 ON mytable (data) WHERE category = 3;
...
CREATE INDEX mytable_cat_N ON mytable (data) WHERE category = N;

This is a bad idea! Almost certainly, you'll be better off with a single non-partial index, declared like

CREATE INDEX mytable_cat_data ON mytable (category, data);

(Put the category column first, for the reasons described in Section 11.3.) While a search in this larger index might have to descend through a couple more tree levels than a search in a smaller index, that's almost certainly going to be cheaper than the planner effort needed to select the appropriate one of the partial indexes. The core of the problem is that the system does not understand the relationship among the partial indexes, and will laboriously test each one to see if it's applicable to the current query.

If your table is large enough that a single index really is a bad idea, you should look into using partitioning instead (see Section 5.11). With that mechanism, the system does understand that the tables and indexes are non-overlapping, so far better performance is possible.

More information about partial indexes can be found in [ston89b], [olson93], and [seshadri95].

Prev | Up | Next
11.7. Indexes on Expressions | Home | 11.9. Index-Only Scans and Covering Indexes


11.9. Index-Only Scans and Covering Indexes
Prev Up Chapter 11. Indexes Home Next

11.9. Index-Only Scans and Covering Indexes

All indexes in PostgreSQL are secondary indexes, meaning that each index is stored separately from the table's main data area (which is called the table's heap in PostgreSQL terminology). This means that in an ordinary index scan, each row retrieval requires fetching data from both the index and the heap. Furthermore, while the index entries that match a given indexable WHERE condition are usually close together in the index, the table rows they reference might be anywhere in the heap. The heap-access portion of an index scan thus involves a lot of random access into the heap, which can be slow, particularly on traditional rotating media. (As described in Section 11.5, bitmap scans try to alleviate this cost by doing the heap accesses in sorted order, but that only goes so far.)

To solve this performance problem, PostgreSQL supports index-only scans, which can answer queries from an index alone without any heap access. The basic idea is to return values directly out of each index entry instead of consulting the associated heap entry. There are two fundamental restrictions on when this method can be used:

  1. The index type must support index-only scans. B-tree indexes always do. GiST and SP-GiST indexes support index-only scans for some operator classes but not others. Other index types have no support. The underlying requirement is that the index must physically store, or else be able to reconstruct, the original data value for each index entry. As a counterexample, GIN indexes cannot support index-only scans because each index entry typically holds only part of the original data value.

  2. The query must reference only columns stored in the index. For example, given an index on columns x and y of a table that also has a column z, these queries could use index-only scans:

    SELECT x, y FROM tab WHERE x = 'key';
SELECT x FROM tab WHERE x = 'key' AND y < 42;

    but these queries could not:

    SELECT x, z FROM tab WHERE x = 'key';
SELECT x FROM tab WHERE x = 'key' AND z < 42;

    (Expression indexes and partial indexes complicate this rule, as discussed below.)

If these two fundamental requirements are met, then all the data values required by the query are available from the index, so an index-only scan is physically possible. But there is an additional requirement for any table scan in PostgreSQL: it must verify that each retrieved row be “visible” to the query's MVCC snapshot, as discussed in Chapter 13. Visibility information is not stored in index entries, only in heap entries; so at first glance it would seem that every row retrieval would require a heap access anyway. And this is indeed the case, if the table row has been modified recently. However, for seldom-changing data there is a way around this problem. PostgreSQL tracks, for each page in a table's heap, whether all rows stored in that page are old enough to be visible to all current and future transactions. This information is stored in a bit in the table's visibility map. An index-only scan, after finding a candidate index entry, checks the visibility map bit for the corresponding heap page. If it's set, the row is known visible and so the data can be returned with no further work. If it's not set, the heap entry must be visited to find out whether it's visible, so no performance advantage is gained over a standard index scan. Even in the successful case, this approach trades visibility map accesses for heap accesses; but since the visibility map is four orders of magnitude smaller than the heap it describes, far less physical I/O is needed to access it. In most situations the visibility map remains cached in memory all the time.

In short, while an index-only scan is possible given the two fundamental requirements, it will be a win only if a significant fraction of the table's heap pages have their all-visible map bits set. But tables in which a large fraction of the rows are unchanging are common enough to make this type of scan very useful in practice.

To make effective use of the index-only scan feature, you might choose to create a covering index, which is an index specifically designed to include the columns needed by a particular type of query that you run frequently. Since queries typically need to retrieve more columns than just the ones they search on, PostgreSQL allows you to create an index in which some columns are just “payload” and are not part of the search key. This is done by adding an INCLUDE clause listing the extra columns. For example, if you commonly run queries like

SELECT y FROM tab WHERE x = 'key';

the traditional approach to speeding up such queries would be to create an index on x only. However, an index defined as

CREATE INDEX tab_x_y ON tab(x) INCLUDE (y);

could handle these queries as index-only scans, because y can be obtained from the index without visiting the heap.

Because column y is not part of the index's search key, it does not have to be of a data type that the index can handle; it's merely stored in the index and is not interpreted by the index machinery. Also, if the index is a unique index, that is

CREATE UNIQUE INDEX tab_x_y ON tab(x) INCLUDE (y);

the uniqueness condition applies to just column x, not to the combination of x and y. (An INCLUDE clause can also be written in UNIQUE and PRIMARY KEY constraints, providing alternative syntax for setting up an index like this.)

It's wise to be conservative about adding non-key payload columns to an index, especially wide columns. If an index tuple exceeds the maximum size allowed for the index type, data insertion will fail. In any case, non-key columns duplicate data from the index's table and bloat the size of the index, thus potentially slowing searches. And remember that there is little point in including payload columns in an index unless the table changes slowly enough that an index-only scan is likely to not need to access the heap. If the heap tuple must be visited anyway, it costs nothing more to get the column's value from there. Other restrictions are that expressions are not currently supported as included columns, and that only B-tree, GiST and SP-GiST indexes currently support included columns.

Before PostgreSQL had the INCLUDE feature, people sometimes made covering indexes by writing the payload columns as ordinary index columns, that is writing

CREATE INDEX tab_x_y ON tab(x, y);

even though they had no intention of ever using y as part of a WHERE clause. This works fine as long as the extra columns are trailing columns; making them be leading columns is unwise for the reasons explained in Section 11.3. However, this method doesn't support the case where you want the index to enforce uniqueness on the key column(s).

Suffix truncation always removes non-key columns from upper B-Tree levels. As payload columns, they are never used to guide index scans. The truncation process also removes one or more trailing key column(s) when the remaining prefix of key column(s) happens to be sufficient to describe tuples on the lowest B-Tree level. In practice, covering indexes without an INCLUDE clause often avoid storing columns that are effectively payload in the upper levels. However, explicitly defining payload columns as non-key columns reliably keeps the tuples in upper levels small.

In principle, index-only scans can be used with expression indexes. For example, given an index on f(x) where x is a table column, it should be possible to execute

SELECT f(x) FROM tab WHERE f(x) < 1;

as an index-only scan; and this is very attractive if f() is an expensive-to-compute function. However, PostgreSQL's planner is currently not very smart about such cases. It considers a query to be potentially executable by index-only scan only when all columns needed by the query are available from the index. In this example, x is not needed except in the context f(x), but the planner does not notice that and concludes that an index-only scan is not possible. If an index-only scan seems sufficiently worthwhile, this can be worked around by adding x as an included column, for example

CREATE INDEX tab_f_x ON tab (f(x)) INCLUDE (x);

An additional caveat, if the goal is to avoid recalculating f(x), is that the planner won't necessarily match uses of f(x) that aren't in indexable WHERE clauses to the index column. It will usually get this right in simple queries such as shown above, but not in queries that involve joins. These deficiencies may be remedied in future versions of PostgreSQL.

Partial indexes also have interesting interactions with index-only scans. Consider the partial index shown in Example 11.3:

CREATE UNIQUE INDEX tests_success_constraint ON tests (subject, target)
    WHERE success;

In principle, we could do an index-only scan on this index to satisfy a query like

SELECT target FROM tests WHERE subject = 'some-subject' AND success;

But there's a problem: the WHERE clause refers to success which is not available as a result column of the index. Nonetheless, an index-only scan is possible because the plan does not need to recheck that part of the WHERE clause at run time: all entries found in the index necessarily have success = true so this need not be explicitly checked in the plan. PostgreSQL versions 9.6 and later will recognize such cases and allow index-only scans to be generated, but older versions will not.

Prev | Up | Next
11.8. Partial Indexes | Home | 11.10. Operator Classes and Operator Families


11.10. Operator Classes and Operator Families
Prev Up Chapter 11. Indexes Home Next

11.10. Operator Classes and Operator Families

An index definition can specify an operator class for each column of an index.

CREATE INDEX name ON table (column opclass [ ( opclass_options ) ] [sort options] [, ...]);

The operator class identifies the operators to be used by the index for that column. For example, a B-tree index on the type int4 would use the int4_ops class; this operator class includes comparison functions for values of type int4. In practice the default operator class for the column's data type is usually sufficient. The main reason for having operator classes is that for some data types, there could be more than one meaningful index behavior. For example, we might want to sort a complex-number data type either by absolute value or by real part. We could do this by defining two operator classes for the data type and then selecting the proper class when making an index. The operator class determines the basic sort ordering (which can then be modified by adding sort options COLLATE, ASC/DESC and/or NULLS FIRST/NULLS LAST).

There are also some built-in operator classes besides the default ones:

  • The operator classes text_pattern_ops, varchar_pattern_ops, and bpchar_pattern_ops support B-tree indexes on the types text, varchar, and char respectively. The difference from the default operator classes is that the values are compared strictly character by character rather than according to the locale-specific collation rules. This makes these operator classes suitable for use by queries involving pattern matching expressions (LIKE or POSIX regular expressions) when the database does not use the standard “C” locale. As an example, you might index a varchar column like this:

    CREATE INDEX test_index ON test_table (col varchar_pattern_ops);

    Note that you should also create an index with the default operator class if you want queries involving ordinary <, <=, >, or >= comparisons to use an index. Such queries cannot use the xxx_pattern_ops operator classes. (Ordinary equality comparisons can use these operator classes, however.) It is possible to create multiple indexes on the same column with different operator classes. If you do use the C locale, you do not need the xxx_pattern_ops operator classes, because an index with the default operator class is usable for pattern-matching queries in the C locale.

The following query shows all defined operator classes:

SELECT am.amname AS index_method,
       opc.opcname AS opclass_name,
       opc.opcintype::regtype AS indexed_type,
       opc.opcdefault AS is_default
    FROM pg_am am, pg_opclass opc
    WHERE opc.opcmethod = am.oid
    ORDER BY index_method, opclass_name;

An operator class is actually just a subset of a larger structure called an operator family. In cases where several data types have similar behaviors, it is frequently useful to define cross-data-type operators and allow these to work with indexes. To do this, the operator classes for each of the types must be grouped into the same operator family. The cross-type operators are members of the family, but are not associated with any single class within the family.

This expanded version of the previous query shows the operator family each operator class belongs to:

SELECT am.amname AS index_method,
       opc.opcname AS opclass_name,
       opf.opfname AS opfamily_name,
       opc.opcintype::regtype AS indexed_type,
       opc.opcdefault AS is_default
    FROM pg_am am, pg_opclass opc, pg_opfamily opf
    WHERE opc.opcmethod = am.oid AND
          opc.opcfamily = opf.oid
    ORDER BY index_method, opclass_name;

This query shows all defined operator families and all the operators included in each family:

SELECT am.amname AS index_method,
       opf.opfname AS opfamily_name,
       amop.amopopr::regoperator AS opfamily_operator
    FROM pg_am am, pg_opfamily opf, pg_amop amop
    WHERE opf.opfmethod = am.oid AND
          amop.amopfamily = opf.oid
    ORDER BY index_method, opfamily_name, opfamily_operator;

Tip

psql has commands \dAc, \dAf, and \dAo, which provide slightly more sophisticated versions of these queries.

Prev | Up | Next
11.9. Index-Only Scans and Covering Indexes | Home | 11.11. Indexes and Collations


11.11. Indexes and Collations
Prev Up Chapter 11. Indexes Home Next

11.11. Indexes and Collations

An index can support only one collation per index column. If multiple collations are of interest, multiple indexes may be needed.

Consider these statements:

CREATE TABLE test1c (
    id integer,
    content varchar COLLATE "x"
);

CREATE INDEX test1c_content_index ON test1c (content);

The index automatically uses the collation of the underlying column. So a query of the form

SELECT * FROM test1c WHERE content > constant;

could use the index, because the comparison will by default use the collation of the column. However, this index cannot accelerate queries that involve some other collation. So if queries of the form, say,

SELECT * FROM test1c WHERE content > constant COLLATE "y";

are also of interest, an additional index could be created that supports the "y" collation, like this:

CREATE INDEX test1c_content_y_index ON test1c (content COLLATE "y");

Prev | Up | Next
11.10. Operator Classes and Operator Families | Home | 11.12. Examining Index Usage


11.12. Examining Index Usage
Prev Up Chapter 11. Indexes Home Next

11.12. Examining Index Usage

Although indexes in PostgreSQL do not need maintenance or tuning, it is still important to check which indexes are actually used by the real-life query workload. Examining index usage for an individual query is done with the EXPLAIN command; its application for this purpose is illustrated in Section 14.1. It is also possible to gather overall statistics about index usage in a running server, as described in Section 28.2.

It is difficult to formulate a general procedure for determining which indexes to create. There are a number of typical cases that have been shown in the examples throughout the previous sections. A good deal of experimentation is often necessary. The rest of this section gives some tips for that:

  • Always run ANALYZE first. This command collects statistics about the distribution of the values in the table. This information is required to estimate the number of rows returned by a query, which is needed by the planner to assign realistic costs to each possible query plan. In absence of any real statistics, some default values are assumed, which are almost certain to be inaccurate. Examining an application's index usage without having run ANALYZE is therefore a lost cause. See Section 25.1.3 and Section 25.1.6 for more information.

  • Use real data for experimentation. Using test data for setting up indexes will tell you what indexes you need for the test data, but that is all.

    It is especially fatal to use very small test data sets. While selecting 1000 out of 100000 rows could be a candidate for an index, selecting 1 out of 100 rows will hardly be, because the 100 rows probably fit within a single disk page, and there is no plan that can beat sequentially fetching 1 disk page.

    Also be careful when making up test data, which is often unavoidable when the application is not yet in production. Values that are very similar, completely random, or inserted in sorted order will skew the statistics away from the distribution that real data would have.

  • When indexes are not used, it can be useful for testing to force their use. There are run-time parameters that can turn off various plan types (see Section 20.7.1). For instance, turning off sequential scans (enable_seqscan) and nested-loop joins (enable_nestloop), which are the most basic plans, will force the system to use a different plan. If the system still chooses a sequential scan or nested-loop join then there is probably a more fundamental reason why the index is not being used; for example, the query condition does not match the index. (What kind of query can use what kind of index is explained in the previous sections.)

  • If forcing index usage does use the index, then there are two possibilities: Either the system is right and using the index is indeed not appropriate, or the cost estimates of the query plans are not reflecting reality. So you should time your query with and without indexes. The EXPLAIN ANALYZE command can be useful here.

  • If it turns out that the cost estimates are wrong, there are, again, two possibilities. The total cost is computed from the per-row costs of each plan node times the selectivity estimate of the plan node. The costs estimated for the plan nodes can be adjusted via run-time parameters (described in Section 20.7.2). An inaccurate selectivity estimate is due to insufficient statistics. It might be possible to improve this by tuning the statistics-gathering parameters (see ALTER TABLE).

    If you do not succeed in adjusting the costs to be more appropriate, then you might have to resort to forcing index usage explicitly. You might also want to contact the PostgreSQL developers to examine the issue.

Prev | Up | Next
11.11. Indexes and Collations | Home | Chapter 12. Full Text Search


Chapter 13. Concurrency Control
Prev Up Part II. The SQL Language Home Next

Chapter 13. Concurrency Control

Table of Contents

13.1. Introduction
13.2. Transaction Isolation
13.2.1. Read Committed Isolation Level
13.2.2. Repeatable Read Isolation Level
13.2.3. Serializable Isolation Level
13.3. Explicit Locking
13.3.1. Table-Level Locks
13.3.2. Row-Level Locks
13.3.3. Page-Level Locks
13.3.4. Deadlocks
13.3.5. Advisory Locks
13.4. Data Consistency Checks at the Application Level
13.4.1. Enforcing Consistency with Serializable Transactions
13.4.2. Enforcing Consistency with Explicit Blocking Locks
13.5. Caveats
13.6. Locking and Indexes

This chapter describes the behavior of the PostgreSQL database system when two or more sessions try to access the same data at the same time. The goals in that situation are to allow efficient access for all sessions while maintaining strict data integrity. Every developer of database applications should be familiar with the topics covered in this chapter.

Prev | Up | Next
12.11. Limitations | Home | 13.1. Introduction


13.1. Introduction
Prev Up Chapter 13. Concurrency Control Home Next

13.1. Introduction

PostgreSQL provides a rich set of tools for developers to manage concurrent access to data. Internally, data consistency is maintained by using a multiversion model (Multiversion Concurrency Control, MVCC). This means that each SQL statement sees a snapshot of data (a database version) as it was some time ago, regardless of the current state of the underlying data. This prevents statements from viewing inconsistent data produced by concurrent transactions performing updates on the same data rows, providing transaction isolation for each database session. MVCC, by eschewing the locking methodologies of traditional database systems, minimizes lock contention in order to allow for reasonable performance in multiuser environments.

The main advantage of using the MVCC model of concurrency control rather than locking is that in MVCC locks acquired for querying (reading) data do not conflict with locks acquired for writing data, and so reading never blocks writing and writing never blocks reading. PostgreSQL maintains this guarantee even when providing the strictest level of transaction isolation through the use of an innovative Serializable Snapshot Isolation (SSI) level.

Table- and row-level locking facilities are also available in PostgreSQL for applications which don't generally need full transaction isolation and prefer to explicitly manage particular points of conflict. However, proper use of MVCC will generally provide better performance than locks. In addition, application-defined advisory locks provide a mechanism for acquiring locks that are not tied to a single transaction.

Prev | Up | Next
Chapter 13. Concurrency Control | Home | 13.2. Transaction Isolation


13.3. Explicit Locking
Prev Up Chapter 13. Concurrency Control Home Next

13.3. Explicit Locking

13.3.1. Table-Level Locks
13.3.2. Row-Level Locks
13.3.3. Page-Level Locks
13.3.4. Deadlocks
13.3.5. Advisory Locks

PostgreSQL provides various lock modes to control concurrent access to data in tables. These modes can be used for application-controlled locking in situations where MVCC does not give the desired behavior. Also, most PostgreSQL commands automatically acquire locks of appropriate modes to ensure that referenced tables are not dropped or modified in incompatible ways while the command executes. (For example, TRUNCATE cannot safely be executed concurrently with other operations on the same table, so it obtains an ACCESS EXCLUSIVE lock on the table to enforce that.)

To examine a list of the currently outstanding locks in a database server, use the pg_locks system view. For more information on monitoring the status of the lock manager subsystem, refer to Chapter 28.

13.3.1. Table-Level Locks

The list below shows the available lock modes and the contexts in which they are used automatically by PostgreSQL. You can also acquire any of these locks explicitly with the command LOCK. Remember that all of these lock modes are table-level locks, even if the name contains the word “row”; the names of the lock modes are historical. To some extent the names reflect the typical usage of each lock mode — but the semantics are all the same. The only real difference between one lock mode and another is the set of lock modes with which each conflicts (see Table 13.2). Two transactions cannot hold locks of conflicting modes on the same table at the same time. (However, a transaction never conflicts with itself. For example, it might acquire ACCESS EXCLUSIVE lock and later acquire ACCESS SHARE lock on the same table.) Non-conflicting lock modes can be held concurrently by many transactions. Notice in particular that some lock modes are self-conflicting (for example, an ACCESS EXCLUSIVE lock cannot be held by more than one transaction at a time) while others are not self-conflicting (for example, an ACCESS SHARE lock can be held by multiple transactions).

Table-Level Lock Modes

ACCESS SHARE

Conflicts with the ACCESS EXCLUSIVE lock mode only.

The SELECT command acquires a lock of this mode on referenced tables. In general, any query that only reads a table and does not modify it will acquire this lock mode.

ROW SHARE

Conflicts with the EXCLUSIVE and ACCESS EXCLUSIVE lock modes.

The SELECT FOR UPDATE and SELECT FOR SHARE commands acquire a lock of this mode on the target table(s) (in addition to ACCESS SHARE locks on any other tables that are referenced but not selected FOR UPDATE/FOR SHARE).

ROW EXCLUSIVE

Conflicts with the SHARE, SHARE ROW EXCLUSIVE, EXCLUSIVE, and ACCESS EXCLUSIVE lock modes.

The commands UPDATE, DELETE, and INSERT acquire this lock mode on the target table (in addition to ACCESS SHARE locks on any other referenced tables). In general, this lock mode will be acquired by any command that modifies data in a table.

SHARE UPDATE EXCLUSIVE

Conflicts with the SHARE UPDATE EXCLUSIVE, SHARE, SHARE ROW EXCLUSIVE, EXCLUSIVE, and ACCESS EXCLUSIVE lock modes. This mode protects a table against concurrent schema changes and VACUUM runs.

Acquired by VACUUM (without FULL), ANALYZE, CREATE INDEX CONCURRENTLY, CREATE STATISTICS, COMMENT ON, REINDEX CONCURRENTLY, and certain ALTER INDEX and ALTER TABLE variants (for full details see the documentation of these commands).

SHARE

Conflicts with the ROW EXCLUSIVE, SHARE UPDATE EXCLUSIVE, SHARE ROW EXCLUSIVE, EXCLUSIVE, and ACCESS EXCLUSIVE lock modes. This mode protects a table against concurrent data changes.

Acquired by CREATE INDEX (without CONCURRENTLY).

SHARE ROW EXCLUSIVE

Conflicts with the ROW EXCLUSIVE, SHARE UPDATE EXCLUSIVE, SHARE, SHARE ROW EXCLUSIVE, EXCLUSIVE, and ACCESS EXCLUSIVE lock modes. This mode protects a table against concurrent data changes, and is self-exclusive so that only one session can hold it at a time.

Acquired by CREATE TRIGGER and some forms of ALTER TABLE.

EXCLUSIVE

Conflicts with the ROW SHARE, ROW EXCLUSIVE, SHARE UPDATE EXCLUSIVE, SHARE, SHARE ROW EXCLUSIVE, EXCLUSIVE, and ACCESS EXCLUSIVE lock modes. This mode allows only concurrent ACCESS SHARE locks, i.e., only reads from the table can proceed in parallel with a transaction holding this lock mode.

Acquired by REFRESH MATERIALIZED VIEW CONCURRENTLY.

ACCESS EXCLUSIVE

Conflicts with locks of all modes (ACCESS SHARE, ROW SHARE, ROW EXCLUSIVE, SHARE UPDATE EXCLUSIVE, SHARE, SHARE ROW EXCLUSIVE, EXCLUSIVE, and ACCESS EXCLUSIVE). This mode guarantees that the holder is the only transaction accessing the table in any way.

Acquired by the DROP TABLE, TRUNCATE, REINDEX, CLUSTER, VACUUM FULL, and REFRESH MATERIALIZED VIEW (without CONCURRENTLY) commands. Many forms of ALTER INDEX and ALTER TABLE also acquire a lock at this level. This is also the default lock mode for LOCK TABLE statements that do not specify a mode explicitly.

Tip

Only an ACCESS EXCLUSIVE lock blocks a SELECT (without FOR UPDATE/SHARE) statement.

Once acquired, a lock is normally held until the end of the transaction. But if a lock is acquired after establishing a savepoint, the lock is released immediately if the savepoint is rolled back to. This is consistent with the principle that ROLLBACK cancels all effects of the commands since the savepoint. The same holds for locks acquired within a PL/pgSQL exception block: an error escape from the block releases locks acquired within it.

Table 13.2. Conflicting Lock Modes

Requested Lock Mode | Existing Lock Mode
 | ACCESS SHARE | ROW SHARE | ROW EXCL. | SHARE UPDATE EXCL. | SHARE | SHARE ROW EXCL. | EXCL. | ACCESS EXCL.
ACCESS SHARE |  |  |  |  |  |  |  | X
ROW SHARE |  |  |  |  |  |  | X | X
ROW EXCL. |  |  |  |  | X | X | X | X
SHARE UPDATE EXCL. |  |  |  | X | X | X | X | X
SHARE |  |  | X | X |  | X | X | X
SHARE ROW EXCL. |  |  | X | X | X | X | X | X
EXCL. |  | X | X | X | X | X | X | X
ACCESS EXCL. | X | X | X | X | X | X | X | X

13.3.2. Row-Level Locks

In addition to table-level locks, there are row-level locks, which are listed as below with the contexts in which they are used automatically by PostgreSQL. See Table 13.3 for a complete table of row-level lock conflicts. Note that a transaction can hold conflicting locks on the same row, even in different subtransactions; but other than that, two transactions can never hold conflicting locks on the same row. Row-level locks do not affect data querying; they block only writers and lockers to the same row. Row-level locks are released at transaction end or during savepoint rollback, just like table-level locks.

Row-Level Lock Modes

FOR UPDATE

FOR UPDATE causes the rows retrieved by the SELECT statement to be locked as though for update. This prevents them from being locked, modified or deleted by other transactions until the current transaction ends. That is, other transactions that attempt UPDATE, DELETE, SELECT FOR UPDATE, SELECT FOR NO KEY UPDATE, SELECT FOR SHARE or SELECT FOR KEY SHARE of these rows will be blocked until the current transaction ends; conversely, SELECT FOR UPDATE will wait for a concurrent transaction that has run any of those commands on the same row, and will then lock and return the updated row (or no row, if the row was deleted). Within a REPEATABLE READ or SERIALIZABLE transaction, however, an error will be thrown if a row to be locked has changed since the transaction started. For further discussion see Section 13.4.

The FOR UPDATE lock mode is also acquired by any DELETE on a row, and also by an UPDATE that modifies the values of certain columns. Currently, the set of columns considered for the UPDATE case are those that have a unique index on them that can be used in a foreign key (so partial indexes and expressional indexes are not considered), but this may change in the future.

FOR NO KEY UPDATE

Behaves similarly to FOR UPDATE, except that the lock acquired is weaker: this lock will not block SELECT FOR KEY SHARE commands that attempt to acquire a lock on the same rows. This lock mode is also acquired by any UPDATE that does not acquire a FOR UPDATE lock.

FOR SHARE

Behaves similarly to FOR NO KEY UPDATE, except that it acquires a shared lock rather than exclusive lock on each retrieved row. A shared lock blocks other transactions from performing UPDATE, DELETE, SELECT FOR UPDATE or SELECT FOR NO KEY UPDATE on these rows, but it does not prevent them from performing SELECT FOR SHARE or SELECT FOR KEY SHARE.

FOR KEY SHARE

Behaves similarly to FOR SHARE, except that the lock is weaker: SELECT FOR UPDATE is blocked, but not SELECT FOR NO KEY UPDATE. A key-shared lock blocks other transactions from performing DELETE or any UPDATE that changes the key values, but not other UPDATE, and neither does it prevent SELECT FOR NO KEY UPDATE, SELECT FOR SHARE, or SELECT FOR KEY SHARE.

PostgreSQL doesn't remember any information about modified rows in memory, so there is no limit on the number of rows locked at one time. However, locking a row might cause a disk write, e.g., SELECT FOR UPDATE modifies selected rows to mark them locked, and so will result in disk writes.

Table 13.3. Conflicting Row-Level Locks

Requested Lock Mode | Current Lock Mode
 | FOR KEY SHARE | FOR SHARE | FOR NO KEY UPDATE | FOR UPDATE
FOR KEY SHARE |  |  |  | X
FOR SHARE |  |  | X | X
FOR NO KEY UPDATE |  | X | X | X
FOR UPDATE | X | X | X | X

13.3.3. Page-Level Locks

In addition to table and row locks, page-level share/exclusive locks are used to control read/write access to table pages in the shared buffer pool. These locks are released immediately after a row is fetched or updated. Application developers normally need not be concerned with page-level locks, but they are mentioned here for completeness.

13.3.4. Deadlocks

The use of explicit locking can increase the likelihood of deadlocks, wherein two (or more) transactions each hold locks that the other wants. For example, if transaction 1 acquires an exclusive lock on table A and then tries to acquire an exclusive lock on table B, while transaction 2 has already exclusive-locked table B and now wants an exclusive lock on table A, then neither one can proceed. PostgreSQL automatically detects deadlock situations and resolves them by aborting one of the transactions involved, allowing the other(s) to complete. (Exactly which transaction will be aborted is difficult to predict and should not be relied upon.)

Note that deadlocks can also occur as the result of row-level locks (and thus, they can occur even if explicit locking is not used). Consider the case in which two concurrent transactions modify a table. The first transaction executes:

UPDATE accounts SET balance = balance + 100.00 WHERE acctnum = 11111;

This acquires a row-level lock on the row with the specified account number. Then, the second transaction executes:

UPDATE accounts SET balance = balance + 100.00 WHERE acctnum = 22222;
UPDATE accounts SET balance = balance - 100.00 WHERE acctnum = 11111;

The first UPDATE statement successfully acquires a row-level lock on the specified row, so it succeeds in updating that row. However, the second UPDATE statement finds that the row it is attempting to update has already been locked, so it waits for the transaction that acquired the lock to complete. Transaction two is now waiting on transaction one to complete before it continues execution. Now, transaction one executes:

UPDATE accounts SET balance = balance - 100.00 WHERE acctnum = 22222;

Transaction one attempts to acquire a row-level lock on the specified row, but it cannot: transaction two already holds such a lock. So it waits for transaction two to complete. Thus, transaction one is blocked on transaction two, and transaction two is blocked on transaction one: a deadlock condition. PostgreSQL will detect this situation and abort one of the transactions.

The best defense against deadlocks is generally to avoid them by being certain that all applications using a database acquire locks on multiple objects in a consistent order. In the example above, if both transactions had updated the rows in the same order, no deadlock would have occurred. One should also ensure that the first lock acquired on an object in a transaction is the most restrictive mode that will be needed for that object. If it is not feasible to verify this in advance, then deadlocks can be handled on-the-fly by retrying transactions that abort due to deadlocks.

So long as no deadlock situation is detected, a transaction seeking either a table-level or row-level lock will wait indefinitely for conflicting locks to be released. This means it is a bad idea for applications to hold transactions open for long periods of time (e.g., while waiting for user input).

13.3.5. Advisory Locks

PostgreSQL provides a means for creating locks that have application-defined meanings. These are called advisory locks, because the system does not enforce their use — it is up to the application to use them correctly. Advisory locks can be useful for locking strategies that are an awkward fit for the MVCC model. For example, a common use of advisory locks is to emulate pessimistic locking strategies typical of so-called “flat file” data management systems. While a flag stored in a table could be used for the same purpose, advisory locks are faster, avoid table bloat, and are automatically cleaned up by the server at the end of the session.

There are two ways to acquire an advisory lock in PostgreSQL: at session level or at transaction level. Once acquired at session level, an advisory lock is held until explicitly released or the session ends. Unlike standard lock requests, session-level advisory lock requests do not honor transaction semantics: a lock acquired during a transaction that is later rolled back will still be held following the rollback, and likewise an unlock is effective even if the calling transaction fails later. A lock can be acquired multiple times by its owning process; for each completed lock request there must be a corresponding unlock request before the lock is actually released. Transaction-level lock requests, on the other hand, behave more like regular lock requests: they are automatically released at the end of the transaction, and there is no explicit unlock operation. This behavior is often more convenient than the session-level behavior for short-term usage of an advisory lock. Session-level and transaction-level lock requests for the same advisory lock identifier will block each other in the expected way. If a session already holds a given advisory lock, additional requests by it will always succeed, even if other sessions are awaiting the lock; this statement is true regardless of whether the existing lock hold and new request are at session level or transaction level.

Like all locks in PostgreSQL, a complete list of advisory locks currently held by any session can be found in the pg_locks system view.

Both advisory locks and regular locks are stored in a shared memory pool whose size is defined by the configuration variables max_locks_per_transaction and max_connections. Care must be taken not to exhaust this memory or the server will be unable to grant any locks at all. This imposes an upper limit on the number of advisory locks grantable by the server, typically in the tens to hundreds of thousands depending on how the server is configured.

In certain cases using advisory locking methods, especially in queries involving explicit ordering and LIMIT clauses, care must be taken to control the locks acquired because of the order in which SQL expressions are evaluated. For example:

SELECT pg_advisory_lock(id) FROM foo WHERE id = 12345; -- ok
SELECT pg_advisory_lock(id) FROM foo WHERE id > 12345 LIMIT 100; -- danger!
SELECT pg_advisory_lock(q.id) FROM
(
  SELECT id FROM foo WHERE id > 12345 LIMIT 100
) q; -- ok

In the above queries, the second form is dangerous because the LIMIT is not guaranteed to be applied before the locking function is executed. This might cause some locks to be acquired that the application was not expecting, and hence would fail to release (until it ends the session). From the point of view of the application, such locks would be dangling, although still viewable in pg_locks.

The functions provided to manipulate advisory locks are described in Section 9.27.10.

Prev | Up | Next
13.2. Transaction Isolation | Home | 13.4. Data Consistency Checks at the Application Level


13.4. Data Consistency Checks at the Application Level
Prev Up Chapter 13. Concurrency Control Home Next

13.4. Data Consistency Checks at the Application Level

13.4.1. Enforcing Consistency with Serializable Transactions
13.4.2. Enforcing Consistency with Explicit Blocking Locks

It is very difficult to enforce business rules regarding data integrity using Read Committed transactions because the view of the data is shifting with each statement, and even a single statement may not restrict itself to the statement's snapshot if a write conflict occurs.

While a Repeatable Read transaction has a stable view of the data throughout its execution, there is a subtle issue with using MVCC snapshots for data consistency checks, involving something known as read/write conflicts. If one transaction writes data and a concurrent transaction attempts to read the same data (whether before or after the write), it cannot see the work of the other transaction. The reader then appears to have executed first regardless of which started first or which committed first. If that is as far as it goes, there is no problem, but if the reader also writes data which is read by a concurrent transaction there is now a transaction which appears to have run before either of the previously mentioned transactions. If the transaction which appears to have executed last actually commits first, it is very easy for a cycle to appear in a graph of the order of execution of the transactions. When such a cycle appears, integrity checks will not work correctly without some help.

As mentioned in Section 13.2.3, Serializable transactions are just Repeatable Read transactions which add nonblocking monitoring for dangerous patterns of read/write conflicts. When a pattern is detected which could cause a cycle in the apparent order of execution, one of the transactions involved is rolled back to break the cycle.

13.4.1. Enforcing Consistency with Serializable Transactions

If the Serializable transaction isolation level is used for all writes and for all reads which need a consistent view of the data, no other effort is required to ensure consistency. Software from other environments which is written to use serializable transactions to ensure consistency should “just work” in this regard in PostgreSQL.

When using this technique, it will avoid creating an unnecessary burden for application programmers if the application software goes through a framework which automatically retries transactions which are rolled back with a serialization failure. It may be a good idea to set default_transaction_isolation to serializable. It would also be wise to take some action to ensure that no other transaction isolation level is used, either inadvertently or to subvert integrity checks, through checks of the transaction isolation level in triggers.

See Section 13.2.3 for performance suggestions.

Warning

This level of integrity protection using Serializable transactions does not yet extend to hot standby mode (Section 27.4). Because of that, those using hot standby may want to use Repeatable Read and explicit locking on the primary.

13.4.2. Enforcing Consistency with Explicit Blocking Locks

When non-serializable writes are possible, to ensure the current validity of a row and protect it against concurrent updates one must use SELECT FOR UPDATE, SELECT FOR SHARE, or an appropriate LOCK TABLE statement. (SELECT FOR UPDATE and SELECT FOR SHARE lock just the returned rows against concurrent updates, while LOCK TABLE locks the whole table.) This should be taken into account when porting applications to PostgreSQL from other environments.

Also of note to those converting from other environments is the fact that SELECT FOR UPDATE does not ensure that a concurrent transaction will not update or delete a selected row. To do that in PostgreSQL you must actually update the row, even if no values need to be changed. SELECT FOR UPDATE temporarily blocks other transactions from acquiring the same lock or executing an UPDATE or DELETE which would affect the locked row, but once the transaction holding this lock commits or rolls back, a blocked transaction will proceed with the conflicting operation unless an actual UPDATE of the row was performed while the lock was held.

Global validity checks require extra thought under non-serializable MVCC. For example, a banking application might wish to check that the sum of all credits in one table equals the sum of debits in another table, when both tables are being actively updated. Comparing the results of two successive SELECT sum(...) commands will not work reliably in Read Committed mode, since the second query will likely include the results of transactions not counted by the first. Doing the two sums in a single repeatable read transaction will give an accurate picture of only the effects of transactions that committed before the repeatable read transaction started — but one might legitimately wonder whether the answer is still relevant by the time it is delivered. If the repeatable read transaction itself applied some changes before trying to make the consistency check, the usefulness of the check becomes even more debatable, since now it includes some but not all post-transaction-start changes. In such cases a careful person might wish to lock all tables needed for the check, in order to get an indisputable picture of current reality. A SHARE mode (or higher) lock guarantees that there are no uncommitted changes in the locked table, other than those of the current transaction.

Note also that if one is relying on explicit locking to prevent concurrent changes, one should either use Read Committed mode, or in Repeatable Read mode be careful to obtain locks before performing queries. A lock obtained by a repeatable read transaction guarantees that no other transactions modifying the table are still running, but if the snapshot seen by the transaction predates obtaining the lock, it might predate some now-committed changes in the table. A repeatable read transaction's snapshot is actually frozen at the start of its first query or data-modification command (SELECT, INSERT, UPDATE, or DELETE), so it is possible to obtain locks explicitly before the snapshot is frozen.

Prev | Up | Next
13.3. Explicit Locking | Home | 13.5. Caveats


13.5. Caveats
Prev Up Chapter 13. Concurrency Control Home Next

13.5. Caveats

Some DDL commands, currently only TRUNCATE and the table-rewriting forms of ALTER TABLE, are not MVCC-safe. This means that after the truncation or rewrite commits, the table will appear empty to concurrent transactions, if they are using a snapshot taken before the DDL command committed. This will only be an issue for a transaction that did not access the table in question before the DDL command started — any transaction that has done so would hold at least an ACCESS SHARE table lock, which would block the DDL command until that transaction completes. So these commands will not cause any apparent inconsistency in the table contents for successive queries on the target table, but they could cause visible inconsistency between the contents of the target table and other tables in the database.

Support for the Serializable transaction isolation level has not yet been added to Hot Standby replication targets (described in Section 27.4). The strictest isolation level currently supported in hot standby mode is Repeatable Read. While performing all permanent database writes within Serializable transactions on the primary will ensure that all standbys will eventually reach a consistent state, a Repeatable Read transaction run on the standby can sometimes see a transient state that is inconsistent with any serial execution of the transactions on the primary.

Internal access to the system catalogs is not done using the isolation level of the current transaction. This means that newly created database objects such as tables are visible to concurrent Repeatable Read and Serializable transactions, even though the rows they contain are not. In contrast, queries that explicitly examine the system catalogs don't see rows representing concurrently created database objects, in the higher isolation levels.

Prev | Up | Next
13.4. Data Consistency Checks at the Application Level | Home | 13.6. Locking and Indexes


13.6. Locking and Indexes
Prev Up Chapter 13. Concurrency Control Home Next

13.6. Locking and Indexes

Though PostgreSQL provides nonblocking read/write access to table data, nonblocking read/write access is not currently offered for every index access method implemented in PostgreSQL. The various index types are handled as follows:

B-tree, GiST and SP-GiST indexes

Short-term share/exclusive page-level locks are used for read/write access. Locks are released immediately after each index row is fetched or inserted. These index types provide the highest concurrency without deadlock conditions.

Hash indexes

Share/exclusive hash-bucket-level locks are used for read/write access. Locks are released after the whole bucket is processed. Bucket-level locks provide better concurrency than index-level ones, but deadlock is possible since the locks are held longer than one index operation.

GIN indexes

Short-term share/exclusive page-level locks are used for read/write access. Locks are released immediately after each index row is fetched or inserted. But note that insertion of a GIN-indexed value usually produces several index key insertions per row, so GIN might do substantial work for a single value's insertion.

Currently, B-tree indexes offer the best performance for concurrent applications; since they also have more features than hash indexes, they are the recommended index type for concurrent applications that need to index scalar data. When dealing with non-scalar data, B-trees are not useful, and GiST, SP-GiST or GIN indexes should be used instead.

Prev | Up | Next
13.5. Caveats | Home | Chapter 14. Performance Tips


Chapter 14. Performance Tips
Prev Up Part II. The SQL Language Home Next

Chapter 14. Performance Tips

Table of Contents

14.1. Using EXPLAIN
14.1.1. EXPLAIN Basics
14.1.2. EXPLAIN ANALYZE
14.1.3. Caveats
14.2. Statistics Used by the Planner
14.2.1. Single-Column Statistics
14.2.2. Extended Statistics
14.3. Controlling the Planner with Explicit JOIN Clauses
14.4. Populating a Database
14.4.1. Disable Autocommit
14.4.2. Use COPY
14.4.3. Remove Indexes
14.4.4. Remove Foreign Key Constraints
14.4.5. Increase maintenance_work_mem
14.4.6. Increase max_wal_size
14.4.7. Disable WAL Archival and Streaming Replication
14.4.8. Run ANALYZE Afterwards
14.4.9. Some Notes about pg_dump
14.5. Non-Durable Settings

Query performance can be affected by many things. Some of these can be controlled by the user, while others are fundamental to the underlying design of the system. This chapter provides some hints about understanding and tuning PostgreSQL performance.

Prev | Up | Next
13.6. Locking and Indexes | Home | 14.1. Using EXPLAIN


14.2. Statistics Used by the Planner
Prev Up Chapter 14. Performance Tips Home Next

14.2. Statistics Used by the Planner

14.2.1. Single-Column Statistics
14.2.2. Extended Statistics

14.2.1. Single-Column Statistics

As we saw in the previous section, the query planner needs to estimate the number of rows retrieved by a query in order to make good choices of query plans. This section provides a quick look at the statistics that the system uses for these estimates.

One component of the statistics is the total number of entries in each table and index, as well as the number of disk blocks occupied by each table and index. This information is kept in the table pg_class, in the columns reltuples and relpages. We can look at it with queries similar to this one:

SELECT relname, relkind, reltuples, relpages
FROM pg_class
WHERE relname LIKE 'tenk1%';

       relname        | relkind | reltuples | relpages
----------------------+---------+-----------+----------
 tenk1                | r       |     10000 |      358
 tenk1_hundred        | i       |     10000 |       30
 tenk1_thous_tenthous | i       |     10000 |       30
 tenk1_unique1        | i       |     10000 |       30
 tenk1_unique2        | i       |     10000 |       30
(5 rows)

Here we can see that tenk1 contains 10000 rows, as do its indexes, but the indexes are (unsurprisingly) much smaller than the table.

For efficiency reasons, reltuples and relpages are not updated on-the-fly, and so they usually contain somewhat out-of-date values. They are updated by VACUUM, ANALYZE, and a few DDL commands such as CREATE INDEX. A VACUUM or ANALYZE operation that does not scan the entire table (which is commonly the case) will incrementally update the reltuples count on the basis of the part of the table it did scan, resulting in an approximate value. In any case, the planner will scale the values it finds in pg_class to match the current physical table size, thus obtaining a closer approximation.

Most queries retrieve only a fraction of the rows in a table, due to WHERE clauses that restrict the rows to be examined. The planner thus needs to make an estimate of the selectivity of WHERE clauses, that is, the fraction of rows that match each condition in the WHERE clause. The information used for this task is stored in the pg_statistic system catalog. Entries in pg_statistic are updated by the ANALYZE and VACUUM ANALYZE commands, and are always approximate even when freshly updated.

Rather than look at pg_statistic directly, it's better to look at its view pg_stats when examining the statistics manually. pg_stats is designed to be more easily readable. Furthermore, pg_stats is readable by all, whereas pg_statistic is only readable by a superuser. (This prevents unprivileged users from learning something about the contents of other people's tables from the statistics. The pg_stats view is restricted to show only rows about tables that the current user can read.) For example, we might do:

SELECT attname, inherited, n_distinct,
       array_to_string(most_common_vals, E'\n') as most_common_vals
FROM pg_stats
WHERE tablename = 'road';

 attname | inherited | n_distinct |          most_common_vals
---------+-----------+------------+------------------------------------
 name    | f         |  -0.363388 | I- 580                        Ramp+
         |           |            | I- 880                        Ramp+
         |           |            | Sp Railroad                       +
         |           |            | I- 580                            +
         |           |            | I- 680                        Ramp
 name    | t         |  -0.284859 | I- 880                        Ramp+
         |           |            | I- 580                        Ramp+
         |           |            | I- 680                        Ramp+
         |           |            | I- 580                            +
         |           |            | State Hwy 13                  Ramp
(2 rows)

Note that two rows are displayed for the same column, one corresponding to the complete inheritance hierarchy starting at the road table (inherited=t), and another one including only the road table itself (inherited=f).

The amount of information stored in pg_statistic by ANALYZE, in particular the maximum number of entries in the most_common_vals and histogram_bounds arrays for each column, can be set on a column-by-column basis using the ALTER TABLE SET STATISTICS command, or globally by setting the default_statistics_target configuration variable. The default limit is presently 100 entries. Raising the limit might allow more accurate planner estimates to be made, particularly for columns with irregular data distributions, at the price of consuming more space in pg_statistic and slightly more time to compute the estimates. Conversely, a lower limit might be sufficient for columns with simple data distributions.

Further details about the planner's use of statistics can be found in Chapter 72.

14.2.2. Extended Statistics

It is common to see slow queries running bad execution plans because multiple columns used in the query clauses are correlated. The planner normally assumes that multiple conditions are independent of each other, an assumption that does not hold when column values are correlated. Regular statistics, because of their per-individual-column nature, cannot capture any knowledge about cross-column correlation. However, PostgreSQL has the ability to compute multivariate statistics, which can capture such information.

Because the number of possible column combinations is very large, it's impractical to compute multivariate statistics automatically. Instead, extended statistics objects, more often called just statistics objects, can be created to instruct the server to obtain statistics across interesting sets of columns.

Statistics objects are created using the CREATE STATISTICS command. Creation of such an object merely creates a catalog entry expressing interest in the statistics. Actual data collection is performed by ANALYZE (either a manual command, or background auto-analyze). The collected values can be examined in the pg_statistic_ext_data catalog.

ANALYZE computes extended statistics based on the same sample of table rows that it takes for computing regular single-column statistics. Since the sample size is increased by increasing the statistics target for the table or any of its columns (as described in the previous section), a larger statistics target will normally result in more accurate extended statistics, as well as more time spent calculating them.

The following subsections describe the kinds of extended statistics that are currently supported.

14.2.2.1. Functional Dependencies

The simplest kind of extended statistics tracks functional dependencies, a concept used in definitions of database normal forms. We say that column b is functionally dependent on column a if knowledge of the value of a is sufficient to determine the value of b, that is there are no two rows having the same value of a but different values of b. In a fully normalized database, functional dependencies should exist only on primary keys and superkeys. However, in practice many data sets are not fully normalized for various reasons; intentional denormalization for performance reasons is a common example. Even in a fully normalized database, there may be partial correlation between some columns, which can be expressed as partial functional dependency.

The existence of functional dependencies directly affects the accuracy of estimates in certain queries. If a query contains conditions on both the independent and the dependent column(s), the conditions on the dependent columns do not further reduce the result size; but without knowledge of the functional dependency, the query planner will assume that the conditions are independent, resulting in underestimating the result size.

To inform the planner about functional dependencies, ANALYZE can collect measurements of cross-column dependency. Assessing the degree of dependency between all sets of columns would be prohibitively expensive, so data collection is limited to those groups of columns appearing together in a statistics object defined with the dependencies option. It is advisable to create dependencies statistics only for column groups that are strongly correlated, to avoid unnecessary overhead in both ANALYZE and later query planning.

Here is an example of collecting functional-dependency statistics:

CREATE STATISTICS stts (dependencies) ON city, zip FROM zipcodes;

ANALYZE zipcodes;

SELECT stxname, stxkeys, stxddependencies
  FROM pg_statistic_ext join pg_statistic_ext_data on (oid = stxoid)
  WHERE stxname = 'stts';
 stxname | stxkeys |             stxddependencies
---------+---------+------------------------------------------
 stts    | 1 5     | {"1 => 5": 1.000000, "5 => 1": 0.423130}
(1 row)

Here it can be seen that column 1 (zip code) fully determines column 5 (city) so the coefficient is 1.0, while city only determines zip code about 42% of the time, meaning that there are many cities (58%) that are represented by more than a single ZIP code.

When computing the selectivity for a query involving functionally dependent columns, the planner adjusts the per-condition selectivity estimates using the dependency coefficients so as not to produce an underestimate.

14.2.2.1.1. Limitations of Functional Dependencies

Functional dependencies are currently only applied when considering simple equality conditions that compare columns to constant values, and IN clauses with constant values. They are not used to improve estimates for equality conditions comparing two columns or comparing a column to an expression, nor for range clauses, LIKE or any other type of condition.

When estimating with functional dependencies, the planner assumes that conditions on the involved columns are compatible and hence redundant. If they are incompatible, the correct estimate would be zero rows, but that possibility is not considered. For example, given a query like

SELECT * FROM zipcodes WHERE city = 'San Francisco' AND zip = '94105';

the planner will disregard the city clause as not changing the selectivity, which is correct. However, it will make the same assumption about

SELECT * FROM zipcodes WHERE city = 'San Francisco' AND zip = '90210';

even though there will really be zero rows satisfying this query. Functional dependency statistics do not provide enough information to conclude that, however.

In many practical situations, this assumption is usually satisfied; for example, there might be a GUI in the application that only allows selecting compatible city and ZIP code values to use in a query. But if that's not the case, functional dependencies may not be a viable option.

14.2.2.2. Multivariate N-Distinct Counts

Single-column statistics store the number of distinct values in each column. Estimates of the number of distinct values when combining more than one column (for example, for GROUP BY a, b) are frequently wrong when the planner only has single-column statistical data, causing it to select bad plans.

To improve such estimates, ANALYZE can collect n-distinct statistics for groups of columns. As before, it's impractical to do this for every possible column grouping, so data is collected only for those groups of columns appearing together in a statistics object defined with the ndistinct option. Data will be collected for each possible combination of two or more columns from the set of listed columns.

Continuing the previous example, the n-distinct counts in a table of ZIP codes might look like the following:

CREATE STATISTICS stts2 (ndistinct) ON city, state, zip FROM zipcodes;

ANALYZE zipcodes;

SELECT stxkeys AS k, stxdndistinct AS nd
  FROM pg_statistic_ext join pg_statistic_ext_data on (oid = stxoid)
  WHERE stxname = 'stts2';
-[ RECORD 1 ]------------------------------------------------------​--
k  | 1 2 5
nd | {"1, 2": 33178, "1, 5": 33178, "2, 5": 27435, "1, 2, 5": 33178}
(1 row)

This indicates that there are three combinations of columns that have 33178 distinct values: ZIP code and state; ZIP code and city; and ZIP code, city and state (the fact that they are all equal is expected given that ZIP code alone is unique in this table). On the other hand, the combination of city and state has only 27435 distinct values.

It's advisable to create ndistinct statistics objects only on combinations of columns that are actually used for grouping, and for which misestimation of the number of groups is resulting in bad plans. Otherwise, the ANALYZE cycles are just wasted.

14.2.2.3. Multivariate MCV Lists

Another type of statistic stored for each column are most-common value lists. This allows very accurate estimates for individual columns, but may result in significant misestimates for queries with conditions on multiple columns.

To improve such estimates, ANALYZE can collect MCV lists on combinations of columns. Similarly to functional dependencies and n-distinct coefficients, it's impractical to do this for every possible column grouping. Even more so in this case, as the MCV list (unlike functional dependencies and n-distinct coefficients) does store the common column values. So data is collected only for those groups of columns appearing together in a statistics object defined with the mcv option.

Continuing the previous example, the MCV list for a table of ZIP codes might look like the following (unlike for simpler types of statistics, a function is required for inspection of MCV contents):

CREATE STATISTICS stts3 (mcv) ON city, state FROM zipcodes;

ANALYZE zipcodes;

SELECT m.* FROM pg_statistic_ext join pg_statistic_ext_data on (oid = stxoid),
                pg_mcv_list_items(stxdmcv) m WHERE stxname = 'stts3';

 index |         values         | nulls | frequency | base_frequency
-------+------------------------+-------+-----------+----------------
     0 | {Washington, DC}       | {f,f} |  0.003467 |        2.7e-05
     1 | {Apo, AE}              | {f,f} |  0.003067 |        1.9e-05
     2 | {Houston, TX}          | {f,f} |  0.002167 |       0.000133
     3 | {El Paso, TX}          | {f,f} |     0.002 |       0.000113
     4 | {New York, NY}         | {f,f} |  0.001967 |       0.000114
     5 | {Atlanta, GA}          | {f,f} |  0.001633 |        3.3e-05
     6 | {Sacramento, CA}       | {f,f} |  0.001433 |        7.8e-05
     7 | {Miami, FL}            | {f,f} |    0.0014 |          6e-05
     8 | {Dallas, TX}           | {f,f} |  0.001367 |        8.8e-05
     9 | {Chicago, IL}          | {f,f} |  0.001333 |        5.1e-05
   ...
(99 rows)

This indicates that the most common combination of city and state is Washington in DC, with actual frequency (in the sample) about 0.35%. The base frequency of the combination (as computed from the simple per-column frequencies) is only 0.0027%, resulting in two orders of magnitude under-estimates.

It's advisable to create MCV statistics objects only on combinations of columns that are actually used in conditions together, and for which misestimation of the number of groups is resulting in bad plans. Otherwise, the ANALYZE and planning cycles are just wasted.

Prev | Up | Next
14.1. Using EXPLAIN | Home | 14.3. Controlling the Planner with Explicit JOIN Clauses


14.3. Controlling the Planner with Explicit JOIN Clauses
Prev Up Chapter 14. Performance Tips Home Next

14.3. Controlling the Planner with Explicit JOIN Clauses

It is possible to control the query planner to some extent by using the explicit JOIN syntax. To see why this matters, we first need some background.

In a simple join query, such as:

SELECT * FROM a, b, c WHERE a.id = b.id AND b.ref = c.id;

the planner is free to join the given tables in any order. For example, it could generate a query plan that joins A to B, using the WHERE condition a.id = b.id, and then joins C to this joined table, using the other WHERE condition. Or it could join B to C and then join A to that result. Or it could join A to C and then join them with B — but that would be inefficient, since the full Cartesian product of A and C would have to be formed, there being no applicable condition in the WHERE clause to allow optimization of the join. (All joins in the PostgreSQL executor happen between two input tables, so it's necessary to build up the result in one or another of these fashions.) The important point is that these different join possibilities give semantically equivalent results but might have hugely different execution costs. Therefore, the planner will explore all of them to try to find the most efficient query plan.

When a query only involves two or three tables, there aren't many join orders to worry about. But the number of possible join orders grows exponentially as the number of tables expands. Beyond ten or so input tables it's no longer practical to do an exhaustive search of all the possibilities, and even for six or seven tables planning might take an annoyingly long time. When there are too many input tables, the PostgreSQL planner will switch from exhaustive search to a genetic probabilistic search through a limited number of possibilities. (The switch-over threshold is set by the geqo_threshold run-time parameter.) The genetic search takes less time, but it won't necessarily find the best possible plan.

When the query involves outer joins, the planner has less freedom than it does for plain (inner) joins. For example, consider:

SELECT * FROM a LEFT JOIN (b JOIN c ON (b.ref = c.id)) ON (a.id = b.id);

Although this query's restrictions are superficially similar to the previous example, the semantics are different because a row must be emitted for each row of A that has no matching row in the join of B and C. Therefore the planner has no choice of join order here: it must join B to C and then join A to that result. Accordingly, this query takes less time to plan than the previous query. In other cases, the planner might be able to determine that more than one join order is safe. For example, given:

SELECT * FROM a LEFT JOIN b ON (a.bid = b.id) LEFT JOIN c ON (a.cid = c.id);

it is valid to join A to either B or C first. Currently, only FULL JOIN completely constrains the join order. Most practical cases involving LEFT JOIN or RIGHT JOIN can be rearranged to some extent.

Explicit inner join syntax (INNER JOIN, CROSS JOIN, or unadorned JOIN) is semantically the same as listing the input relations in FROM, so it does not constrain the join order.

Even though most kinds of JOIN don't completely constrain the join order, it is possible to instruct the PostgreSQL query planner to treat all JOIN clauses as constraining the join order anyway. For example, these three queries are logically equivalent:

SELECT * FROM a, b, c WHERE a.id = b.id AND b.ref = c.id;
SELECT * FROM a CROSS JOIN b CROSS JOIN c WHERE a.id = b.id AND b.ref = c.id;
SELECT * FROM a JOIN (b JOIN c ON (b.ref = c.id)) ON (a.id = b.id);

But if we tell the planner to honor the JOIN order, the second and third take less time to plan than the first. This effect is not worth worrying about for only three tables, but it can be a lifesaver with many tables.

To force the planner to follow the join order laid out by explicit JOINs, set the join_collapse_limit run-time parameter to 1. (Other possible values are discussed below.)

You do not need to constrain the join order completely in order to cut search time, because it's OK to use JOIN operators within items of a plain FROM list. For example, consider:

SELECT * FROM a CROSS JOIN b, c, d, e WHERE ...;

With join_collapse_limit = 1, this forces the planner to join A to B before joining them to other tables, but doesn't constrain its choices otherwise. In this example, the number of possible join orders is reduced by a factor of 5.

Constraining the planner's search in this way is a useful technique both for reducing planning time and for directing the planner to a good query plan. If the planner chooses a bad join order by default, you can force it to choose a better order via JOIN syntax — assuming that you know of a better order, that is. Experimentation is recommended.

A closely related issue that affects planning time is collapsing of subqueries into their parent query. For example, consider:

SELECT *
FROM x, y,
    (SELECT * FROM a, b, c WHERE something) AS ss
WHERE somethingelse;

This situation might arise from use of a view that contains a join; the view's SELECT rule will be inserted in place of the view reference, yielding a query much like the above. Normally, the planner will try to collapse the subquery into the parent, yielding:

SELECT * FROM x, y, a, b, c WHERE something AND somethingelse;

This usually results in a better plan than planning the subquery separately. (For example, the outer WHERE conditions might be such that joining X to A first eliminates many rows of A, thus avoiding the need to form the full logical output of the subquery.) But at the same time, we have increased the planning time; here, we have a five-way join problem replacing two separate three-way join problems. Because of the exponential growth of the number of possibilities, this makes a big difference. The planner tries to avoid getting stuck in huge join search problems by not collapsing a subquery if more than from_collapse_limit FROM items would result in the parent query. You can trade off planning time against quality of plan by adjusting this run-time parameter up or down.

from_collapse_limit and join_collapse_limit are similarly named because they do almost the same thing: one controls when the planner will “flatten out” subqueries, and the other controls when it will flatten out explicit joins. Typically you would either set join_collapse_limit equal to from_collapse_limit (so that explicit joins and subqueries act similarly) or set join_collapse_limit to 1 (if you want to control join order with explicit joins). But you might set them differently if you are trying to fine-tune the trade-off between planning time and run time.

Prev | Up | Next
14.2. Statistics Used by the Planner | Home | 14.4. Populating a Database


14.4. Populating a Database
Prev Up Chapter 14. Performance Tips Home Next

14.4. Populating a Database

14.4.1. Disable Autocommit
14.4.2. Use COPY
14.4.3. Remove Indexes
14.4.4. Remove Foreign Key Constraints
14.4.5. Increase maintenance_work_mem
14.4.6. Increase max_wal_size
14.4.7. Disable WAL Archival and Streaming Replication
14.4.8. Run ANALYZE Afterwards
14.4.9. Some Notes about pg_dump

One might need to insert a large amount of data when first populating a database. This section contains some suggestions on how to make this process as efficient as possible.

14.4.1. Disable Autocommit

When using multiple INSERTs, turn off autocommit and just do one commit at the end. (In plain SQL, this means issuing BEGIN at the start and COMMIT at the end. Some client libraries might do this behind your back, in which case you need to make sure the library does it when you want it done.) If you allow each insertion to be committed separately, PostgreSQL is doing a lot of work for each row that is added. An additional benefit of doing all insertions in one transaction is that if the insertion of one row were to fail then the insertion of all rows inserted up to that point would be rolled back, so you won't be stuck with partially loaded data.

14.4.2. Use COPY

Use COPY to load all the rows in one command, instead of using a series of INSERT commands. The COPY command is optimized for loading large numbers of rows; it is less flexible than INSERT, but incurs significantly less overhead for large data loads. Since COPY is a single command, there is no need to disable autocommit if you use this method to populate a table.

If you cannot use COPY, it might help to use PREPARE to create a prepared INSERT statement, and then use EXECUTE as many times as required. This avoids some of the overhead of repeatedly parsing and planning INSERT. Different interfaces provide this facility in different ways; look for “prepared statements” in the interface documentation.

Note that loading a large number of rows using COPY is almost always faster than using INSERT, even if PREPARE is used and multiple insertions are batched into a single transaction.

COPY is fastest when used within the same transaction as an earlier CREATE TABLE or TRUNCATE command. In such cases no WAL needs to be written, because in case of an error, the files containing the newly loaded data will be removed anyway. However, this consideration only applies when wal_level is minimal as all commands must write WAL otherwise.

14.4.3. Remove Indexes

If you are loading a freshly created table, the fastest method is to create the table, bulk load the table's data using COPY, then create any indexes needed for the table. Creating an index on pre-existing data is quicker than updating it incrementally as each row is loaded.

If you are adding large amounts of data to an existing table, it might be a win to drop the indexes, load the table, and then recreate the indexes. Of course, the database performance for other users might suffer during the time the indexes are missing. One should also think twice before dropping a unique index, since the error checking afforded by the unique constraint will be lost while the index is missing.

14.4.4. Remove Foreign Key Constraints

Just as with indexes, a foreign key constraint can be checked “in bulk” more efficiently than row-by-row. So it might be useful to drop foreign key constraints, load data, and re-create the constraints. Again, there is a trade-off between data load speed and loss of error checking while the constraint is missing.

What's more, when you load data into a table with existing foreign key constraints, each new row requires an entry in the server's list of pending trigger events (since it is the firing of a trigger that checks the row's foreign key constraint). Loading many millions of rows can cause the trigger event queue to overflow available memory, leading to intolerable swapping or even outright failure of the command. Therefore it may be necessary, not just desirable, to drop and re-apply foreign keys when loading large amounts of data. If temporarily removing the constraint isn't acceptable, the only other recourse may be to split up the load operation into smaller transactions.

14.4.5. Increase maintenance_work_mem

Temporarily increasing the maintenance_work_mem configuration variable when loading large amounts of data can lead to improved performance. This will help to speed up CREATE INDEX commands and ALTER TABLE ADD FOREIGN KEY commands. It won't do much for COPY itself, so this advice is only useful when you are using one or both of the above techniques.

14.4.6. Increase max_wal_size

Temporarily increasing the max_wal_size configuration variable can also make large data loads faster. This is because loading a large amount of data into PostgreSQL will cause checkpoints to occur more often than the normal checkpoint frequency (specified by the checkpoint_timeout configuration variable). Whenever a checkpoint occurs, all dirty pages must be flushed to disk. By increasing max_wal_size temporarily during bulk data loads, the number of checkpoints that are required can be reduced.

14.4.7. Disable WAL Archival and Streaming Replication

When loading large amounts of data into an installation that uses WAL archiving or streaming replication, it might be faster to take a new base backup after the load has completed than to process a large amount of incremental WAL data. To prevent incremental WAL logging while loading, disable archiving and streaming replication, by setting wal_level to minimal, archive_mode to off, and max_wal_senders to zero. But note that changing these settings requires a server restart, and makes any base backups taken before unavailable for archive recovery and standby server, which may lead to data loss.

Aside from avoiding the time for the archiver or WAL sender to process the WAL data, doing this will actually make certain commands faster, because they do not to write WAL at all if wal_level is minimal and the current subtransaction (or top-level transaction) created or truncated the table or index they change. (They can guarantee crash safety more cheaply by doing an fsync at the end than by writing WAL.)

14.4.8. Run ANALYZE Afterwards

Whenever you have significantly altered the distribution of data within a table, running ANALYZE is strongly recommended. This includes bulk loading large amounts of data into the table. Running ANALYZE (or VACUUM ANALYZE) ensures that the planner has up-to-date statistics about the table. With no statistics or obsolete statistics, the planner might make poor decisions during query planning, leading to poor performance on any tables with inaccurate or nonexistent statistics. Note that if the autovacuum daemon is enabled, it might run ANALYZE automatically; see Section 25.1.3 and Section 25.1.6 for more information.

14.4.9. Some Notes about pg_dump

Dump scripts generated by pg_dump automatically apply several, but not all, of the above guidelines. To reload a pg_dump dump as quickly as possible, you need to do a few extra things manually. (Note that these points apply while restoring a dump, not while creating it. The same points apply whether loading a text dump with psql or using pg_restore to load from a pg_dump archive file.)

By default, pg_dump uses COPY, and when it is generating a complete schema-and-data dump, it is careful to load data before creating indexes and foreign keys. So in this case several guidelines are handled automatically. What is left for you to do is to:

  • Set appropriate (i.e., larger than normal) values for maintenance_work_mem and max_wal_size.

  • If using WAL archiving or streaming replication, consider disabling them during the restore. To do that, set archive_mode to off, wal_level to minimal, and max_wal_senders to zero before loading the dump. Afterwards, set them back to the right values and take a fresh base backup.

  • Experiment with the parallel dump and restore modes of both pg_dump and pg_restore and find the optimal number of concurrent jobs to use. Dumping and restoring in parallel by means of the -j option should give you a significantly higher performance over the serial mode.

  • Consider whether the whole dump should be restored as a single transaction. To do that, pass the -1 or --single-transaction command-line option to psql or pg_restore. When using this mode, even the smallest of errors will rollback the entire restore, possibly discarding many hours of processing. Depending on how interrelated the data is, that might seem preferable to manual cleanup, or not. COPY commands will run fastest if you use a single transaction and have WAL archiving turned off.

  • If multiple CPUs are available in the database server, consider using pg_restore's --jobs option. This allows concurrent data loading and index creation.

  • Run ANALYZE afterwards.

A data-only dump will still use COPY, but it does not drop or recreate indexes, and it does not normally touch foreign keys. [14] So when loading a data-only dump, it is up to you to drop and recreate indexes and foreign keys if you wish to use those techniques. It's still useful to increase max_wal_size while loading the data, but don't bother increasing maintenance_work_mem; rather, you'd do that while manually recreating indexes and foreign keys afterwards. And don't forget to ANALYZE when you're done; see Section 25.1.3 and Section 25.1.6 for more information.

[14] You can get the effect of disabling foreign keys by using the --disable-triggers option — but realize that that eliminates, rather than just postpones, foreign key validation, and so it is possible to insert bad data if you use it.

Prev | Up | Next
14.3. Controlling the Planner with Explicit JOIN Clauses | Home | 14.5. Non-Durable Settings


14.5. Non-Durable Settings
Prev Up Chapter 14. Performance Tips Home Next

14.5. Non-Durable Settings

Durability is a database feature that guarantees the recording of committed transactions even if the server crashes or loses power. However, durability adds significant database overhead, so if your site does not require such a guarantee, PostgreSQL can be configured to run much faster. The following are configuration changes you can make to improve performance in such cases. Except as noted below, durability is still guaranteed in case of a crash of the database software; only an abrupt operating system crash creates a risk of data loss or corruption when these settings are used.

  • Place the database cluster's data directory in a memory-backed file system (i.e., RAM disk). This eliminates all database disk I/O, but limits data storage to the amount of available memory (and perhaps swap).

  • Turn off fsync; there is no need to flush data to disk.

  • Turn off synchronous_commit; there might be no need to force WAL writes to disk on every commit. This setting does risk transaction loss (though not data corruption) in case of a crash of the database.

  • Turn off full_page_writes; there is no need to guard against partial page writes.

  • Increase max_wal_size and checkpoint_timeout; this reduces the frequency of checkpoints, but increases the storage requirements of /pg_wal.

  • Create unlogged tables to avoid WAL writes, though it makes the tables non-crash-safe.

Prev | Up | Next
14.4. Populating a Database | Home | Chapter 15. Parallel Query


Chapter 15. Parallel Query
Prev Up Part II. The SQL Language Home Next

Chapter 15. Parallel Query

Table of Contents

15.1. How Parallel Query Works
15.2. When Can Parallel Query Be Used?
15.3. Parallel Plans
15.3.1. Parallel Scans
15.3.2. Parallel Joins
15.3.3. Parallel Aggregation
15.3.4. Parallel Append
15.3.5. Parallel Plan Tips
15.4. Parallel Safety
15.4.1. Parallel Labeling for Functions and Aggregates

PostgreSQL can devise query plans that can leverage multiple CPUs in order to answer queries faster. This feature is known as parallel query. Many queries cannot benefit from parallel query, either due to limitations of the current implementation or because there is no imaginable query plan that is any faster than the serial query plan. However, for queries that can benefit, the speedup from parallel query is often very significant. Many queries can run more than twice as fast when using parallel query, and some queries can run four times faster or even more. Queries that touch a large amount of data but return only a few rows to the user will typically benefit most. This chapter explains some details of how parallel query works and in which situations it can be used so that users who wish to make use of it can understand what to expect.

Prev | Up | Next
14.5. Non-Durable Settings | Home | 15.1. How Parallel Query Works


15.1. How Parallel Query Works
Prev Up Chapter 15. Parallel Query Home Next

15.1. How Parallel Query Works

When the optimizer determines that parallel query is the fastest execution strategy for a particular query, it will create a query plan that includes a Gather or Gather Merge node. Here is a simple example:

EXPLAIN SELECT * FROM pgbench_accounts WHERE filler LIKE '%x%';
                                     QUERY PLAN
-------------------------------------------------------------------​------------------
 Gather  (cost=1000.00..217018.43 rows=1 width=97)
   Workers Planned: 2
   ->  Parallel Seq Scan on pgbench_accounts  (cost=0.00..216018.33 rows=1 width=97)
         Filter: (filler ~~ '%x%'::text)
(4 rows)

In all cases, the Gather or Gather Merge node will have exactly one child plan, which is the portion of the plan that will be executed in parallel. If the Gather or Gather Merge node is at the very top of the plan tree, then the entire query will execute in parallel. If it is somewhere else in the plan tree, then only the portion of the plan below it will run in parallel. In the example above, the query accesses only one table, so there is only one plan node other than the Gather node itself; since that plan node is a child of the Gather node, it will run in parallel.

Using EXPLAIN, you can see the number of workers chosen by the planner. When the Gather node is reached during query execution, the process that is implementing the user's session will request a number of background worker processes equal to the number of workers chosen by the planner. The number of background workers that the planner will consider using is limited to at most max_parallel_workers_per_gather. The total number of background workers that can exist at any one time is limited by both max_worker_processes and max_parallel_workers. Therefore, it is possible for a parallel query to run with fewer workers than planned, or even with no workers at all. The optimal plan may depend on the number of workers that are available, so this can result in poor query performance. If this occurrence is frequent, consider increasing max_worker_processes and max_parallel_workers so that more workers can be run simultaneously or alternatively reducing max_parallel_workers_per_gather so that the planner requests fewer workers.

Every background worker process that is successfully started for a given parallel query will execute the parallel portion of the plan. The leader will also execute that portion of the plan, but it has an additional responsibility: it must also read all of the tuples generated by the workers. When the parallel portion of the plan generates only a small number of tuples, the leader will often behave very much like an additional worker, speeding up query execution. Conversely, when the parallel portion of the plan generates a large number of tuples, the leader may be almost entirely occupied with reading the tuples generated by the workers and performing any further processing steps that are required by plan nodes above the level of the Gather node or Gather Merge node. In such cases, the leader will do very little of the work of executing the parallel portion of the plan.

When the node at the top of the parallel portion of the plan is Gather Merge rather than Gather, it indicates that each process executing the parallel portion of the plan is producing tuples in sorted order, and that the leader is performing an order-preserving merge. In contrast, Gather reads tuples from the workers in whatever order is convenient, destroying any sort order that may have existed.

Prev | Up | Next
Chapter 15. Parallel Query | Home | 15.2. When Can Parallel Query Be Used?


15.3. Parallel Plans
Prev Up Chapter 15. Parallel Query Home Next

15.3. Parallel Plans

15.3.1. Parallel Scans
15.3.2. Parallel Joins
15.3.3. Parallel Aggregation
15.3.4. Parallel Append
15.3.5. Parallel Plan Tips

Because each worker executes the parallel portion of the plan to completion, it is not possible to simply take an ordinary query plan and run it using multiple workers. Each worker would produce a full copy of the output result set, so the query would not run any faster than normal but would produce incorrect results. Instead, the parallel portion of the plan must be what is known internally to the query optimizer as a partial plan; that is, it must be constructed so that each process that executes the plan will generate only a subset of the output rows in such a way that each required output row is guaranteed to be generated by exactly one of the cooperating processes. Generally, this means that the scan on the driving table of the query must be a parallel-aware scan.

15.3.1. Parallel Scans

The following types of parallel-aware table scans are currently supported.

  • In a parallel sequential scan, the table's blocks will be divided among the cooperating processes. Blocks are handed out one at a time, so that access to the table remains sequential.

  • In a parallel bitmap heap scan, one process is chosen as the leader. That process performs a scan of one or more indexes and builds a bitmap indicating which table blocks need to be visited. These blocks are then divided among the cooperating processes as in a parallel sequential scan. In other words, the heap scan is performed in parallel, but the underlying index scan is not.

  • In a parallel index scan or parallel index-only scan, the cooperating processes take turns reading data from the index. Currently, parallel index scans are supported only for btree indexes. Each process will claim a single index block and will scan and return all tuples referenced by that block; other processes can at the same time be returning tuples from a different index block. The results of a parallel btree scan are returned in sorted order within each worker process.

Other scan types, such as scans of non-btree indexes, may support parallel scans in the future.

15.3.2. Parallel Joins

Just as in a non-parallel plan, the driving table may be joined to one or more other tables using a nested loop, hash join, or merge join. The inner side of the join may be any kind of non-parallel plan that is otherwise supported by the planner provided that it is safe to run within a parallel worker. Depending on the join type, the inner side may also be a parallel plan.

  • In a nested loop join, the inner side is always non-parallel. Although it is executed in full, this is efficient if the inner side is an index scan, because the outer tuples and thus the loops that look up values in the index are divided over the cooperating processes.

  • In a merge join, the inner side is always a non-parallel plan and therefore executed in full. This may be inefficient, especially if a sort must be performed, because the work and resulting data are duplicated in every cooperating process.

  • In a hash join (without the "parallel" prefix), the inner side is executed in full by every cooperating process to build identical copies of the hash table. This may be inefficient if the hash table is large or the plan is expensive. In a parallel hash join, the inner side is a parallel hash that divides the work of building a shared hash table over the cooperating processes.

15.3.3. Parallel Aggregation

PostgreSQL supports parallel aggregation by aggregating in two stages. First, each process participating in the parallel portion of the query performs an aggregation step, producing a partial result for each group of which that process is aware. This is reflected in the plan as a Partial Aggregate node. Second, the partial results are transferred to the leader via Gather or Gather Merge. Finally, the leader re-aggregates the results across all workers in order to produce the final result. This is reflected in the plan as a Finalize Aggregate node.

Because the Finalize Aggregate node runs on the leader process, queries that produce a relatively large number of groups in comparison to the number of input rows will appear less favorable to the query planner. For example, in the worst-case scenario the number of groups seen by the Finalize Aggregate node could be as many as the number of input rows that were seen by all worker processes in the Partial Aggregate stage. For such cases, there is clearly going to be no performance benefit to using parallel aggregation. The query planner takes this into account during the planning process and is unlikely to choose parallel aggregate in this scenario.

Parallel aggregation is not supported in all situations. Each aggregate must be safe for parallelism and must have a combine function. If the aggregate has a transition state of type internal, it must have serialization and deserialization functions. See CREATE AGGREGATE for more details. Parallel aggregation is not supported if any aggregate function call contains DISTINCT or ORDER BY clause and is also not supported for ordered set aggregates or when the query involves GROUPING SETS. It can only be used when all joins involved in the query are also part of the parallel portion of the plan.

15.3.4. Parallel Append

Whenever PostgreSQL needs to combine rows from multiple sources into a single result set, it uses an Append or MergeAppend plan node. This commonly happens when implementing UNION ALL or when scanning a partitioned table. Such nodes can be used in parallel plans just as they can in any other plan. However, in a parallel plan, the planner may instead use a Parallel Append node.

When an Append node is used in a parallel plan, each process will execute the child plans in the order in which they appear, so that all participating processes cooperate to execute the first child plan until it is complete and then move to the second plan at around the same time. When a Parallel Append is used instead, the executor will instead spread out the participating processes as evenly as possible across its child plans, so that multiple child plans are executed simultaneously. This avoids contention, and also avoids paying the startup cost of a child plan in those processes that never execute it.

Also, unlike a regular Append node, which can only have partial children when used within a parallel plan, a Parallel Append node can have both partial and non-partial child plans. Non-partial children will be scanned by only a single process, since scanning them more than once would produce duplicate results. Plans that involve appending multiple results sets can therefore achieve coarse-grained parallelism even when efficient partial plans are not available. For example, consider a query against a partitioned table that can only be implemented efficiently by using an index that does not support parallel scans. The planner might choose a Parallel Append of regular Index Scan plans; each individual index scan would have to be executed to completion by a single process, but different scans could be performed at the same time by different processes.

enable_parallel_append can be used to disable this feature.

15.3.5. Parallel Plan Tips

If a query that is expected to do so does not produce a parallel plan, you can try reducing parallel_setup_cost or parallel_tuple_cost. Of course, this plan may turn out to be slower than the serial plan that the planner preferred, but this will not always be the case. If you don't get a parallel plan even with very small values of these settings (e.g., after setting them both to zero), there may be some reason why the query planner is unable to generate a parallel plan for your query. See Section 15.2 and Section 15.4 for information on why this may be the case.

When executing a parallel plan, you can use EXPLAIN (ANALYZE, VERBOSE) to display per-worker statistics for each plan node. This may be useful in determining whether the work is being evenly distributed between all plan nodes and more generally in understanding the performance characteristics of the plan.

Prev | Up | Next
15.2. When Can Parallel Query Be Used? | Home | 15.4. Parallel Safety


15.4. Parallel Safety
Prev Up Chapter 15. Parallel Query Home Next

15.4. Parallel Safety

15.4.1. Parallel Labeling for Functions and Aggregates

The planner classifies operations involved in a query as either parallel safe, parallel restricted, or parallel unsafe. A parallel safe operation is one that does not conflict with the use of parallel query. A parallel restricted operation is one that cannot be performed in a parallel worker, but that can be performed in the leader while parallel query is in use. Therefore, parallel restricted operations can never occur below a Gather or Gather Merge node, but can occur elsewhere in a plan that contains such a node. A parallel unsafe operation is one that cannot be performed while parallel query is in use, not even in the leader. When a query contains anything that is parallel unsafe, parallel query is completely disabled for that query.

The following operations are always parallel restricted:

  • Scans of common table expressions (CTEs).

  • Scans of temporary tables.

  • Scans of foreign tables, unless the foreign data wrapper has an IsForeignScanParallelSafe API that indicates otherwise.

  • Plan nodes to which an InitPlan is attached.

  • Plan nodes that reference a correlated SubPlan.

15.4.1. Parallel Labeling for Functions and Aggregates

The planner cannot automatically determine whether a user-defined function or aggregate is parallel safe, parallel restricted, or parallel unsafe, because this would require predicting every operation that the function could possibly perform. In general, this is equivalent to the Halting Problem and therefore impossible. Even for simple functions where it could conceivably be done, we do not try, since this would be expensive and error-prone. Instead, all user-defined functions are assumed to be parallel unsafe unless otherwise marked. When using CREATE FUNCTION or ALTER FUNCTION, markings can be set by specifying PARALLEL SAFE, PARALLEL RESTRICTED, or PARALLEL UNSAFE as appropriate. When using CREATE AGGREGATE, the PARALLEL option can be specified with SAFE, RESTRICTED, or UNSAFE as the corresponding value.

Functions and aggregates must be marked PARALLEL UNSAFE if they write to the database, access sequences, change the transaction state even temporarily (e.g., a PL/pgSQL function that establishes an EXCEPTION block to catch errors), or make persistent changes to settings. Similarly, functions must be marked PARALLEL RESTRICTED if they access temporary tables, client connection state, cursors, prepared statements, or miscellaneous backend-local state that the system cannot synchronize across workers. For example, setseed and random are parallel restricted for this last reason.

In general, if a function is labeled as being safe when it is restricted or unsafe, or if it is labeled as being restricted when it is in fact unsafe, it may throw errors or produce wrong answers when used in a parallel query. C-language functions could in theory exhibit totally undefined behavior if mislabeled, since there is no way for the system to protect itself against arbitrary C code, but in most likely cases the result will be no worse than for any other function. If in doubt, it is probably best to label functions as UNSAFE.

If a function executed within a parallel worker acquires locks that are not held by the leader, for example by querying a table not referenced in the query, those locks will be released at worker exit, not end of transaction. If you write a function that does this, and this behavior difference is important to you, mark such functions as PARALLEL RESTRICTED to ensure that they execute only in the leader.

Note that the query planner does not consider deferring the evaluation of parallel-restricted functions or aggregates involved in the query in order to obtain a superior plan. So, for example, if a WHERE clause applied to a particular table is parallel restricted, the query planner will not consider performing a scan of that table in the parallel portion of a plan. In some cases, it would be possible (and perhaps even efficient) to include the scan of that table in the parallel portion of the query and defer the evaluation of the WHERE clause so that it happens above the Gather node. However, the planner does not do this.

Prev | Up | Next
15.3. Parallel Plans | Home | Part III. Server Administration


Part III. Server Administration
Prev Up PostgreSQL 14.4 Documentation Home Next

Part III. Server Administration

This part covers topics that are of interest to a PostgreSQL database administrator. This includes installation of the software, set up and configuration of the server, management of users and databases, and maintenance tasks. Anyone who runs a PostgreSQL server, even for personal use, but especially in production, should be familiar with the topics covered in this part.

The information in this part is arranged approximately in the order in which a new user should read it. But the chapters are self-contained and can be read individually as desired. The information in this part is presented in a narrative fashion in topical units. Readers looking for a complete description of a particular command should see Part VI.

The first few chapters are written so they can be understood without prerequisite knowledge, so new users who need to set up their own server can begin their exploration with this part. The rest of this part is about tuning and management; that material assumes that the reader is familiar with the general use of the PostgreSQL database system. Readers are encouraged to look at Part I and Part II for additional information.

Table of Contents

16. Installation from Binaries
17. Installation from Source Code
17.1. Short Version
17.2. Requirements
17.3. Getting the Source
17.4. Installation Procedure
17.5. Post-Installation Setup
17.6. Supported Platforms
17.7. Platform-Specific Notes
18. Installation from Source Code on Windows
18.1. Building with Visual C++ or the Microsoft Windows SDK
19. Server Setup and Operation
19.1. The PostgreSQL User Account
19.2. Creating a Database Cluster
19.3. Starting the Database Server
19.4. Managing Kernel Resources
19.5. Shutting Down the Server
19.6. Upgrading a PostgreSQL Cluster
19.7. Preventing Server Spoofing
19.8. Encryption Options
19.9. Secure TCP/IP Connections with SSL
19.10. Secure TCP/IP Connections with GSSAPI Encryption
19.11. Secure TCP/IP Connections with SSH Tunnels
19.12. Registering Event Log on Windows
20. Server Configuration
20.1. Setting Parameters
20.2. File Locations
20.3. Connections and Authentication
20.4. Resource Consumption
20.5. Write Ahead Log
20.6. Replication
20.7. Query Planning
20.8. Error Reporting and Logging
20.9. Run-time Statistics
20.10. Automatic Vacuuming
20.11. Client Connection Defaults
20.12. Lock Management
20.13. Version and Platform Compatibility
20.14. Error Handling
20.15. Preset Options
20.16. Customized Options
20.17. Developer Options
20.18. Short Options
21. Client Authentication
21.1. The pg_hba.conf File
21.2. User Name Maps
21.3. Authentication Methods
21.4. Trust Authentication
21.5. Password Authentication
21.6. GSSAPI Authentication
21.7. SSPI Authentication
21.8. Ident Authentication
21.9. Peer Authentication
21.10. LDAP Authentication
21.11. RADIUS Authentication
21.12. Certificate Authentication
21.13. PAM Authentication
21.14. BSD Authentication
21.15. Authentication Problems
22. Database Roles
22.1. Database Roles
22.2. Role Attributes
22.3. Role Membership
22.4. Dropping Roles
22.5. Predefined Roles
22.6. Function Security
23. Managing Databases
23.1. Overview
23.2. Creating a Database
23.3. Template Databases
23.4. Database Configuration
23.5. Destroying a Database
23.6. Tablespaces
24. Localization
24.1. Locale Support
24.2. Collation Support
24.3. Character Set Support
25. Routine Database Maintenance Tasks
25.1. Routine Vacuuming
25.2. Routine Reindexing
25.3. Log File Maintenance
26. Backup and Restore
26.1. SQL Dump
26.2. File System Level Backup
26.3. Continuous Archiving and Point-in-Time Recovery (PITR)
27. High Availability, Load Balancing, and Replication
27.1. Comparison of Different Solutions
27.2. Log-Shipping Standby Servers
27.3. Failover
27.4. Hot Standby
28. Monitoring Database Activity
28.1. Standard Unix Tools
28.2. The Statistics Collector
28.3. Viewing Locks
28.4. Progress Reporting
28.5. Dynamic Tracing
29. Monitoring Disk Usage
29.1. Determining Disk Usage
29.2. Disk Full Failure
30. Reliability and the Write-Ahead Log
30.1. Reliability
30.2. Data Checksums
30.3. Write-Ahead Logging (WAL)
30.4. Asynchronous Commit
30.5. WAL Configuration
30.6. WAL Internals
31. Logical Replication
31.1. Publication
31.2. Subscription
31.3. Conflicts
31.4. Restrictions
31.5. Architecture
31.6. Monitoring
31.7. Security
31.8. Configuration Settings
31.9. Quick Setup
32. Just-in-Time Compilation (JIT)
32.1. What Is JIT compilation?
32.2. When to JIT?
32.3. Configuration
32.4. Extensibility
33. Regression Tests
33.1. Running the Tests
33.2. Test Evaluation
33.3. Variant Comparison Files
33.4. TAP Tests
33.5. Test Coverage Examination
Prev | Up | Next
15.4. Parallel Safety | Home | Chapter 16. Installation from Binaries


Chapter 16. Installation from Binaries
Prev Up Part III. Server Administration Home Next

Chapter 16. Installation from Binaries

PostgreSQL is available in the form of binary packages for most common operating systems today. When available, this is the recommended way to install PostgreSQL for users of the system. Building from source (see Chapter 17) is only recommended for people developing PostgreSQL or extensions.

For an updated list of platforms providing binary packages, please visit the download section on the PostgreSQL website at https://www.postgresql.org/download/ and follow the instructions for the specific platform.

Prev | Up | Next
Part III. Server Administration | Home | Chapter 17. Installation from Source Code


Chapter 17. Installation from Source Code
Prev Up Part III. Server Administration Home Next

Chapter 17. Installation from Source Code

Table of Contents

17.1. Short Version
17.2. Requirements
17.3. Getting the Source
17.4. Installation Procedure
17.4.1. configure Options
17.4.2. configure Environment Variables
17.5. Post-Installation Setup
17.5.1. Shared Libraries
17.5.2. Environment Variables
17.6. Supported Platforms
17.7. Platform-Specific Notes
17.7.1. AIX
17.7.2. Cygwin
17.7.3. macOS
17.7.4. MinGW/Native Windows
17.7.5. Solaris

This chapter describes the installation of PostgreSQL using the source code distribution. If you are installing a pre-packaged distribution, such as an RPM or Debian package, ignore this chapter and see Chapter 16 instead.

If you are building PostgreSQL for Microsoft Windows, read this chapter if you intend to build with MinGW or Cygwin; but if you intend to build with Microsoft's Visual C++, see Chapter 18 instead.

Prev | Up | Next
Chapter 16. Installation from Binaries | Home | 17.1. Short Version


17.1. Short Version
Prev Up Chapter 17. Installation from Source Code Home Next

17.1. Short Version

./configure
make
su
make install
adduser postgres
mkdir /usr/local/pgsql/data
chown postgres /usr/local/pgsql/data
su - postgres
/usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data
/usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l logfile start
/usr/local/pgsql/bin/createdb test
/usr/local/pgsql/bin/psql test

The long version is the rest of this chapter.

Prev | Up | Next
Chapter 17. Installation from Source Code | Home | 17.2. Requirements


17.2. Requirements
Prev Up Chapter 17. Installation from Source Code Home Next

17.2. Requirements

In general, a modern Unix-compatible platform should be able to run PostgreSQL. The platforms that had received specific testing at the time of release are described in Section 17.6 below.

The following software packages are required for building PostgreSQL:

  • GNU make version 3.80 or newer is required; other make programs or older GNU make versions will not work. (GNU make is sometimes installed under the name gmake.) To test for GNU make enter:

    make --version

  • You need an ISO/ANSI C compiler (at least C99-compliant). Recent versions of GCC are recommended, but PostgreSQL is known to build using a wide variety of compilers from different vendors.

  • tar is required to unpack the source distribution, in addition to either gzip or bzip2.

  • The GNU Readline library is used by default. It allows psql (the PostgreSQL command line SQL interpreter) to remember each command you type, and allows you to use arrow keys to recall and edit previous commands. This is very helpful and is strongly recommended. If you don't want to use it then you must specify the --without-readline option to configure. As an alternative, you can often use the BSD-licensed libedit library, originally developed on NetBSD. The libedit library is GNU Readline-compatible and is used if libreadline is not found, or if --with-libedit-preferred is used as an option to configure. If you are using a package-based Linux distribution, be aware that you need both the readline and readline-devel packages, if those are separate in your distribution.

  • The zlib compression library is used by default. If you don't want to use it then you must specify the --without-zlib option to configure. Using this option disables support for compressed archives in pg_dump and pg_restore.

The following packages are optional. They are not required in the default configuration, but they are needed when certain build options are enabled, as explained below:

  • To build the server programming language PL/Perl you need a full Perl installation, including the libperl library and the header files. The minimum required version is Perl 5.8.3. Since PL/Perl will be a shared library, the libperl library must be a shared library also on most platforms. This appears to be the default in recent Perl versions, but it was not in earlier versions, and in any case it is the choice of whomever installed Perl at your site. configure will fail if building PL/Perl is selected but it cannot find a shared libperl. In that case, you will have to rebuild and install Perl manually to be able to build PL/Perl. During the configuration process for Perl, request a shared library.

    If you intend to make more than incidental use of PL/Perl, you should ensure that the Perl installation was built with the usemultiplicity option enabled (perl -V will show whether this is the case).

  • To build the PL/Python server programming language, you need a Python installation with the header files and the sysconfig module. The minimum required version is Python 2.7. Python 3 is supported if it's version 3.2 or later; but see Section 46.1 when using Python 3.

    Since PL/Python will be a shared library, the libpython library must be a shared library also on most platforms. This is not the case in a default Python installation built from source, but a shared library is available in many operating system distributions. configure will fail if building PL/Python is selected but it cannot find a shared libpython. That might mean that you either have to install additional packages or rebuild (part of) your Python installation to provide this shared library. When building from source, run Python's configure with the --enable-shared flag.

  • To build the PL/Tcl procedural language, you of course need a Tcl installation. The minimum required version is Tcl 8.4.

  • To enable Native Language Support (NLS), that is, the ability to display a program's messages in a language other than English, you need an implementation of the Gettext API. Some operating systems have this built-in (e.g., Linux, NetBSD, Solaris), for other systems you can download an add-on package from https://www.gnu.org/software/gettext/. If you are using the Gettext implementation in the GNU C library then you will additionally need the GNU Gettext package for some utility programs. For any of the other implementations you will not need it.

  • You need OpenSSL, if you want to support encrypted client connections. OpenSSL is also required for random number generation on platforms that do not have /dev/urandom (except Windows). The minimum version required is 1.0.1.

  • You need Kerberos, OpenLDAP, and/or PAM, if you want to support authentication using those services.

  • You need LZ4, if you want to support compression of data with this method; see default_toast_compression.

  • To build the PostgreSQL documentation, there is a separate set of requirements; see Section J.2.

If you are building from a Git tree instead of using a released source package, or if you want to do server development, you also need the following packages:

  • Flex and Bison are needed to build from a Git checkout, or if you changed the actual scanner and parser definition files. If you need them, be sure to get Flex 2.5.31 or later and Bison 1.875 or later. Other lex and yacc programs cannot be used.

  • Perl 5.8.3 or later is needed to build from a Git checkout, or if you changed the input files for any of the build steps that use Perl scripts. If building on Windows you will need Perl in any case. Perl is also required to run some test suites.

If you need to get a GNU package, you can find it at your local GNU mirror site (see https://www.gnu.org/prep/ftp for a list) or at ftp://ftp.gnu.org/gnu/.

Also check that you have sufficient disk space. You will need about 350 MB for the source tree during compilation and about 60 MB for the installation directory. An empty database cluster takes about 40 MB; databases take about five times the amount of space that a flat text file with the same data would take. If you are going to run the regression tests you will temporarily need up to an extra 300 MB. Use the df command to check free disk space.

Prev | Up | Next
17.1. Short Version | Home | 17.3. Getting the Source


17.3. Getting the Source
Prev Up Chapter 17. Installation from Source Code Home Next

17.3. Getting the Source

The PostgreSQL 14.4 sources can be obtained from the download section of our website: https://www.postgresql.org/download/. You should get a file named postgresql-14.4.tar.gz or postgresql-14.4.tar.bz2. After you have obtained the file, unpack it:

gunzip postgresql-14.4.tar.gz
tar xf postgresql-14.4.tar

(Use bunzip2 instead of gunzip if you have the .bz2 file. Also, note that most modern versions of tar can unpack compressed archives directly, so you don't really need the separate gunzip or bunzip2 step.) This will create a directory postgresql-14.4 under the current directory with the PostgreSQL sources. Change into that directory for the rest of the installation procedure.

You can also get the source directly from the version control repository, see Appendix I.

Prev | Up | Next
17.2. Requirements | Home | 17.4. Installation Procedure


17.4. Installation Procedure
Prev Up Chapter 17. Installation from Source Code Home Next

17.4. Installation Procedure

17.4.1. configure Options
17.4.2. configure Environment Variables

  1. Configuration

    The first step of the installation procedure is to configure the source tree for your system and choose the options you would like. This is done by running the configure script. For a default installation simply enter:

    ./configure

    This script will run a number of tests to determine values for various system dependent variables and detect any quirks of your operating system, and finally will create several files in the build tree to record what it found.

    You can also run configure in a directory outside the source tree, and then build there, if you want to keep the build directory separate from the original source files. This procedure is called a VPATH build. Here's how:

    mkdir build_dir
cd build_dir
/path/to/source/tree/configure [options go here]
make

    The default configuration will build the server and utilities, as well as all client applications and interfaces that require only a C compiler. All files will be installed under /usr/local/pgsql by default.

    You can customize the build and installation process by supplying one or more command line options to configure. Typically you would customize the install location, or the set of optional features that are built. configure has a large number of options, which are described in Section 17.4.1.

    Also, configure responds to certain environment variables, as described in Section 17.4.2. These provide additional ways to customize the configuration.

  2. Build

    To start the build, type either of:

    make
make all

    (Remember to use GNU make.) The build will take a few minutes depending on your hardware.

    If you want to build everything that can be built, including the documentation (HTML and man pages), and the additional modules (contrib), type instead:

    make world

    If you want to build everything that can be built, including the additional modules (contrib), but without the documentation, type instead:

    make world-bin

    If you want to invoke the build from another makefile rather than manually, you must unset MAKELEVEL or set it to zero, for instance like this:

    build-postgresql:
        $(MAKE) -C postgresql MAKELEVEL=0 all

    Failure to do that can lead to strange error messages, typically about missing header files.

  3. Regression Tests

    If you want to test the newly built server before you install it, you can run the regression tests at this point. The regression tests are a test suite to verify that PostgreSQL runs on your machine in the way the developers expected it to. Type:

    make check

    (This won't work as root; do it as an unprivileged user.) See Chapter 33 for detailed information about interpreting the test results. You can repeat this test at any later time by issuing the same command.

  4. Installing the Files

    Note

    If you are upgrading an existing system be sure to read Section 19.6, which has instructions about upgrading a cluster.

    To install PostgreSQL enter:

    make install

    This will install files into the directories that were specified in Step 1. Make sure that you have appropriate permissions to write into that area. Normally you need to do this step as root. Alternatively, you can create the target directories in advance and arrange for appropriate permissions to be granted.

    To install the documentation (HTML and man pages), enter:

    make install-docs

    If you built the world above, type instead:

    make install-world

    This also installs the documentation.

    If you built the world without the documentation above, type instead:

    make install-world-bin

    You can use make install-strip instead of make install to strip the executable files and libraries as they are installed. This will save some space. If you built with debugging support, stripping will effectively remove the debugging support, so it should only be done if debugging is no longer needed. install-strip tries to do a reasonable job saving space, but it does not have perfect knowledge of how to strip every unneeded byte from an executable file, so if you want to save all the disk space you possibly can, you will have to do manual work.

    The standard installation provides all the header files needed for client application development as well as for server-side program development, such as custom functions or data types written in C.

    Client-only installation: If you want to install only the client applications and interface libraries, then you can use these commands:

    make -C src/bin install
make -C src/include install
make -C src/interfaces install
make -C doc install

    src/bin has a few binaries for server-only use, but they are small.

Uninstallation: To undo the installation use the command make uninstall. However, this will not remove any created directories.

Cleaning: After the installation you can free disk space by removing the built files from the source tree with the command make clean. This will preserve the files made by the configure program, so that you can rebuild everything with make later on. To reset the source tree to the state in which it was distributed, use make distclean. If you are going to build for several platforms within the same source tree you must do this and re-configure for each platform. (Alternatively, use a separate build tree for each platform, so that the source tree remains unmodified.)

If you perform a build and then discover that your configure options were wrong, or if you change anything that configure investigates (for example, software upgrades), then it's a good idea to do make distclean before reconfiguring and rebuilding. Without this, your changes in configuration choices might not propagate everywhere they need to.

17.4.1. configure Options

configure's command line options are explained below. This list is not exhaustive (use ./configure --help to get one that is). The options not covered here are meant for advanced use-cases such as cross-compilation, and are documented in the standard Autoconf documentation.

17.4.1.1. Installation Locations

These options control where make install will put the files. The --prefix option is sufficient for most cases. If you have special needs, you can customize the installation subdirectories with the other options described in this section. Beware however that changing the relative locations of the different subdirectories may render the installation non-relocatable, meaning you won't be able to move it after installation. (The man and doc locations are not affected by this restriction.) For relocatable installs, you might want to use the --disable-rpath option described later.

--prefix=PREFIX

Install all files under the directory PREFIX instead of /usr/local/pgsql. The actual files will be installed into various subdirectories; no files will ever be installed directly into the PREFIX directory.

--exec-prefix=EXEC-PREFIX

You can install architecture-dependent files under a different prefix, EXEC-PREFIX, than what PREFIX was set to. This can be useful to share architecture-independent files between hosts. If you omit this, then EXEC-PREFIX is set equal to PREFIX and both architecture-dependent and independent files will be installed under the same tree, which is probably what you want.

--bindir=DIRECTORY

Specifies the directory for executable programs. The default is EXEC-PREFIX/bin, which normally means /usr/local/pgsql/bin.

--sysconfdir=DIRECTORY

Sets the directory for various configuration files, PREFIX/etc by default.

--libdir=DIRECTORY

Sets the location to install libraries and dynamically loadable modules. The default is EXEC-PREFIX/lib.

--includedir=DIRECTORY

Sets the directory for installing C and C++ header files. The default is PREFIX/include.

--datarootdir=DIRECTORY

Sets the root directory for various types of read-only data files. This only sets the default for some of the following options. The default is PREFIX/share.

--datadir=DIRECTORY

Sets the directory for read-only data files used by the installed programs. The default is DATAROOTDIR. Note that this has nothing to do with where your database files will be placed.

--localedir=DIRECTORY

Sets the directory for installing locale data, in particular message translation catalog files. The default is DATAROOTDIR/locale.

--mandir=DIRECTORY

The man pages that come with PostgreSQL will be installed under this directory, in their respective manx subdirectories. The default is DATAROOTDIR/man.

--docdir=DIRECTORY

Sets the root directory for installing documentation files, except “man” pages. This only sets the default for the following options. The default value for this option is DATAROOTDIR/doc/postgresql.

--htmldir=DIRECTORY

The HTML-formatted documentation for PostgreSQL will be installed under this directory. The default is DATAROOTDIR.

Note

Care has been taken to make it possible to install PostgreSQL into shared installation locations (such as /usr/local/include) without interfering with the namespace of the rest of the system. First, the string “/postgresql” is automatically appended to datadir, sysconfdir, and docdir, unless the fully expanded directory name already contains the string “postgres” or “pgsql”. For example, if you choose /usr/local as prefix, the documentation will be installed in /usr/local/doc/postgresql, but if the prefix is /opt/postgres, then it will be in /opt/postgres/doc. The public C header files of the client interfaces are installed into includedir and are namespace-clean. The internal header files and the server header files are installed into private directories under includedir. See the documentation of each interface for information about how to access its header files. Finally, a private subdirectory will also be created, if appropriate, under libdir for dynamically loadable modules.

17.4.1.2. PostgreSQL Features

The options described in this section enable building of various PostgreSQL features that are not built by default. Most of these are non-default only because they require additional software, as described in Section 17.2.

--enable-nls[=LANGUAGES]

Enables Native Language Support (NLS), that is, the ability to display a program's messages in a language other than English. LANGUAGES is an optional space-separated list of codes of the languages that you want supported, for example --enable-nls='de fr'. (The intersection between your list and the set of actually provided translations will be computed automatically.) If you do not specify a list, then all available translations are installed.

To use this option, you will need an implementation of the Gettext API.

--with-perl

Build the PL/Perl server-side language.

--with-python

Build the PL/Python server-side language.

--with-tcl

Build the PL/Tcl server-side language.

--with-tclconfig=DIRECTORY

Tcl installs the file tclConfig.sh, which contains configuration information needed to build modules interfacing to Tcl. This file is normally found automatically at a well-known location, but if you want to use a different version of Tcl you can specify the directory in which to look for tclConfig.sh.

--with-icu

Build with support for the ICU library, enabling use of ICU collation features (see Section 24.2). This requires the ICU4C package to be installed. The minimum required version of ICU4C is currently 4.2.

By default, pkg-config will be used to find the required compilation options. This is supported for ICU4C version 4.6 and later. For older versions, or if pkg-config is not available, the variables ICU_CFLAGS and ICU_LIBS can be specified to configure, like in this example:

./configure ... --with-icu ICU_CFLAGS='-I/some/where/include' ICU_LIBS='-L/some/where/lib -licui18n -licuuc -licudata'

(If ICU4C is in the default search path for the compiler, then you still need to specify nonempty strings in order to avoid use of pkg-config, for example, ICU_CFLAGS=' '.)

--with-llvm

Build with support for LLVM based JIT compilation (see Chapter 32). This requires the LLVM library to be installed. The minimum required version of LLVM is currently 3.9.

llvm-config will be used to find the required compilation options. llvm-config, and then llvm-config-$major-$minor for all supported versions, will be searched for in your PATH. If that would not yield the desired program, use LLVM_CONFIG to specify a path to the correct llvm-config. For example

./configure ... --with-llvm LLVM_CONFIG='/path/to/llvm/bin/llvm-config'

LLVM support requires a compatible clang compiler (specified, if necessary, using the CLANG environment variable), and a working C++ compiler (specified, if necessary, using the CXX environment variable).

--with-lz4

Build with LZ4 compression support. This allows the use of LZ4 for compression of table data.

--with-ssl=LIBRARY

Build with support for SSL (encrypted) connections. The only LIBRARY supported is openssl. This requires the OpenSSL package to be installed. configure will check for the required header files and libraries to make sure that your OpenSSL installation is sufficient before proceeding.

--with-openssl

Obsolete equivalent of --with-ssl=openssl.

--with-gssapi

Build with support for GSSAPI authentication. On many systems, the GSSAPI system (usually a part of the Kerberos installation) is not installed in a location that is searched by default (e.g., /usr/include, /usr/lib), so you must use the options --with-includes and --with-libraries in addition to this option. configure will check for the required header files and libraries to make sure that your GSSAPI installation is sufficient before proceeding.

--with-ldap

Build with LDAP support for authentication and connection parameter lookup (see Section 34.18 and Section 21.10 for more information). On Unix, this requires the OpenLDAP package to be installed. On Windows, the default WinLDAP library is used. configure will check for the required header files and libraries to make sure that your OpenLDAP installation is sufficient before proceeding.

--with-pam

Build with PAM (Pluggable Authentication Modules) support.

--with-bsd-auth

Build with BSD Authentication support. (The BSD Authentication framework is currently only available on OpenBSD.)

--with-systemd

Build with support for systemd service notifications. This improves integration if the server is started under systemd but has no impact otherwise; see Section 19.3 for more information. libsystemd and the associated header files need to be installed to use this option.

--with-bonjour

Build with support for Bonjour automatic service discovery. This requires Bonjour support in your operating system. Recommended on macOS.

--with-uuid=LIBRARY

Build the uuid-ossp module (which provides functions to generate UUIDs), using the specified UUID library. LIBRARY must be one of:

  • bsd to use the UUID functions found in FreeBSD, NetBSD, and some other BSD-derived systems

  • e2fs to use the UUID library created by the e2fsprogs project; this library is present in most Linux systems and in macOS, and can be obtained for other platforms as well

  • ossp to use the OSSP UUID library

--with-ossp-uuid

Obsolete equivalent of --with-uuid=ossp.

--with-libxml

Build with libxml2, enabling SQL/XML support. Libxml2 version 2.6.23 or later is required for this feature.

To detect the required compiler and linker options, PostgreSQL will query pkg-config, if that is installed and knows about libxml2. Otherwise the program xml2-config, which is installed by libxml2, will be used if it is found. Use of pkg-config is preferred, because it can deal with multi-architecture installations better.

To use a libxml2 installation that is in an unusual location, you can set pkg-config-related environment variables (see its documentation), or set the environment variable XML2_CONFIG to point to the xml2-config program belonging to the libxml2 installation, or set the variables XML2_CFLAGS and XML2_LIBS. (If pkg-config is installed, then to override its idea of where libxml2 is you must either set XML2_CONFIG or set both XML2_CFLAGS and XML2_LIBS to nonempty strings.)

--with-libxslt

Build with libxslt, enabling the xml2 module to perform XSL transformations of XML. --with-libxml must be specified as well.

17.4.1.3. Anti-Features

The options described in this section allow disabling certain PostgreSQL features that are built by default, but which might need to be turned off if the required software or system features are not available. Using these options is not recommended unless really necessary.

--without-readline

Prevents use of the Readline library (and libedit as well). This option disables command-line editing and history in psql.

--with-libedit-preferred

Favors the use of the BSD-licensed libedit library rather than GPL-licensed Readline. This option is significant only if you have both libraries installed; the default in that case is to use Readline.

--without-zlib

Prevents use of the Zlib library. This disables support for compressed archives in pg_dump and pg_restore.

--disable-spinlocks

Allow the build to succeed even if PostgreSQL has no CPU spinlock support for the platform. The lack of spinlock support will result in very poor performance; therefore, this option should only be used if the build aborts and informs you that the platform lacks spinlock support. If this option is required to build PostgreSQL on your platform, please report the problem to the PostgreSQL developers.

--disable-atomics

Disable use of CPU atomic operations. This option does nothing on platforms that lack such operations. On platforms that do have them, this will result in poor performance. This option is only useful for debugging or making performance comparisons.

--disable-thread-safety

Disable the thread-safety of client libraries. This prevents concurrent threads in libpq and ECPG programs from safely controlling their private connection handles. Use this only on platforms with deficient threading support.

17.4.1.4. Build Process Details

--with-includes=DIRECTORIES

DIRECTORIES is a colon-separated list of directories that will be added to the list the compiler searches for header files. If you have optional packages (such as GNU Readline) installed in a non-standard location, you have to use this option and probably also the corresponding --with-libraries option.

Example: --with-includes=/opt/gnu/include:/usr/sup/include.

--with-libraries=DIRECTORIES

DIRECTORIES is a colon-separated list of directories to search for libraries. You will probably have to use this option (and the corresponding --with-includes option) if you have packages installed in non-standard locations.

Example: --with-libraries=/opt/gnu/lib:/usr/sup/lib.

--with-system-tzdata=DIRECTORY

PostgreSQL includes its own time zone database, which it requires for date and time operations. This time zone database is in fact compatible with the IANA time zone database provided by many operating systems such as FreeBSD, Linux, and Solaris, so it would be redundant to install it again. When this option is used, the system-supplied time zone database in DIRECTORY is used instead of the one included in the PostgreSQL source distribution. DIRECTORY must be specified as an absolute path. /usr/share/zoneinfo is a likely directory on some operating systems. Note that the installation routine will not detect mismatching or erroneous time zone data. If you use this option, you are advised to run the regression tests to verify that the time zone data you have pointed to works correctly with PostgreSQL.

This option is mainly aimed at binary package distributors who know their target operating system well. The main advantage of using this option is that the PostgreSQL package won't need to be upgraded whenever any of the many local daylight-saving time rules change. Another advantage is that PostgreSQL can be cross-compiled more straightforwardly if the time zone database files do not need to be built during the installation.

--with-extra-version=STRING

Append STRING to the PostgreSQL version number. You can use this, for example, to mark binaries built from unreleased Git snapshots or containing custom patches with an extra version string, such as a git describe identifier or a distribution package release number.

--disable-rpath

Do not mark PostgreSQL's executables to indicate that they should search for shared libraries in the installation's library directory (see --libdir). On most platforms, this marking uses an absolute path to the library directory, so that it will be unhelpful if you relocate the installation later. However, you will then need to provide some other way for the executables to find the shared libraries. Typically this requires configuring the operating system's dynamic linker to search the library directory; see Section 17.5.1 for more detail.

17.4.1.5. Miscellaneous

It's fairly common, particularly for test builds, to adjust the default port number with --with-pgport. The other options in this section are recommended only for advanced users.

--with-pgport=NUMBER

Set NUMBER as the default port number for server and clients. The default is 5432. The port can always be changed later on, but if you specify it here then both server and clients will have the same default compiled in, which can be very convenient. Usually the only good reason to select a non-default value is if you intend to run multiple PostgreSQL servers on the same machine.

--with-krb-srvnam=NAME

The default name of the Kerberos service principal used by GSSAPI. postgres is the default. There's usually no reason to change this unless you are building for a Windows environment, in which case it must be set to upper case POSTGRES.

--with-segsize=SEGSIZE

Set the segment size, in gigabytes. Large tables are divided into multiple operating-system files, each of size equal to the segment size. This avoids problems with file size limits that exist on many platforms. The default segment size, 1 gigabyte, is safe on all supported platforms. If your operating system has “largefile” support (which most do, nowadays), you can use a larger segment size. This can be helpful to reduce the number of file descriptors consumed when working with very large tables. But be careful not to select a value larger than is supported by your platform and the file systems you intend to use. Other tools you might wish to use, such as tar, could also set limits on the usable file size. It is recommended, though not absolutely required, that this value be a power of 2. Note that changing this value breaks on-disk database compatibility, meaning you cannot use pg_upgrade to upgrade to a build with a different segment size.

--with-blocksize=BLOCKSIZE

Set the block size, in kilobytes. This is the unit of storage and I/O within tables. The default, 8 kilobytes, is suitable for most situations; but other values may be useful in special cases. The value must be a power of 2 between 1 and 32 (kilobytes). Note that changing this value breaks on-disk database compatibility, meaning you cannot use pg_upgrade to upgrade to a build with a different block size.

--with-wal-blocksize=BLOCKSIZE

Set the WAL block size, in kilobytes. This is the unit of storage and I/O within the WAL log. The default, 8 kilobytes, is suitable for most situations; but other values may be useful in special cases. The value must be a power of 2 between 1 and 64 (kilobytes). Note that changing this value breaks on-disk database compatibility, meaning you cannot use pg_upgrade to upgrade to a build with a different WAL block size.

17.4.1.6. Developer Options

Most of the options in this section are only of interest for developing or debugging PostgreSQL. They are not recommended for production builds, except for --enable-debug, which can be useful to enable detailed bug reports in the unlucky event that you encounter a bug. On platforms supporting DTrace, --enable-dtrace may also be reasonable to use in production.

When building an installation that will be used to develop code inside the server, it is recommended to use at least the options --enable-debug and --enable-cassert.

--enable-debug

Compiles all programs and libraries with debugging symbols. This means that you can run the programs in a debugger to analyze problems. This enlarges the size of the installed executables considerably, and on non-GCC compilers it usually also disables compiler optimization, causing slowdowns. However, having the symbols available is extremely helpful for dealing with any problems that might arise. Currently, this option is recommended for production installations only if you use GCC. But you should always have it on if you are doing development work or running a beta version.

--enable-cassert

Enables assertion checks in the server, which test for many “cannot happen” conditions. This is invaluable for code development purposes, but the tests can slow down the server significantly. Also, having the tests turned on won't necessarily enhance the stability of your server! The assertion checks are not categorized for severity, and so what might be a relatively harmless bug will still lead to server restarts if it triggers an assertion failure. This option is not recommended for production use, but you should have it on for development work or when running a beta version.

--enable-tap-tests

Enable tests using the Perl TAP tools. This requires a Perl installation and the Perl module IPC::Run. See Section 33.4 for more information.

--enable-depend

Enables automatic dependency tracking. With this option, the makefiles are set up so that all affected object files will be rebuilt when any header file is changed. This is useful if you are doing development work, but is just wasted overhead if you intend only to compile once and install. At present, this option only works with GCC.

--enable-coverage

If using GCC, all programs and libraries are compiled with code coverage testing instrumentation. When run, they generate files in the build directory with code coverage metrics. See Section 33.5 for more information. This option is for use only with GCC and when doing development work.

--enable-profiling

If using GCC, all programs and libraries are compiled so they can be profiled. On backend exit, a subdirectory will be created that contains the gmon.out file containing profile data. This option is for use only with GCC and when doing development work.

--enable-dtrace

Compiles PostgreSQL with support for the dynamic tracing tool DTrace. See Section 28.5 for more information.

To point to the dtrace program, the environment variable DTRACE can be set. This will often be necessary because dtrace is typically installed under /usr/sbin, which might not be in your PATH.

Extra command-line options for the dtrace program can be specified in the environment variable DTRACEFLAGS. On Solaris, to include DTrace support in a 64-bit binary, you must specify DTRACEFLAGS="-64". For example, using the GCC compiler:

./configure CC='gcc -m64' --enable-dtrace DTRACEFLAGS='-64' ...

Using Sun's compiler:

./configure CC='/opt/SUNWspro/bin/cc -xtarget=native64' --enable-dtrace DTRACEFLAGS='-64' ...

17.4.2. configure Environment Variables

In addition to the ordinary command-line options described above, configure responds to a number of environment variables. You can specify environment variables on the configure command line, for example:

./configure CC=/opt/bin/gcc CFLAGS='-O2 -pipe'

In this usage an environment variable is little different from a command-line option. You can also set such variables beforehand:

export CC=/opt/bin/gcc
export CFLAGS='-O2 -pipe'
./configure

This usage can be convenient because many programs' configuration scripts respond to these variables in similar ways.

The most commonly used of these environment variables are CC and CFLAGS. If you prefer a C compiler different from the one configure picks, you can set the variable CC to the program of your choice. By default, configure will pick gcc if available, else the platform's default (usually cc). Similarly, you can override the default compiler flags if needed with the CFLAGS variable.

Here is a list of the significant variables that can be set in this manner:

BISON

Bison program

CC

C compiler

CFLAGS

options to pass to the C compiler

CLANG

path to clang program used to process source code for inlining when compiling with --with-llvm

CPP

C preprocessor

CPPFLAGS

options to pass to the C preprocessor

CXX

C++ compiler

CXXFLAGS

options to pass to the C++ compiler

DTRACE

location of the dtrace program

DTRACEFLAGS

options to pass to the dtrace program

FLEX

Flex program

LDFLAGS

options to use when linking either executables or shared libraries

LDFLAGS_EX

additional options for linking executables only

LDFLAGS_SL

additional options for linking shared libraries only

LLVM_CONFIG

llvm-config program used to locate the LLVM installation

MSGFMT

msgfmt program for native language support

PERL

Perl interpreter program. This will be used to determine the dependencies for building PL/Perl. The default is perl.

PYTHON

Python interpreter program. This will be used to determine the dependencies for building PL/Python. Also, whether Python 2 or 3 is specified here (or otherwise implicitly chosen) determines which variant of the PL/Python language becomes available. See Section 46.1 for more information. If this is not set, the following are probed in this order: python python3 python2.

TCLSH

Tcl interpreter program. This will be used to determine the dependencies for building PL/Tcl. If this is not set, the following are probed in this order: tclsh tcl tclsh8.6 tclsh86 tclsh8.5 tclsh85 tclsh8.4 tclsh84.

XML2_CONFIG

xml2-config program used to locate the libxml2 installation

Sometimes it is useful to add compiler flags after-the-fact to the set that were chosen by configure. An important example is that gcc's -Werror option cannot be included in the CFLAGS passed to configure, because it will break many of configure's built-in tests. To add such flags, include them in the COPT environment variable while running make. The contents of COPT are added to both the CFLAGS and LDFLAGS options set up by configure. For example, you could do

make COPT='-Werror'

or

export COPT='-Werror'
make

Note

If using GCC, it is best to build with an optimization level of at least -O1, because using no optimization (-O0) disables some important compiler warnings (such as the use of uninitialized variables). However, non-zero optimization levels can complicate debugging because stepping through compiled code will usually not match up one-to-one with source code lines. If you get confused while trying to debug optimized code, recompile the specific files of interest with -O0. An easy way to do this is by passing an option to make: make PROFILE=-O0 file.o.

The COPT and PROFILE environment variables are actually handled identically by the PostgreSQL makefiles. Which to use is a matter of preference, but a common habit among developers is to use PROFILE for one-time flag adjustments, while COPT might be kept set all the time.

Prev | Up | Next
17.3. Getting the Source | Home | 17.5. Post-Installation Setup


17.5. Post-Installation Setup
Prev Up Chapter 17. Installation from Source Code Home Next

17.5. Post-Installation Setup

17.5.1. Shared Libraries
17.5.2. Environment Variables

17.5.1. Shared Libraries

On some systems with shared libraries you need to tell the system how to find the newly installed shared libraries. The systems on which this is not necessary include FreeBSD, HP-UX, Linux, NetBSD, OpenBSD, and Solaris.

The method to set the shared library search path varies between platforms, but the most widely-used method is to set the environment variable LD_LIBRARY_PATH like so: In Bourne shells (sh, ksh, bash, zsh):

LD_LIBRARY_PATH=/usr/local/pgsql/lib
export LD_LIBRARY_PATH

or in csh or tcsh:

setenv LD_LIBRARY_PATH /usr/local/pgsql/lib

Replace /usr/local/pgsql/lib with whatever you set --libdir to in Step 1. You should put these commands into a shell start-up file such as /etc/profile or ~/.bash_profile. Some good information about the caveats associated with this method can be found at http://xahlee.info/UnixResource_dir/_/ldpath.html.

On some systems it might be preferable to set the environment variable LD_RUN_PATH before building.

On Cygwin, put the library directory in the PATH or move the .dll files into the bin directory.

If in doubt, refer to the manual pages of your system (perhaps ld.so or rld). If you later get a message like:

psql: error in loading shared libraries
libpq.so.2.1: cannot open shared object file: No such file or directory

then this step was necessary. Simply take care of it then.

If you are on Linux and you have root access, you can run:

/sbin/ldconfig /usr/local/pgsql/lib

(or equivalent directory) after installation to enable the run-time linker to find the shared libraries faster. Refer to the manual page of ldconfig for more information. On FreeBSD, NetBSD, and OpenBSD the command is:

/sbin/ldconfig -m /usr/local/pgsql/lib

instead. Other systems are not known to have an equivalent command.

17.5.2. Environment Variables

If you installed into /usr/local/pgsql or some other location that is not searched for programs by default, you should add /usr/local/pgsql/bin (or whatever you set --bindir to in Step 1) into your PATH. Strictly speaking, this is not necessary, but it will make the use of PostgreSQL much more convenient.

To do this, add the following to your shell start-up file, such as ~/.bash_profile (or /etc/profile, if you want it to affect all users):

PATH=/usr/local/pgsql/bin:$PATH
export PATH

If you are using csh or tcsh, then use this command:

set path = ( /usr/local/pgsql/bin $path )

To enable your system to find the man documentation, you need to add lines like the following to a shell start-up file unless you installed into a location that is searched by default:

MANPATH=/usr/local/pgsql/share/man:$MANPATH
export MANPATH

The environment variables PGHOST and PGPORT specify to client applications the host and port of the database server, overriding the compiled-in defaults. If you are going to run client applications remotely then it is convenient if every user that plans to use the database sets PGHOST. This is not required, however; the settings can be communicated via command line options to most client programs.

Prev | Up | Next
17.4. Installation Procedure | Home | 17.6. Supported Platforms


17.7. Platform-Specific Notes
Prev Up Chapter 17. Installation from Source Code Home Next

17.7. Platform-Specific Notes

17.7.1. AIX
17.7.2. Cygwin
17.7.3. macOS
17.7.4. MinGW/Native Windows
17.7.5. Solaris

This section documents additional platform-specific issues regarding the installation and setup of PostgreSQL. Be sure to read the installation instructions, and in particular Section 17.2 as well. Also, check Chapter 33 regarding the interpretation of regression test results.

Platforms that are not covered here have no known platform-specific installation issues.

17.7.1. AIX

PostgreSQL works on AIX, but AIX versions before about 6.1 have various issues and are not recommended. You can use GCC or the native IBM compiler xlc.

17.7.1.1. Memory Management

AIX can be somewhat peculiar with regards to the way it does memory management. You can have a server with many multiples of gigabytes of RAM free, but still get out of memory or address space errors when running applications. One example is loading of extensions failing with unusual errors. For example, running as the owner of the PostgreSQL installation:

=# CREATE EXTENSION plperl;
ERROR:  could not load library "/opt/dbs/pgsql/lib/plperl.so": A memory address is not in the address space for the process.

Running as a non-owner in the group possessing the PostgreSQL installation:

=# CREATE EXTENSION plperl;
ERROR:  could not load library "/opt/dbs/pgsql/lib/plperl.so": Bad address

Another example is out of memory errors in the PostgreSQL server logs, with every memory allocation near or greater than 256 MB failing.

The overall cause of all these problems is the default bittedness and memory model used by the server process. By default, all binaries built on AIX are 32-bit. This does not depend upon hardware type or kernel in use. These 32-bit processes are limited to 4 GB of memory laid out in 256 MB segments using one of a few models. The default allows for less than 256 MB in the heap as it shares a single segment with the stack.

In the case of the plperl example, above, check your umask and the permissions of the binaries in your PostgreSQL installation. The binaries involved in that example were 32-bit and installed as mode 750 instead of 755. Due to the permissions being set in this fashion, only the owner or a member of the possessing group can load the library. Since it isn't world-readable, the loader places the object into the process' heap instead of the shared library segments where it would otherwise be placed.

The “ideal” solution for this is to use a 64-bit build of PostgreSQL, but that is not always practical, because systems with 32-bit processors can build, but not run, 64-bit binaries.

If a 32-bit binary is desired, set LDR_CNTRL to MAXDATA=0xn0000000, where 1 <= n <= 8, before starting the PostgreSQL server, and try different values and postgresql.conf settings to find a configuration that works satisfactorily. This use of LDR_CNTRL tells AIX that you want the server to have MAXDATA bytes set aside for the heap, allocated in 256 MB segments. When you find a workable configuration, ldedit can be used to modify the binaries so that they default to using the desired heap size. PostgreSQL can also be rebuilt, passing configure LDFLAGS="-Wl,-bmaxdata:0xn0000000" to achieve the same effect.

For a 64-bit build, set OBJECT_MODE to 64 and pass CC="gcc -maix64" and LDFLAGS="-Wl,-bbigtoc" to configure. (Options for xlc might differ.) If you omit the export of OBJECT_MODE, your build may fail with linker errors. When OBJECT_MODE is set, it tells AIX's build utilities such as ar, as, and ld what type of objects to default to handling.

By default, overcommit of paging space can happen. While we have not seen this occur, AIX will kill processes when it runs out of memory and the overcommit is accessed. The closest to this that we have seen is fork failing because the system decided that there was not enough memory for another process. Like many other parts of AIX, the paging space allocation method and out-of-memory kill is configurable on a system- or process-wide basis if this becomes a problem.

17.7.2. Cygwin

PostgreSQL can be built using Cygwin, a Linux-like environment for Windows, but that method is inferior to the native Windows build (see Chapter 18) and running a server under Cygwin is no longer recommended.

When building from source, proceed according to the Unix-style installation procedure (i.e., ./configure; make; etc.), noting the following Cygwin-specific differences:

  • Set your path to use the Cygwin bin directory before the Windows utilities. This will help prevent problems with compilation.

  • The adduser command is not supported; use the appropriate user management application on Windows NT, 2000, or XP. Otherwise, skip this step.

  • The su command is not supported; use ssh to simulate su on Windows NT, 2000, or XP. Otherwise, skip this step.

  • OpenSSL is not supported.

  • Start cygserver for shared memory support. To do this, enter the command /usr/sbin/cygserver &. This program needs to be running anytime you start the PostgreSQL server or initialize a database cluster (initdb). The default cygserver configuration may need to be changed (e.g., increase SEMMNS) to prevent PostgreSQL from failing due to a lack of system resources.

  • Building might fail on some systems where a locale other than C is in use. To fix this, set the locale to C by doing export LANG=C.utf8 before building, and then setting it back to the previous setting after you have installed PostgreSQL.

  • The parallel regression tests (make check) can generate spurious regression test failures due to overflowing the listen() backlog queue which causes connection refused errors or hangs. You can limit the number of connections using the make variable MAX_CONNECTIONS thus:

    make MAX_CONNECTIONS=5 check

    (On some systems you can have up to about 10 simultaneous connections.)

It is possible to install cygserver and the PostgreSQL server as Windows NT services. For information on how to do this, please refer to the README document included with the PostgreSQL binary package on Cygwin. It is installed in the directory /usr/share/doc/Cygwin.

17.7.3. macOS

To build PostgreSQL from source on macOS, you will need to install Apple's command line developer tools, which can be done by issuing

xcode-select --install

(note that this will pop up a GUI dialog window for confirmation). You may or may not wish to also install Xcode.

On recent macOS releases, it's necessary to embed the “sysroot” path in the include switches used to find some system header files. This results in the outputs of the configure script varying depending on which SDK version was used during configure. That shouldn't pose any problem in simple scenarios, but if you are trying to do something like building an extension on a different machine than the server code was built on, you may need to force use of a different sysroot path. To do that, set PG_SYSROOT, for example

make PG_SYSROOT=/desired/path all

To find out the appropriate path on your machine, run

xcrun --show-sdk-path

Note that building an extension using a different sysroot version than was used to build the core server is not really recommended; in the worst case it could result in hard-to-debug ABI inconsistencies.

You can also select a non-default sysroot path when configuring, by specifying PG_SYSROOT to configure:

./configure ... PG_SYSROOT=/desired/path

This would primarily be useful to cross-compile for some other macOS version. There is no guarantee that the resulting executables will run on the current host.

To suppress the -isysroot options altogether, use

./configure ... PG_SYSROOT=none

(any nonexistent pathname will work). This might be useful if you wish to build with a non-Apple compiler, but beware that that case is not tested or supported by the PostgreSQL developers.

macOS's “System Integrity Protection” (SIP) feature breaks make check, because it prevents passing the needed setting of DYLD_LIBRARY_PATH down to the executables being tested. You can work around that by doing make install before make check. Most PostgreSQL developers just turn off SIP, though.

17.7.4. MinGW/Native Windows

PostgreSQL for Windows can be built using MinGW, a Unix-like build environment for Microsoft operating systems, or using Microsoft's Visual C++ compiler suite. The MinGW build procedure uses the normal build system described in this chapter; the Visual C++ build works completely differently and is described in Chapter 18.

The native Windows port requires a 32 or 64-bit version of Windows 2000 or later. Earlier operating systems do not have sufficient infrastructure (but Cygwin may be used on those). MinGW, the Unix-like build tools, and MSYS, a collection of Unix tools required to run shell scripts like configure, can be downloaded from http://www.mingw.org/. Neither is required to run the resulting binaries; they are needed only for creating the binaries.

To build 64 bit binaries using MinGW, install the 64 bit tool set from https://mingw-w64.org/, put its bin directory in the PATH, and run configure with the --host=x86_64-w64-mingw32 option.

After you have everything installed, it is suggested that you run psql under CMD.EXE, as the MSYS console has buffering issues.

17.7.4.1. Collecting Crash Dumps on Windows

If PostgreSQL on Windows crashes, it has the ability to generate minidumps that can be used to track down the cause for the crash, similar to core dumps on Unix. These dumps can be read using the Windows Debugger Tools or using Visual Studio. To enable the generation of dumps on Windows, create a subdirectory named crashdumps inside the cluster data directory. The dumps will then be written into this directory with a unique name based on the identifier of the crashing process and the current time of the crash.

17.7.5. Solaris

PostgreSQL is well-supported on Solaris. The more up to date your operating system, the fewer issues you will experience.

17.7.5.1. Required Tools

You can build with either GCC or Sun's compiler suite. For better code optimization, Sun's compiler is strongly recommended on the SPARC architecture. If you are using Sun's compiler, be careful not to select /usr/ucb/cc; use /opt/SUNWspro/bin/cc.

You can download Sun Studio from https://www.oracle.com/technetwork/server-storage/solarisstudio/downloads/. Many GNU tools are integrated into Solaris 10, or they are present on the Solaris companion CD. If you need packages for older versions of Solaris, you can find these tools at http://www.sunfreeware.com. If you prefer sources, look at https://www.gnu.org/prep/ftp.

17.7.5.2. configure Complains About a Failed Test Program

If configure complains about a failed test program, this is probably a case of the run-time linker being unable to find some library, probably libz, libreadline or some other non-standard library such as libssl. To point it to the right location, set the LDFLAGS environment variable on the configure command line, e.g.,

configure ... LDFLAGS="-R /usr/sfw/lib:/opt/sfw/lib:/usr/local/lib"

See the ld man page for more information.

17.7.5.3. Compiling for Optimal Performance

On the SPARC architecture, Sun Studio is strongly recommended for compilation. Try using the -xO5 optimization flag to generate significantly faster binaries. Do not use any flags that modify behavior of floating-point operations and errno processing (e.g., -fast).

If you do not have a reason to use 64-bit binaries on SPARC, prefer the 32-bit version. The 64-bit operations are slower and 64-bit binaries are slower than the 32-bit variants. On the other hand, 32-bit code on the AMD64 CPU family is not native, so 32-bit code is significantly slower on that CPU family.

17.7.5.4. Using DTrace for Tracing PostgreSQL

Yes, using DTrace is possible. See Section 28.5 for further information.

If you see the linking of the postgres executable abort with an error message like:

Undefined                       first referenced
 symbol                             in file
AbortTransaction                    utils/probes.o
CommitTransaction                   utils/probes.o
ld: fatal: Symbol referencing errors. No output written to postgres
collect2: ld returned 1 exit status
make: *** [postgres] Error 1

your DTrace installation is too old to handle probes in static functions. You need Solaris 10u4 or newer to use DTrace.

Prev | Up | Next
17.6. Supported Platforms | Home | Chapter 18. Installation from Source Code on Windows


Chapter 18. Installation from Source Code on Windows
Prev Up Part III. Server Administration Home Next

Chapter 18. Installation from Source Code on Windows

Table of Contents

18.1. Building with Visual C++ or the Microsoft Windows SDK
18.1.1. Requirements
18.1.2. Special Considerations for 64-Bit Windows
18.1.3. Building
18.1.4. Cleaning and Installing
18.1.5. Running the Regression Tests

It is recommended that most users download the binary distribution for Windows, available as a graphical installer package from the PostgreSQL website at https://www.postgresql.org/download/. Building from source is only intended for people developing PostgreSQL or extensions.

There are several different ways of building PostgreSQL on Windows. The simplest way to build with Microsoft tools is to install Visual Studio 2022 and use the included compiler. It is also possible to build with the full Microsoft Visual C++ 2013 to 2022. In some cases that requires the installation of the Windows SDK in addition to the compiler.

It is also possible to build PostgreSQL using the GNU compiler tools provided by MinGW, or using Cygwin for older versions of Windows.

Building using MinGW or Cygwin uses the normal build system, see Chapter 17 and the specific notes in Section 17.7.4 and Section 17.7.2. To produce native 64 bit binaries in these environments, use the tools from MinGW-w64. These tools can also be used to cross-compile for 32 bit and 64 bit Windows targets on other hosts, such as Linux and macOS. Cygwin is not recommended for running a production server, and it should only be used for running on older versions of Windows where the native build does not work. The official binaries are built using Visual Studio.

Native builds of psql don't support command line editing. The Cygwin build does support command line editing, so it should be used where psql is needed for interactive use on Windows.

Prev | Up | Next
17.7. Platform-Specific Notes | Home | 18.1. Building with Visual C++ or the Microsoft Windows SDK


18.1. Building with Visual C++ or the Microsoft Windows SDK
Prev Up Chapter 18. Installation from Source Code on Windows Home Next

18.1. Building with Visual C++ or the Microsoft Windows SDK

18.1.1. Requirements
18.1.2. Special Considerations for 64-Bit Windows
18.1.3. Building
18.1.4. Cleaning and Installing
18.1.5. Running the Regression Tests

PostgreSQL can be built using the Visual C++ compiler suite from Microsoft. These compilers can be either from Visual Studio, Visual Studio Express or some versions of the Microsoft Windows SDK. If you do not already have a Visual Studio environment set up, the easiest ways are to use the compilers from Visual Studio 2022 or those in the Windows SDK 10, which are both free downloads from Microsoft.

Both 32-bit and 64-bit builds are possible with the Microsoft Compiler suite. 32-bit PostgreSQL builds are possible with Visual Studio 2013 to Visual Studio 2022, as well as standalone Windows SDK releases 8.1a to 10. 64-bit PostgreSQL builds are supported with Microsoft Windows SDK version 8.1a to 10 or Visual Studio 2013 and above. Compilation is supported down to Windows 7 and Windows Server 2008 R2 SP1 when building with Visual Studio 2013 to Visual Studio 2022.

The tools for building using Visual C++ or Platform SDK are in the src\tools\msvc directory. When building, make sure there are no tools from MinGW or Cygwin present in your system PATH. Also, make sure you have all the required Visual C++ tools available in the PATH. In Visual Studio, start the Visual Studio Command Prompt. If you wish to build a 64-bit version, you must use the 64-bit version of the command, and vice versa. Starting with Visual Studio 2017 this can be done from the command line using VsDevCmd.bat, see -help for the available options and their default values. vsvars32.bat is available in Visual Studio 2015 and earlier versions for the same purpose. From the Visual Studio Command Prompt, you can change the targeted CPU architecture, build type, and target OS by using the vcvarsall.bat command, e.g., vcvarsall.bat x64 10.0.10240.0 to target Windows 10 with a 64-bit release build. See -help for the other options of vcvarsall.bat. All commands should be run from the src\tools\msvc directory.

Before you build, you may need to edit the file config.pl to reflect any configuration options you want to change, or the paths to any third party libraries to use. The complete configuration is determined by first reading and parsing the file config_default.pl, and then apply any changes from config.pl. For example, to specify the location of your Python installation, put the following in config.pl:

$config->{python} = 'c:\python26';

You only need to specify those parameters that are different from what's in config_default.pl.

If you need to set any other environment variables, create a file called buildenv.pl and put the required commands there. For example, to add the path for bison when it's not in the PATH, create a file containing:

$ENV{PATH}=$ENV{PATH} . ';c:\some\where\bison\bin';

To pass additional command line arguments to the Visual Studio build command (msbuild or vcbuild):

$ENV{MSBFLAGS}="/m";

18.1.1. Requirements

The following additional products are required to build PostgreSQL. Use the config.pl file to specify which directories the libraries are available in.

Microsoft Windows SDK

If your build environment doesn't ship with a supported version of the Microsoft Windows SDK it is recommended that you upgrade to the latest version (currently version 10), available for download from https://www.microsoft.com/download.

You must always include the Windows Headers and Libraries part of the SDK. If you install a Windows SDK including the Visual C++ Compilers, you don't need Visual Studio to build. Note that as of Version 8.0a the Windows SDK no longer ships with a complete command-line build environment.

ActiveState Perl

ActiveState Perl is required to run the build generation scripts. MinGW or Cygwin Perl will not work. It must also be present in the PATH. Binaries can be downloaded from https://www.activestate.com (Note: version 5.8.3 or later is required, the free Standard Distribution is sufficient).

The following additional products are not required to get started, but are required to build the complete package. Use the config.pl file to specify which directories the libraries are available in.

ActiveState TCL

Required for building PL/Tcl (Note: version 8.4 is required, the free Standard Distribution is sufficient).

Bison and Flex

Bison and Flex are required to build from Git, but not required when building from a release file. Only Bison 1.875 or versions 2.2 and later will work. Flex must be version 2.5.31 or later.

Both Bison and Flex are included in the msys tool suite, available from http://www.mingw.org/wiki/MSYS as part of the MinGW compiler suite.

You will need to add the directory containing flex.exe and bison.exe to the PATH environment variable in buildenv.pl unless they are already in PATH. In the case of MinGW, the directory is the \msys\1.0\bin subdirectory of your MinGW installation directory.

Note

The Bison distribution from GnuWin32 appears to have a bug that causes Bison to malfunction when installed in a directory with spaces in the name, such as the default location on English installations C:\Program Files\GnuWin32. Consider installing into C:\GnuWin32 or use the NTFS short name path to GnuWin32 in your PATH environment setting (e.g., C:\PROGRA~1\GnuWin32).

Diff

Diff is required to run the regression tests, and can be downloaded from http://gnuwin32.sourceforge.net.

Gettext

Gettext is required to build with NLS support, and can be downloaded from http://gnuwin32.sourceforge.net. Note that binaries, dependencies and developer files are all needed.

MIT Kerberos

Required for GSSAPI authentication support. MIT Kerberos can be downloaded from https://web.mit.edu/Kerberos/dist/index.html.

libxml2 and libxslt

Required for XML support. Binaries can be downloaded from https://zlatkovic.com/pub/libxml or source from http://xmlsoft.org. Note that libxml2 requires iconv, which is available from the same download location.

LZ4

Required for supporting LZ4 compression method for compressing the table data. Binaries and source can be downloaded from https://github.com/lz4/lz4/releases.

OpenSSL

Required for SSL support. Binaries can be downloaded from https://slproweb.com/products/Win32OpenSSL.html or source from https://www.openssl.org.

ossp-uuid

Required for UUID-OSSP support (contrib only). Source can be downloaded from http://www.ossp.org/pkg/lib/uuid/.

Python

Required for building PL/Python. Binaries can be downloaded from https://www.python.org.

zlib

Required for compression support in pg_dump and pg_restore. Binaries can be downloaded from https://www.zlib.net.

18.1.2. Special Considerations for 64-Bit Windows

PostgreSQL will only build for the x64 architecture on 64-bit Windows, there is no support for Itanium processors.

Mixing 32- and 64-bit versions in the same build tree is not supported. The build system will automatically detect if it's running in a 32- or 64-bit environment, and build PostgreSQL accordingly. For this reason, it is important to start the correct command prompt before building.

To use a server-side third party library such as python or OpenSSL, this library must also be 64-bit. There is no support for loading a 32-bit library in a 64-bit server. Several of the third party libraries that PostgreSQL supports may only be available in 32-bit versions, in which case they cannot be used with 64-bit PostgreSQL.

18.1.3. Building

To build all of PostgreSQL in release configuration (the default), run the command:

build

To build all of PostgreSQL in debug configuration, run the command:

build DEBUG

To build just a single project, for example psql, run the commands:

build psql
build DEBUG psql

To change the default build configuration to debug, put the following in the buildenv.pl file:

$ENV{CONFIG}="Debug";

It is also possible to build from inside the Visual Studio GUI. In this case, you need to run:

perl mkvcbuild.pl

from the command prompt, and then open the generated pgsql.sln (in the root directory of the source tree) in Visual Studio.

18.1.4. Cleaning and Installing

Most of the time, the automatic dependency tracking in Visual Studio will handle changed files. But if there have been large changes, you may need to clean the installation. To do this, simply run the clean.bat command, which will automatically clean out all generated files. You can also run it with the dist parameter, in which case it will behave like make distclean and remove the flex/bison output files as well.

By default, all files are written into a subdirectory of the debug or release directories. To install these files using the standard layout, and also generate the files required to initialize and use the database, run the command:

install c:\destination\directory

If you want to install only the client applications and interface libraries, then you can use these commands:

install c:\destination\directory client

18.1.5. Running the Regression Tests

To run the regression tests, make sure you have completed the build of all required parts first. Also, make sure that the DLLs required to load all parts of the system (such as the Perl and Python DLLs for the procedural languages) are present in the system path. If they are not, set it through the buildenv.pl file. To run the tests, run one of the following commands from the src\tools\msvc directory:

vcregress check
vcregress installcheck
vcregress plcheck
vcregress contribcheck
vcregress modulescheck
vcregress ecpgcheck
vcregress isolationcheck
vcregress bincheck
vcregress recoverycheck
vcregress upgradecheck

To change the schedule used (default is parallel), append it to the command line like:

vcregress check serial

For more information about the regression tests, see Chapter 33.

Running the regression tests on client programs, with vcregress bincheck, or on recovery tests, with vcregress recoverycheck, requires an additional Perl module to be installed:

IPC::Run

As of this writing, IPC::Run is not included in the ActiveState Perl installation, nor in the ActiveState Perl Package Manager (PPM) library. To install, download the IPC-Run-<version>.tar.gz source archive from CPAN, at https://metacpan.org/release/IPC-Run, and uncompress. Edit the buildenv.pl file, and add a PERL5LIB variable to point to the lib subdirectory from the extracted archive. For example:

$ENV{PERL5LIB}=$ENV{PERL5LIB} . ';c:\IPC-Run-0.94\lib';

The TAP tests run with vcregress support the environment variables PROVE_TESTS, that is expanded automatically using the name patterns given, and PROVE_FLAGS. These can be set on a Windows terminal, before running vcregress:

set PROVE_FLAGS=--timer --jobs 2
set PROVE_TESTS=t/020*.pl t/010*.pl

It is also possible to set up those parameters in buildenv.pl:

$ENV{PROVE_FLAGS}='--timer --jobs 2'
$ENV{PROVE_TESTS}='t/020*.pl t/010*.pl'

Some of the TAP tests depend on a set of external commands that would optionally trigger tests related to them. Each one of those variables can be set or unset in buildenv.pl:

GZIP_PROGRAM

Path to a gzip command. The default is gzip, that would be the command found in PATH.

LZ4

Path to a lz4 command. The default is lz4, that would be the command found in PATH.

TAR

Path to a tar command. The default is tar, that would be the command found in PATH.

Prev | Up | Next
Chapter 18. Installation from Source Code on Windows | Home | Chapter 19. Server Setup and Operation


19.1. The PostgreSQL User Account
Prev Up Chapter 19. Server Setup and Operation Home Next

19.1. The PostgreSQL User Account

As with any server daemon that is accessible to the outside world, it is advisable to run PostgreSQL under a separate user account. This user account should only own the data that is managed by the server, and should not be shared with other daemons. (For example, using the user nobody is a bad idea.) In particular, it is advisable that this user account not own the PostgreSQL executable files, to ensure that a compromised server process could not modify those executables.

Pre-packaged versions of PostgreSQL will typically create a suitable user account automatically during package installation.

To add a Unix user account to your system, look for a command useradd or adduser. The user name postgres is often used, and is assumed throughout this book, but you can use another name if you like.

Prev | Up | Next
Chapter 19. Server Setup and Operation | Home | 19.2. Creating a Database Cluster


19.2. Creating a Database Cluster
Prev Up Chapter 19. Server Setup and Operation Home Next

19.2. Creating a Database Cluster

19.2.1. Use of Secondary File Systems
19.2.2. File Systems

Before you can do anything, you must initialize a database storage area on disk. We call this a database cluster. (The SQL standard uses the term catalog cluster.) A database cluster is a collection of databases that is managed by a single instance of a running database server. After initialization, a database cluster will contain a database named postgres, which is meant as a default database for use by utilities, users and third party applications. The database server itself does not require the postgres database to exist, but many external utility programs assume it exists. Another database created within each cluster during initialization is called template1. As the name suggests, this will be used as a template for subsequently created databases; it should not be used for actual work. (See Chapter 23 for information about creating new databases within a cluster.)

In file system terms, a database cluster is a single directory under which all data will be stored. We call this the data directory or data area. It is completely up to you where you choose to store your data. There is no default, although locations such as /usr/local/pgsql/data or /var/lib/pgsql/data are popular. The data directory must be initialized before being used, using the program initdb which is installed with PostgreSQL.

If you are using a pre-packaged version of PostgreSQL, it may well have a specific convention for where to place the data directory, and it may also provide a script for creating the data directory. In that case you should use that script in preference to running initdb directly. Consult the package-level documentation for details.

To initialize a database cluster manually, run initdb and specify the desired file system location of the database cluster with the -D option, for example:

$ initdb -D /usr/local/pgsql/data

Note that you must execute this command while logged into the PostgreSQL user account, which is described in the previous section.

Tip

As an alternative to the -D option, you can set the environment variable PGDATA.

Alternatively, you can run initdb via the pg_ctl program like so:

$ pg_ctl -D /usr/local/pgsql/data initdb

This may be more intuitive if you are using pg_ctl for starting and stopping the server (see Section 19.3), so that pg_ctl would be the sole command you use for managing the database server instance.

initdb will attempt to create the directory you specify if it does not already exist. Of course, this will fail if initdb does not have permissions to write in the parent directory. It's generally recommendable that the PostgreSQL user own not just the data directory but its parent directory as well, so that this should not be a problem. If the desired parent directory doesn't exist either, you will need to create it first, using root privileges if the grandparent directory isn't writable. So the process might look like this:

root# mkdir /usr/local/pgsql
root# chown postgres /usr/local/pgsql
root# su postgres
postgres$ initdb -D /usr/local/pgsql/data

initdb will refuse to run if the data directory exists and already contains files; this is to prevent accidentally overwriting an existing installation.

Because the data directory contains all the data stored in the database, it is essential that it be secured from unauthorized access. initdb therefore revokes access permissions from everyone but the PostgreSQL user, and optionally, group. Group access, when enabled, is read-only. This allows an unprivileged user in the same group as the cluster owner to take a backup of the cluster data or perform other operations that only require read access.

Note that enabling or disabling group access on an existing cluster requires the cluster to be shut down and the appropriate mode to be set on all directories and files before restarting PostgreSQL. Otherwise, a mix of modes might exist in the data directory. For clusters that allow access only by the owner, the appropriate modes are 0700 for directories and 0600 for files. For clusters that also allow reads by the group, the appropriate modes are 0750 for directories and 0640 for files.

However, while the directory contents are secure, the default client authentication setup allows any local user to connect to the database and even become the database superuser. If you do not trust other local users, we recommend you use one of initdb's -W, --pwprompt or --pwfile options to assign a password to the database superuser. Also, specify -A scram-sha-256 so that the default trust authentication mode is not used; or modify the generated pg_hba.conf file after running initdb, but before you start the server for the first time. (Other reasonable approaches include using peer authentication or file system permissions to restrict connections. See Chapter 21 for more information.)

initdb also initializes the default locale for the database cluster. Normally, it will just take the locale settings in the environment and apply them to the initialized database. It is possible to specify a different locale for the database; more information about that can be found in Section 24.1. The default sort order used within the particular database cluster is set by initdb, and while you can create new databases using different sort order, the order used in the template databases that initdb creates cannot be changed without dropping and recreating them. There is also a performance impact for using locales other than C or POSIX. Therefore, it is important to make this choice correctly the first time.

initdb also sets the default character set encoding for the database cluster. Normally this should be chosen to match the locale setting. For details see Section 24.3.

Non-C and non-POSIX locales rely on the operating system's collation library for character set ordering. This controls the ordering of keys stored in indexes. For this reason, a cluster cannot switch to an incompatible collation library version, either through snapshot restore, binary streaming replication, a different operating system, or an operating system upgrade.

19.2.1. Use of Secondary File Systems

Many installations create their database clusters on file systems (volumes) other than the machine's “root” volume. If you choose to do this, it is not advisable to try to use the secondary volume's topmost directory (mount point) as the data directory. Best practice is to create a directory within the mount-point directory that is owned by the PostgreSQL user, and then create the data directory within that. This avoids permissions problems, particularly for operations such as pg_upgrade, and it also ensures clean failures if the secondary volume is taken offline.

19.2.2. File Systems

Generally, any file system with POSIX semantics can be used for PostgreSQL. Users prefer different file systems for a variety of reasons, including vendor support, performance, and familiarity. Experience suggests that, all other things being equal, one should not expect major performance or behavior changes merely from switching file systems or making minor file system configuration changes.

19.2.2.1. NFS

It is possible to use an NFS file system for storing the PostgreSQL data directory. PostgreSQL does nothing special for NFS file systems, meaning it assumes NFS behaves exactly like locally-connected drives. PostgreSQL does not use any functionality that is known to have nonstandard behavior on NFS, such as file locking.

The only firm requirement for using NFS with PostgreSQL is that the file system is mounted using the hard option. With the hard option, processes can “hang” indefinitely if there are network problems, so this configuration will require a careful monitoring setup. The soft option will interrupt system calls in case of network problems, but PostgreSQL will not repeat system calls interrupted in this way, so any such interruption will result in an I/O error being reported.

It is not necessary to use the sync mount option. The behavior of the async option is sufficient, since PostgreSQL issues fsync calls at appropriate times to flush the write caches. (This is analogous to how it works on a local file system.) However, it is strongly recommended to use the sync export option on the NFS server on systems where it exists (mainly Linux). Otherwise, an fsync or equivalent on the NFS client is not actually guaranteed to reach permanent storage on the server, which could cause corruption similar to running with the parameter fsync off. The defaults of these mount and export options differ between vendors and versions, so it is recommended to check and perhaps specify them explicitly in any case to avoid any ambiguity.

In some cases, an external storage product can be accessed either via NFS or a lower-level protocol such as iSCSI. In the latter case, the storage appears as a block device and any available file system can be created on it. That approach might relieve the DBA from having to deal with some of the idiosyncrasies of NFS, but of course the complexity of managing remote storage then happens at other levels.

Prev | Up | Next
19.1. The PostgreSQL User Account | Home | 19.3. Starting the Database Server


19.4. Managing Kernel Resources
Prev Up Chapter 19. Server Setup and Operation Home Next

19.4. Managing Kernel Resources

19.4.1. Shared Memory and Semaphores
19.4.2. systemd RemoveIPC
19.4.3. Resource Limits
19.4.4. Linux Memory Overcommit
19.4.5. Linux Huge Pages

PostgreSQL can sometimes exhaust various operating system resource limits, especially when multiple copies of the server are running on the same system, or in very large installations. This section explains the kernel resources used by PostgreSQL and the steps you can take to resolve problems related to kernel resource consumption.

19.4.1. Shared Memory and Semaphores

PostgreSQL requires the operating system to provide inter-process communication (IPC) features, specifically shared memory and semaphores. Unix-derived systems typically provide “System V” IPC, “POSIX” IPC, or both. Windows has its own implementation of these features and is not discussed here.

By default, PostgreSQL allocates a very small amount of System V shared memory, as well as a much larger amount of anonymous mmap shared memory. Alternatively, a single large System V shared memory region can be used (see shared_memory_type). In addition a significant number of semaphores, which can be either System V or POSIX style, are created at server startup. Currently, POSIX semaphores are used on Linux and FreeBSD systems while other platforms use System V semaphores.

System V IPC features are typically constrained by system-wide allocation limits. When PostgreSQL exceeds one of these limits, the server will refuse to start and should leave an instructive error message describing the problem and what to do about it. (See also Section 19.3.1.) The relevant kernel parameters are named consistently across different systems; Table 19.1 gives an overview. The methods to set them, however, vary. Suggestions for some platforms are given below.

Table 19.1. System V IPC Parameters

Name | Description | Values needed to run one PostgreSQL instance
SHMMAX | Maximum size of shared memory segment (bytes) | at least 1kB, but the default is usually much higher
SHMMIN | Minimum size of shared memory segment (bytes) | 1
SHMALL | Total amount of shared memory available (bytes or pages) | same as SHMMAX if bytes, or ceil(SHMMAX/PAGE_SIZE) if pages, plus room for other applications
SHMSEG | Maximum number of shared memory segments per process | only 1 segment is needed, but the default is much higher
SHMMNI | Maximum number of shared memory segments system-wide | like SHMSEG plus room for other applications
SEMMNI | Maximum number of semaphore identifiers (i.e., sets) | at least ceil((max_connections + autovacuum_max_workers + max_wal_senders + max_worker_processes + 5) / 16) plus room for other applications
SEMMNS | Maximum number of semaphores system-wide | ceil((max_connections + autovacuum_max_workers + max_wal_senders + max_worker_processes + 5) / 16) * 17 plus room for other applications
SEMMSL | Maximum number of semaphores per set | at least 17
SEMMAP | Number of entries in semaphore map | see text
SEMVMX | Maximum value of semaphore | at least 1000 (The default is often 32767; do not change unless necessary)

PostgreSQL requires a few bytes of System V shared memory (typically 48 bytes, on 64-bit platforms) for each copy of the server. On most modern operating systems, this amount can easily be allocated. However, if you are running many copies of the server or you explicitly configure the server to use large amounts of System V shared memory (see shared_memory_type and dynamic_shared_memory_type), it may be necessary to increase SHMALL, which is the total amount of System V shared memory system-wide. Note that SHMALL is measured in pages rather than bytes on many systems.

Less likely to cause problems is the minimum size for shared memory segments (SHMMIN), which should be at most approximately 32 bytes for PostgreSQL (it is usually just 1). The maximum number of segments system-wide (SHMMNI) or per-process (SHMSEG) are unlikely to cause a problem unless your system has them set to zero.

When using System V semaphores, PostgreSQL uses one semaphore per allowed connection (max_connections), allowed autovacuum worker process (autovacuum_max_workers) and allowed background process (max_worker_processes), in sets of 16. Each such set will also contain a 17th semaphore which contains a “magic number”, to detect collision with semaphore sets used by other applications. The maximum number of semaphores in the system is set by SEMMNS, which consequently must be at least as high as max_connections plus autovacuum_max_workers plus max_wal_senders, plus max_worker_processes, plus one extra for each 16 allowed connections plus workers (see the formula in Table 19.1). The parameter SEMMNI determines the limit on the number of semaphore sets that can exist on the system at one time. Hence this parameter must be at least ceil((max_connections + autovacuum_max_workers + max_wal_senders + max_worker_processes + 5) / 16). Lowering the number of allowed connections is a temporary workaround for failures, which are usually confusingly worded “No space left on device”, from the function semget.

In some cases it might also be necessary to increase SEMMAP to be at least on the order of SEMMNS. If the system has this parameter (many do not), it defines the size of the semaphore resource map, in which each contiguous block of available semaphores needs an entry. When a semaphore set is freed it is either added to an existing entry that is adjacent to the freed block or it is registered under a new map entry. If the map is full, the freed semaphores get lost (until reboot). Fragmentation of the semaphore space could over time lead to fewer available semaphores than there should be.

Various other settings related to “semaphore undo”, such as SEMMNU and SEMUME, do not affect PostgreSQL.

When using POSIX semaphores, the number of semaphores needed is the same as for System V, that is one semaphore per allowed connection (max_connections), allowed autovacuum worker process (autovacuum_max_workers) and allowed background process (max_worker_processes). On the platforms where this option is preferred, there is no specific kernel limit on the number of POSIX semaphores.

AIX

It should not be necessary to do any special configuration for such parameters as SHMMAX, as it appears this is configured to allow all memory to be used as shared memory. That is the sort of configuration commonly used for other databases such as DB/2.

It might, however, be necessary to modify the global ulimit information in /etc/security/limits, as the default hard limits for file sizes (fsize) and numbers of files (nofiles) might be too low.

FreeBSD

The default shared memory settings are usually good enough, unless you have set shared_memory_type to sysv. System V semaphores are not used on this platform.

The default IPC settings can be changed using the sysctl or loader interfaces. The following parameters can be set using sysctl:

# sysctl kern.ipc.shmall=32768
# sysctl kern.ipc.shmmax=134217728

To make these settings persist over reboots, modify /etc/sysctl.conf.

If you have set shared_memory_type to sysv, you might also want to configure your kernel to lock System V shared memory into RAM and prevent it from being paged out to swap. This can be accomplished using the sysctl setting kern.ipc.shm_use_phys.

If running in a FreeBSD jail, you should set its sysvshm parameter to new, so that it has its own separate System V shared memory namespace. (Before FreeBSD 11.0, it was necessary to enable shared access to the host's IPC namespace from jails, and take measures to avoid collisions.)

NetBSD

The default shared memory settings are usually good enough, unless you have set shared_memory_type to sysv. You will usually want to increase kern.ipc.semmni and kern.ipc.semmns, as NetBSD's default settings for these are uncomfortably small.

IPC parameters can be adjusted using sysctl, for example:

# sysctl -w kern.ipc.semmni=100

To make these settings persist over reboots, modify /etc/sysctl.conf.

If you have set shared_memory_type to sysv, you might also want to configure your kernel to lock System V shared memory into RAM and prevent it from being paged out to swap. This can be accomplished using the sysctl setting kern.ipc.shm_use_phys.

OpenBSD

The default shared memory settings are usually good enough, unless you have set shared_memory_type to sysv. You will usually want to increase kern.seminfo.semmni and kern.seminfo.semmns, as OpenBSD's default settings for these are uncomfortably small.

IPC parameters can be adjusted using sysctl, for example:

# sysctl kern.seminfo.semmni=100

To make these settings persist over reboots, modify /etc/sysctl.conf.

HP-UX

The default settings tend to suffice for normal installations.

IPC parameters can be set in the System Administration Manager (SAM) under Kernel Configuration → Configurable Parameters. Choose Create A New Kernel when you're done.

Linux

The default shared memory settings are usually good enough, unless you have set shared_memory_type to sysv, and even then only on older kernel versions that shipped with low defaults. System V semaphores are not used on this platform.

The shared memory size settings can be changed via the sysctl interface. For example, to allow 16 GB:

$ sysctl -w kernel.shmmax=17179869184
$ sysctl -w kernel.shmall=4194304

To make these settings persist over reboots, see /etc/sysctl.conf.

macOS

The default shared memory and semaphore settings are usually good enough, unless you have set shared_memory_type to sysv.

The recommended method for configuring shared memory in macOS is to create a file named /etc/sysctl.conf, containing variable assignments such as:

kern.sysv.shmmax=4194304
kern.sysv.shmmin=1
kern.sysv.shmmni=32
kern.sysv.shmseg=8
kern.sysv.shmall=1024

Note that in some macOS versions, all five shared-memory parameters must be set in /etc/sysctl.conf, else the values will be ignored.

SHMMAX can only be set to a multiple of 4096.

SHMALL is measured in 4 kB pages on this platform.

It is possible to change all but SHMMNI on the fly, using sysctl. But it's still best to set up your preferred values via /etc/sysctl.conf, so that the values will be kept across reboots.

Solaris
illumos

The default shared memory and semaphore settings are usually good enough for most PostgreSQL applications. Solaris defaults to a SHMMAX of one-quarter of system RAM. To further adjust this setting, use a project setting associated with the postgres user. For example, run the following as root:

projadd -c "PostgreSQL DB User" -K "project.max-shm-memory=(privileged,8GB,deny)" -U postgres -G postgres user.postgres

This command adds the user.postgres project and sets the shared memory maximum for the postgres user to 8GB, and takes effect the next time that user logs in, or when you restart PostgreSQL (not reload). The above assumes that PostgreSQL is run by the postgres user in the postgres group. No server reboot is required.

Other recommended kernel setting changes for database servers which will have a large number of connections are:

project.max-shm-ids=(priv,32768,deny)
project.max-sem-ids=(priv,4096,deny)
project.max-msg-ids=(priv,4096,deny)

Additionally, if you are running PostgreSQL inside a zone, you may need to raise the zone resource usage limits as well. See "Chapter2: Projects and Tasks" in the System Administrator's Guide for more information on projects and prctl.

19.4.2. systemd RemoveIPC

If systemd is in use, some care must be taken that IPC resources (including shared memory) are not prematurely removed by the operating system. This is especially of concern when installing PostgreSQL from source. Users of distribution packages of PostgreSQL are less likely to be affected, as the postgres user is then normally created as a system user.

The setting RemoveIPC in logind.conf controls whether IPC objects are removed when a user fully logs out. System users are exempt. This setting defaults to on in stock systemd, but some operating system distributions default it to off.

A typical observed effect when this setting is on is that shared memory objects used for parallel query execution are removed at apparently random times, leading to errors and warnings while attempting to open and remove them, like

WARNING:  could not remove shared memory segment "/PostgreSQL.1450751626": No such file or directory

Different types of IPC objects (shared memory vs. semaphores, System V vs. POSIX) are treated slightly differently by systemd, so one might observe that some IPC resources are not removed in the same way as others. But it is not advisable to rely on these subtle differences.

A “user logging out” might happen as part of a maintenance job or manually when an administrator logs in as the postgres user or something similar, so it is hard to prevent in general.

What is a “system user” is determined at systemd compile time from the SYS_UID_MAX setting in /etc/login.defs.

Packaging and deployment scripts should be careful to create the postgres user as a system user by using useradd -r, adduser --system, or equivalent.

Alternatively, if the user account was created incorrectly or cannot be changed, it is recommended to set

RemoveIPC=no

in /etc/systemd/logind.conf or another appropriate configuration file.

Caution

At least one of these two things has to be ensured, or the PostgreSQL server will be very unreliable.

19.4.3. Resource Limits

Unix-like operating systems enforce various kinds of resource limits that might interfere with the operation of your PostgreSQL server. Of particular importance are limits on the number of processes per user, the number of open files per process, and the amount of memory available to each process. Each of these have a “hard” and a “soft” limit. The soft limit is what actually counts but it can be changed by the user up to the hard limit. The hard limit can only be changed by the root user. The system call setrlimit is responsible for setting these parameters. The shell's built-in command ulimit (Bourne shells) or limit (csh) is used to control the resource limits from the command line. On BSD-derived systems the file /etc/login.conf controls the various resource limits set during login. See the operating system documentation for details. The relevant parameters are maxproc, openfiles, and datasize. For example:

default:\
...
        :datasize-cur=256M:\
        :maxproc-cur=256:\
        :openfiles-cur=256:\
...

(-cur is the soft limit. Append -max to set the hard limit.)

Kernels can also have system-wide limits on some resources.

  • On Linux /proc/sys/fs/file-max determines the maximum number of open files that the kernel will support. It can be changed by writing a different number into the file or by adding an assignment in /etc/sysctl.conf. The maximum limit of files per process is fixed at the time the kernel is compiled; see /usr/src/linux/Documentation/proc.txt for more information.

The PostgreSQL server uses one process per connection so you should provide for at least as many processes as allowed connections, in addition to what you need for the rest of your system. This is usually not a problem but if you run several servers on one machine things might get tight.

The factory default limit on open files is often set to “socially friendly” values that allow many users to coexist on a machine without using an inappropriate fraction of the system resources. If you run many servers on a machine this is perhaps what you want, but on dedicated servers you might want to raise this limit.

On the other side of the coin, some systems allow individual processes to open large numbers of files; if more than a few processes do so then the system-wide limit can easily be exceeded. If you find this happening, and you do not want to alter the system-wide limit, you can set PostgreSQL's max_files_per_process configuration parameter to limit the consumption of open files.

19.4.4. Linux Memory Overcommit

The default virtual memory behavior on Linux is not optimal for PostgreSQL. Because of the way that the kernel implements memory overcommit, the kernel might terminate the PostgreSQL postmaster (the supervisor server process) if the memory demands of either PostgreSQL or another process cause the system to run out of virtual memory.

If this happens, you will see a kernel message that looks like this (consult your system documentation and configuration on where to look for such a message):

Out of Memory: Killed process 12345 (postgres).

This indicates that the postgres process has been terminated due to memory pressure. Although existing database connections will continue to function normally, no new connections will be accepted. To recover, PostgreSQL will need to be restarted.

One way to avoid this problem is to run PostgreSQL on a machine where you can be sure that other processes will not run the machine out of memory. If memory is tight, increasing the swap space of the operating system can help avoid the problem, because the out-of-memory (OOM) killer is invoked only when physical memory and swap space are exhausted.

If PostgreSQL itself is the cause of the system running out of memory, you can avoid the problem by changing your configuration. In some cases, it may help to lower memory-related configuration parameters, particularly shared_buffers, work_mem, and hash_mem_multiplier. In other cases, the problem may be caused by allowing too many connections to the database server itself. In many cases, it may be better to reduce max_connections and instead make use of external connection-pooling software.

It is possible to modify the kernel's behavior so that it will not “overcommit” memory. Although this setting will not prevent the OOM killer from being invoked altogether, it will lower the chances significantly and will therefore lead to more robust system behavior. This is done by selecting strict overcommit mode via sysctl:

sysctl -w vm.overcommit_memory=2

or placing an equivalent entry in /etc/sysctl.conf. You might also wish to modify the related setting vm.overcommit_ratio. For details see the kernel documentation file https://www.kernel.org/doc/Documentation/vm/overcommit-accounting.

Another approach, which can be used with or without altering vm.overcommit_memory, is to set the process-specific OOM score adjustment value for the postmaster process to -1000, thereby guaranteeing it will not be targeted by the OOM killer. The simplest way to do this is to execute

echo -1000 > /proc/self/oom_score_adj

in the postmaster's startup script just before invoking the postmaster. Note that this action must be done as root, or it will have no effect; so a root-owned startup script is the easiest place to do it. If you do this, you should also set these environment variables in the startup script before invoking the postmaster:

export PG_OOM_ADJUST_FILE=/proc/self/oom_score_adj
export PG_OOM_ADJUST_VALUE=0

These settings will cause postmaster child processes to run with the normal OOM score adjustment of zero, so that the OOM killer can still target them at need. You could use some other value for PG_OOM_ADJUST_VALUE if you want the child processes to run with some other OOM score adjustment. (PG_OOM_ADJUST_VALUE can also be omitted, in which case it defaults to zero.) If you do not set PG_OOM_ADJUST_FILE, the child processes will run with the same OOM score adjustment as the postmaster, which is unwise since the whole point is to ensure that the postmaster has a preferential setting.

19.4.5. Linux Huge Pages

Using huge pages reduces overhead when using large contiguous chunks of memory, as PostgreSQL does, particularly when using large values of shared_buffers. To use this feature in PostgreSQL you need a kernel with CONFIG_HUGETLBFS=y and CONFIG_HUGETLB_PAGE=y. You will also have to configure the operating system to provide enough huge pages of the desired size. To estimate the number of huge pages needed, start PostgreSQL without huge pages enabled and check the postmaster's anonymous shared memory segment size, as well as the system's default and supported huge page sizes, using the /proc and /sys file systems. This might look like:

$ head -1 $PGDATA/postmaster.pid
4170
$ pmap 4170 | awk '/rw-s/ && /zero/ {print $2}'
6490428K
$ grep ^Hugepagesize /proc/meminfo
Hugepagesize:       2048 kB
$ ls /sys/kernel/mm/hugepages
hugepages-1048576kB  hugepages-2048kB

In this example the default is 2MB, but you can also explicitly request either 2MB or 1GB with huge_page_size. Assuming 2MB huge pages, 6490428 / 2048 gives approximately 3169.154, so in this example we need at least 3170 huge pages. A larger setting would be appropriate if other programs on the machine also need huge pages. We can set this with:

# sysctl -w vm.nr_hugepages=3170

Don't forget to add this setting to /etc/sysctl.conf so that it is reapplied after reboots. For non-default huge page sizes, we can instead use:

# echo 3170 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

It is also possible to provide these settings at boot time using kernel parameters such as hugepagesz=2M hugepages=3170.

Sometimes the kernel is not able to allocate the desired number of huge pages immediately due to fragmentation, so it might be necessary to repeat the command or to reboot. (Immediately after a reboot, most of the machine's memory should be available to convert into huge pages.) To verify the huge page allocation situation for a given size, use:

$ cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

It may also be necessary to give the database server's operating system user permission to use huge pages by setting vm.hugetlb_shm_group via sysctl, and/or give permission to lock memory with ulimit -l.

The default behavior for huge pages in PostgreSQL is to use them when possible, with the system's default huge page size, and to fall back to normal pages on failure. To enforce the use of huge pages, you can set huge_pages to on in postgresql.conf. Note that with this setting PostgreSQL will fail to start if not enough huge pages are available.

For a detailed description of the Linux huge pages feature have a look at https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt.

Prev | Up | Next
19.3. Starting the Database Server | Home | 19.5. Shutting Down the Server


19.7. Preventing Server Spoofing
Prev Up Chapter 19. Server Setup and Operation Home Next

19.7. Preventing Server Spoofing

While the server is running, it is not possible for a malicious user to take the place of the normal database server. However, when the server is down, it is possible for a local user to spoof the normal server by starting their own server. The spoof server could read passwords and queries sent by clients, but could not return any data because the PGDATA directory would still be secure because of directory permissions. Spoofing is possible because any user can start a database server; a client cannot identify an invalid server unless it is specially configured.

One way to prevent spoofing of local connections is to use a Unix domain socket directory (unix_socket_directories) that has write permission only for a trusted local user. This prevents a malicious user from creating their own socket file in that directory. If you are concerned that some applications might still reference /tmp for the socket file and hence be vulnerable to spoofing, during operating system startup create a symbolic link /tmp/.s.PGSQL.5432 that points to the relocated socket file. You also might need to modify your /tmp cleanup script to prevent removal of the symbolic link.

Another option for local connections is for clients to use requirepeer to specify the required owner of the server process connected to the socket.

To prevent spoofing on TCP connections, either use SSL certificates and make sure that clients check the server's certificate, or use GSSAPI encryption (or both, if they're on separate connections).

To prevent spoofing with SSL, the server must be configured to accept only hostssl connections (Section 21.1) and have SSL key and certificate files (Section 19.9). The TCP client must connect using sslmode=verify-ca or verify-full and have the appropriate root certificate file installed (Section 34.19.1).

To prevent spoofing with GSSAPI, the server must be configured to accept only hostgssenc connections (Section 21.1) and use gss authentication with them. The TCP client must connect using gssencmode=require.

Prev | Up | Next
19.6. Upgrading a PostgreSQL Cluster | Home | 19.8. Encryption Options


19.8. Encryption Options
Prev Up Chapter 19. Server Setup and Operation Home Next

19.8. Encryption Options

PostgreSQL offers encryption at several levels, and provides flexibility in protecting data from disclosure due to database server theft, unscrupulous administrators, and insecure networks. Encryption might also be required to secure sensitive data such as medical records or financial transactions.

Password Encryption

Database user passwords are stored as hashes (determined by the setting password_encryption), so the administrator cannot determine the actual password assigned to the user. If SCRAM or MD5 encryption is used for client authentication, the unencrypted password is never even temporarily present on the server because the client encrypts it before being sent across the network. SCRAM is preferred, because it is an Internet standard and is more secure than the PostgreSQL-specific MD5 authentication protocol.

Encryption For Specific Columns

The pgcrypto module allows certain fields to be stored encrypted. This is useful if only some of the data is sensitive. The client supplies the decryption key and the data is decrypted on the server and then sent to the client.

The decrypted data and the decryption key are present on the server for a brief time while it is being decrypted and communicated between the client and server. This presents a brief moment where the data and keys can be intercepted by someone with complete access to the database server, such as the system administrator.

Data Partition Encryption

Storage encryption can be performed at the file system level or the block level. Linux file system encryption options include eCryptfs and EncFS, while FreeBSD uses PEFS. Block level or full disk encryption options include dm-crypt + LUKS on Linux and GEOM modules geli and gbde on FreeBSD. Many other operating systems support this functionality, including Windows.

This mechanism prevents unencrypted data from being read from the drives if the drives or the entire computer is stolen. This does not protect against attacks while the file system is mounted, because when mounted, the operating system provides an unencrypted view of the data. However, to mount the file system, you need some way for the encryption key to be passed to the operating system, and sometimes the key is stored somewhere on the host that mounts the disk.

Encrypting Data Across A Network

SSL connections encrypt all data sent across the network: the password, the queries, and the data returned. The pg_hba.conf file allows administrators to specify which hosts can use non-encrypted connections (host) and which require SSL-encrypted connections (hostssl). Also, clients can specify that they connect to servers only via SSL.

GSSAPI-encrypted connections encrypt all data sent across the network, including queries and data returned. (No password is sent across the network.) The pg_hba.conf file allows administrators to specify which hosts can use non-encrypted connections (host) and which require GSSAPI-encrypted connections (hostgssenc). Also, clients can specify that they connect to servers only on GSSAPI-encrypted connections (gssencmode=require).

Stunnel or SSH can also be used to encrypt transmissions.

SSL Host Authentication

It is possible for both the client and server to provide SSL certificates to each other. It takes some extra configuration on each side, but this provides stronger verification of identity than the mere use of passwords. It prevents a computer from pretending to be the server just long enough to read the password sent by the client. It also helps prevent “man in the middle” attacks where a computer between the client and server pretends to be the server and reads and passes all data between the client and server.

Client-Side Encryption

If the system administrator for the server's machine cannot be trusted, it is necessary for the client to encrypt the data; this way, unencrypted data never appears on the database server. Data is encrypted on the client before being sent to the server, and database results have to be decrypted on the client before being used.

Prev | Up | Next
19.7. Preventing Server Spoofing | Home | 19.9. Secure TCP/IP Connections with SSL


19.10. Secure TCP/IP Connections with GSSAPI Encryption
Prev Up Chapter 19. Server Setup and Operation Home Next

19.10. Secure TCP/IP Connections with GSSAPI Encryption

19.10.1. Basic Setup

PostgreSQL also has native support for using GSSAPI to encrypt client/server communications for increased security. Support requires that a GSSAPI implementation (such as MIT Kerberos) is installed on both client and server systems, and that support in PostgreSQL is enabled at build time (see Chapter 17).

19.10.1. Basic Setup

The PostgreSQL server will listen for both normal and GSSAPI-encrypted connections on the same TCP port, and will negotiate with any connecting client whether to use GSSAPI for encryption (and for authentication). By default, this decision is up to the client (which means it can be downgraded by an attacker); see Section 21.1 about setting up the server to require the use of GSSAPI for some or all connections.

When using GSSAPI for encryption, it is common to use GSSAPI for authentication as well, since the underlying mechanism will determine both client and server identities (according to the GSSAPI implementation) in any case. But this is not required; another PostgreSQL authentication method can be chosen to perform additional verification.

Other than configuration of the negotiation behavior, GSSAPI encryption requires no setup beyond that which is necessary for GSSAPI authentication. (For more information on configuring that, see Section 21.6.)

Prev | Up | Next
19.9. Secure TCP/IP Connections with SSL | Home | 19.11. Secure TCP/IP Connections with SSH Tunnels


19.12. Registering Event Log on Windows
Prev Up Chapter 19. Server Setup and Operation Home Next

19.12. Registering Event Log on Windows

To register a Windows event log library with the operating system, issue this command:

regsvr32 pgsql_library_directory/pgevent.dll

This creates registry entries used by the event viewer, under the default event source named PostgreSQL.

To specify a different event source name (see event_source), use the /n and /i options:

regsvr32 /n /i:event_source_name pgsql_library_directory/pgevent.dll

To unregister the event log library from the operating system, issue this command:

regsvr32 /u [/i:event_source_name] pgsql_library_directory/pgevent.dll

Note

To enable event logging in the database server, modify log_destination to include eventlog in postgresql.conf.

Prev | Up | Next
19.11. Secure TCP/IP Connections with SSH Tunnels | Home | Chapter 20. Server Configuration


20.1. Setting Parameters
Prev Up Chapter 20. Server Configuration Home Next

20.1. Setting Parameters

20.1.1. Parameter Names and Values
20.1.2. Parameter Interaction via the Configuration File
20.1.3. Parameter Interaction via SQL
20.1.4. Parameter Interaction via the Shell
20.1.5. Managing Configuration File Contents

20.1.1. Parameter Names and Values

All parameter names are case-insensitive. Every parameter takes a value of one of five types: boolean, string, integer, floating point, or enumerated (enum). The type determines the syntax for setting the parameter:

  • Boolean: Values can be written as on, off, true, false, yes, no, 1, 0 (all case-insensitive) or any unambiguous prefix of one of these.

  • String: In general, enclose the value in single quotes, doubling any single quotes within the value. Quotes can usually be omitted if the value is a simple number or identifier, however. (Values that match an SQL keyword require quoting in some contexts.)

  • Numeric (integer and floating point): Numeric parameters can be specified in the customary integer and floating-point formats; fractional values are rounded to the nearest integer if the parameter is of integer type. Integer parameters additionally accept hexadecimal input (beginning with 0x) and octal input (beginning with 0), but these formats cannot have a fraction. Do not use thousands separators. Quotes are not required, except for hexadecimal input.

  • Numeric with Unit: Some numeric parameters have an implicit unit, because they describe quantities of memory or time. The unit might be bytes, kilobytes, blocks (typically eight kilobytes), milliseconds, seconds, or minutes. An unadorned numeric value for one of these settings will use the setting's default unit, which can be learned from pg_settings.unit. For convenience, settings can be given with a unit specified explicitly, for example '120 ms' for a time value, and they will be converted to whatever the parameter's actual unit is. Note that the value must be written as a string (with quotes) to use this feature. The unit name is case-sensitive, and there can be whitespace between the numeric value and the unit.

    • Valid memory units are B (bytes), kB (kilobytes), MB (megabytes), GB (gigabytes), and TB (terabytes). The multiplier for memory units is 1024, not 1000.

    • Valid time units are us (microseconds), ms (milliseconds), s (seconds), min (minutes), h (hours), and d (days).

    If a fractional value is specified with a unit, it will be rounded to a multiple of the next smaller unit if there is one. For example, 30.1 GB will be converted to 30822 MB not 32319628902 B. If the parameter is of integer type, a final rounding to integer occurs after any unit conversion.

  • Enumerated: Enumerated-type parameters are written in the same way as string parameters, but are restricted to have one of a limited set of values. The values allowable for such a parameter can be found from pg_settings.enumvals. Enum parameter values are case-insensitive.

20.1.2. Parameter Interaction via the Configuration File

The most fundamental way to set these parameters is to edit the file postgresql.conf, which is normally kept in the data directory. A default copy is installed when the database cluster directory is initialized. An example of what this file might look like is:

# This is a comment
log_connections = yes
log_destination = 'syslog'
search_path = '"$user", public'
shared_buffers = 128MB

One parameter is specified per line. The equal sign between name and value is optional. Whitespace is insignificant (except within a quoted parameter value) and blank lines are ignored. Hash marks (#) designate the remainder of the line as a comment. Parameter values that are not simple identifiers or numbers must be single-quoted. To embed a single quote in a parameter value, write either two quotes (preferred) or backslash-quote. If the file contains multiple entries for the same parameter, all but the last one are ignored.

Parameters set in this way provide default values for the cluster. The settings seen by active sessions will be these values unless they are overridden. The following sections describe ways in which the administrator or user can override these defaults.

The configuration file is reread whenever the main server process receives a SIGHUP signal; this signal is most easily sent by running pg_ctl reload from the command line or by calling the SQL function pg_reload_conf(). The main server process also propagates this signal to all currently running server processes, so that existing sessions also adopt the new values (this will happen after they complete any currently-executing client command). Alternatively, you can send the signal to a single server process directly. Some parameters can only be set at server start; any changes to their entries in the configuration file will be ignored until the server is restarted. Invalid parameter settings in the configuration file are likewise ignored (but logged) during SIGHUP processing.

In addition to postgresql.conf, a PostgreSQL data directory contains a file postgresql.auto.conf, which has the same format as postgresql.conf but is intended to be edited automatically, not manually. This file holds settings provided through the ALTER SYSTEM command. This file is read whenever postgresql.conf is, and its settings take effect in the same way. Settings in postgresql.auto.conf override those in postgresql.conf.

External tools may also modify postgresql.auto.conf. It is not recommended to do this while the server is running, since a concurrent ALTER SYSTEM command could overwrite such changes. Such tools might simply append new settings to the end, or they might choose to remove duplicate settings and/or comments (as ALTER SYSTEM will).

The system view pg_file_settings can be helpful for pre-testing changes to the configuration files, or for diagnosing problems if a SIGHUP signal did not have the desired effects.

20.1.3. Parameter Interaction via SQL

PostgreSQL provides three SQL commands to establish configuration defaults. The already-mentioned ALTER SYSTEM command provides an SQL-accessible means of changing global defaults; it is functionally equivalent to editing postgresql.conf. In addition, there are two commands that allow setting of defaults on a per-database or per-role basis:

  • The ALTER DATABASE command allows global settings to be overridden on a per-database basis.

  • The ALTER ROLE command allows both global and per-database settings to be overridden with user-specific values.

Values set with ALTER DATABASE and ALTER ROLE are applied only when starting a fresh database session. They override values obtained from the configuration files or server command line, and constitute defaults for the rest of the session. Note that some settings cannot be changed after server start, and so cannot be set with these commands (or the ones listed below).

Once a client is connected to the database, PostgreSQL provides two additional SQL commands (and equivalent functions) to interact with session-local configuration settings:

  • The SHOW command allows inspection of the current value of any parameter. The corresponding SQL function is current_setting(setting_name text) (see Section 9.27.1).

  • The SET command allows modification of the current value of those parameters that can be set locally to a session; it has no effect on other sessions. The corresponding SQL function is set_config(setting_name, new_value, is_local) (see Section 9.27.1).

In addition, the system view pg_settings can be used to view and change session-local values:

  • Querying this view is similar to using SHOW ALL but provides more detail. It is also more flexible, since it's possible to specify filter conditions or join against other relations.

  • Using UPDATE on this view, specifically updating the setting column, is the equivalent of issuing SET commands. For example, the equivalent of

    SET configuration_parameter TO DEFAULT;

    is:

    UPDATE pg_settings SET setting = reset_val WHERE name = 'configuration_parameter';

20.1.4. Parameter Interaction via the Shell

In addition to setting global defaults or attaching overrides at the database or role level, you can pass settings to PostgreSQL via shell facilities. Both the server and libpq client library accept parameter values via the shell.

  • During server startup, parameter settings can be passed to the postgres command via the -c command-line parameter. For example,

    postgres -c log_connections=yes -c log_destination='syslog'

    Settings provided in this way override those set via postgresql.conf or ALTER SYSTEM, so they cannot be changed globally without restarting the server.

  • When starting a client session via libpq, parameter settings can be specified using the PGOPTIONS environment variable. Settings established in this way constitute defaults for the life of the session, but do not affect other sessions. For historical reasons, the format of PGOPTIONS is similar to that used when launching the postgres command; specifically, the -c flag must be specified. For example,

    env PGOPTIONS="-c geqo=off -c statement_timeout=5min" psql

    Other clients and libraries might provide their own mechanisms, via the shell or otherwise, that allow the user to alter session settings without direct use of SQL commands.

20.1.5. Managing Configuration File Contents

PostgreSQL provides several features for breaking down complex postgresql.conf files into sub-files. These features are especially useful when managing multiple servers with related, but not identical, configurations.

In addition to individual parameter settings, the postgresql.conf file can contain include directives, which specify another file to read and process as if it were inserted into the configuration file at this point. This feature allows a configuration file to be divided into physically separate parts. Include directives simply look like:

include 'filename'

If the file name is not an absolute path, it is taken as relative to the directory containing the referencing configuration file. Inclusions can be nested.

There is also an include_if_exists directive, which acts the same as the include directive, except when the referenced file does not exist or cannot be read. A regular include will consider this an error condition, but include_if_exists merely logs a message and continues processing the referencing configuration file.

The postgresql.conf file can also contain include_dir directives, which specify an entire directory of configuration files to include. These look like

include_dir 'directory'

Non-absolute directory names are taken as relative to the directory containing the referencing configuration file. Within the specified directory, only non-directory files whose names end with the suffix .conf will be included. File names that start with the . character are also ignored, to prevent mistakes since such files are hidden on some platforms. Multiple files within an include directory are processed in file name order (according to C locale rules, i.e., numbers before letters, and uppercase letters before lowercase ones).

Include files or directories can be used to logically separate portions of the database configuration, rather than having a single large postgresql.conf file. Consider a company that has two database servers, each with a different amount of memory. There are likely elements of the configuration both will share, for things such as logging. But memory-related parameters on the server will vary between the two. And there might be server specific customizations, too. One way to manage this situation is to break the custom configuration changes for your site into three files. You could add this to the end of your postgresql.conf file to include them:

include 'shared.conf'
include 'memory.conf'
include 'server.conf'

All systems would have the same shared.conf. Each server with a particular amount of memory could share the same memory.conf; you might have one for all servers with 8GB of RAM, another for those having 16GB. And finally server.conf could have truly server-specific configuration information in it.

Another possibility is to create a configuration file directory and put this information into files there. For example, a conf.d directory could be referenced at the end of postgresql.conf:

include_dir 'conf.d'

Then you could name the files in the conf.d directory like this:

00shared.conf
01memory.conf
02server.conf

This naming convention establishes a clear order in which these files will be loaded. This is important because only the last setting encountered for a particular parameter while the server is reading configuration files will be used. In this example, something set in conf.d/02server.conf would override a value set in conf.d/01memory.conf.

You might instead use this approach to naming the files descriptively:

00shared.conf
01memory-8GB.conf
02server-foo.conf

This sort of arrangement gives a unique name for each configuration file variation. This can help eliminate ambiguity when several servers have their configurations all stored in one place, such as in a version control repository. (Storing database configuration files under version control is another good practice to consider.)

Prev | Up | Next
Chapter 20. Server Configuration | Home | 20.2. File Locations


Chapter 21. Client Authentication
Prev Up Part III. Server Administration Home Next

Chapter 21. Client Authentication

Table of Contents

21.1. The pg_hba.conf File
21.2. User Name Maps
21.3. Authentication Methods
21.4. Trust Authentication
21.5. Password Authentication
21.6. GSSAPI Authentication
21.7. SSPI Authentication
21.8. Ident Authentication
21.9. Peer Authentication
21.10. LDAP Authentication
21.11. RADIUS Authentication
21.12. Certificate Authentication
21.13. PAM Authentication
21.14. BSD Authentication
21.15. Authentication Problems

When a client application connects to the database server, it specifies which PostgreSQL database user name it wants to connect as, much the same way one logs into a Unix computer as a particular user. Within the SQL environment the active database user name determines access privileges to database objects — see Chapter 22 for more information. Therefore, it is essential to restrict which database users can connect.

Note

As explained in Chapter 22, PostgreSQL actually does privilege management in terms of “roles”. In this chapter, we consistently use database user to mean “role with the LOGIN privilege”.

Authentication is the process by which the database server establishes the identity of the client, and by extension determines whether the client application (or the user who runs the client application) is permitted to connect with the database user name that was requested.

PostgreSQL offers a number of different client authentication methods. The method used to authenticate a particular client connection can be selected on the basis of (client) host address, database, and user.

PostgreSQL database user names are logically separate from user names of the operating system in which the server runs. If all the users of a particular server also have accounts on the server's machine, it makes sense to assign database user names that match their operating system user names. However, a server that accepts remote connections might have many database users who have no local operating system account, and in such cases there need be no connection between database user names and OS user names.

Prev | Up | Next
20.18. Short Options | Home | 21.1. The pg_hba.conf File


21.1. The pg_hba.conf File
Prev Up Chapter 21. Client Authentication Home Next

21.1. The pg_hba.conf File

Client authentication is controlled by a configuration file, which traditionally is named pg_hba.conf and is stored in the database cluster's data directory. (HBA stands for host-based authentication.) A default pg_hba.conf file is installed when the data directory is initialized by initdb. It is possible to place the authentication configuration file elsewhere, however; see the hba_file configuration parameter.

The general format of the pg_hba.conf file is a set of records, one per line. Blank lines are ignored, as is any text after the # comment character. A record can be continued onto the next line by ending the line with a backslash. (Backslashes are not special except at the end of a line.) A record is made up of a number of fields which are separated by spaces and/or tabs. Fields can contain white space if the field value is double-quoted. Quoting one of the keywords in a database, user, or address field (e.g., all or replication) makes the word lose its special meaning, and just match a database, user, or host with that name. Backslash line continuation applies even within quoted text or comments.

Each record specifies a connection type, a client IP address range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters. The first record with a matching connection type, client address, requested database, and user name is used to perform authentication. There is no “fall-through” or “backup”: if one record is chosen and the authentication fails, subsequent records are not considered. If no record matches, access is denied.

A record can have several formats:

local         database  user  auth-method [auth-options]
host          database  user  address     auth-method  [auth-options]
hostssl       database  user  address     auth-method  [auth-options]
hostnossl     database  user  address     auth-method  [auth-options]
hostgssenc    database  user  address     auth-method  [auth-options]
hostnogssenc  database  user  address     auth-method  [auth-options]
host          database  user  IP-address  IP-mask      auth-method  [auth-options]
hostssl       database  user  IP-address  IP-mask      auth-method  [auth-options]
hostnossl     database  user  IP-address  IP-mask      auth-method  [auth-options]
hostgssenc    database  user  IP-address  IP-mask      auth-method  [auth-options]
hostnogssenc  database  user  IP-address  IP-mask      auth-method  [auth-options]

The meaning of the fields is as follows:

local

This record matches connection attempts using Unix-domain sockets. Without a record of this type, Unix-domain socket connections are disallowed.

host

This record matches connection attempts made using TCP/IP. host records match SSL or non-SSL connection attempts as well as GSSAPI encrypted or non-GSSAPI encrypted connection attempts.

Note

Remote TCP/IP connections will not be possible unless the server is started with an appropriate value for the listen_addresses configuration parameter, since the default behavior is to listen for TCP/IP connections only on the local loopback address localhost.

hostssl

This record matches connection attempts made using TCP/IP, but only when the connection is made with SSL encryption.

To make use of this option the server must be built with SSL support. Furthermore, SSL must be enabled by setting the ssl configuration parameter (see Section 19.9 for more information). Otherwise, the hostssl record is ignored except for logging a warning that it cannot match any connections.

hostnossl

This record type has the opposite behavior of hostssl; it only matches connection attempts made over TCP/IP that do not use SSL.

hostgssenc

This record matches connection attempts made using TCP/IP, but only when the connection is made with GSSAPI encryption.

To make use of this option the server must be built with GSSAPI support. Otherwise, the hostgssenc record is ignored except for logging a warning that it cannot match any connections.

hostnogssenc

This record type has the opposite behavior of hostgssenc; it only matches connection attempts made over TCP/IP that do not use GSSAPI encryption.

database

Specifies which database name(s) this record matches. The value all specifies that it matches all databases. The value sameuser specifies that the record matches if the requested database has the same name as the requested user. The value samerole specifies that the requested user must be a member of the role with the same name as the requested database. (samegroup is an obsolete but still accepted spelling of samerole.) Superusers are not considered to be members of a role for the purposes of samerole unless they are explicitly members of the role, directly or indirectly, and not just by virtue of being a superuser. The value replication specifies that the record matches if a physical replication connection is requested, however, it doesn't match with logical replication connections. Note that physical replication connections do not specify any particular database whereas logical replication connections do specify it. Otherwise, this is the name of a specific PostgreSQL database. Multiple database names can be supplied by separating them with commas. A separate file containing database names can be specified by preceding the file name with @.

user

Specifies which database user name(s) this record matches. The value all specifies that it matches all users. Otherwise, this is either the name of a specific database user, or a group name preceded by +. (Recall that there is no real distinction between users and groups in PostgreSQL; a + mark really means “match any of the roles that are directly or indirectly members of this role”, while a name without a + mark matches only that specific role.) For this purpose, a superuser is only considered to be a member of a role if they are explicitly a member of the role, directly or indirectly, and not just by virtue of being a superuser. Multiple user names can be supplied by separating them with commas. A separate file containing user names can be specified by preceding the file name with @.

address

Specifies the client machine address(es) that this record matches. This field can contain either a host name, an IP address range, or one of the special key words mentioned below.

An IP address range is specified using standard numeric notation for the range's starting address, then a slash (/) and a CIDR mask length. The mask length indicates the number of high-order bits of the client IP address that must match. Bits to the right of this should be zero in the given IP address. There must not be any white space between the IP address, the /, and the CIDR mask length.

Typical examples of an IPv4 address range specified this way are 172.20.143.89/32 for a single host, or 172.20.143.0/24 for a small network, or 10.6.0.0/16 for a larger one. An IPv6 address range might look like ::1/128 for a single host (in this case the IPv6 loopback address) or fe80::7a31:c1ff:0000:0000/96 for a small network. 0.0.0.0/0 represents all IPv4 addresses, and ::0/0 represents all IPv6 addresses. To specify a single host, use a mask length of 32 for IPv4 or 128 for IPv6. In a network address, do not omit trailing zeroes.

An entry given in IPv4 format will match only IPv4 connections, and an entry given in IPv6 format will match only IPv6 connections, even if the represented address is in the IPv4-in-IPv6 range. Note that entries in IPv6 format will be rejected if the system's C library does not have support for IPv6 addresses.

You can also write all to match any IP address, samehost to match any of the server's own IP addresses, or samenet to match any address in any subnet that the server is directly connected to.

If a host name is specified (anything that is not an IP address range or a special key word is treated as a host name), that name is compared with the result of a reverse name resolution of the client's IP address (e.g., reverse DNS lookup, if DNS is used). Host name comparisons are case insensitive. If there is a match, then a forward name resolution (e.g., forward DNS lookup) is performed on the host name to check whether any of the addresses it resolves to are equal to the client's IP address. If both directions match, then the entry is considered to match. (The host name that is used in pg_hba.conf should be the one that address-to-name resolution of the client's IP address returns, otherwise the line won't be matched. Some host name databases allow associating an IP address with multiple host names, but the operating system will only return one host name when asked to resolve an IP address.)

A host name specification that starts with a dot (.) matches a suffix of the actual host name. So .example.com would match foo.example.com (but not just example.com).

When host names are specified in pg_hba.conf, you should make sure that name resolution is reasonably fast. It can be of advantage to set up a local name resolution cache such as nscd. Also, you may wish to enable the configuration parameter log_hostname to see the client's host name instead of the IP address in the log.

These fields do not apply to local records.

Note

Users sometimes wonder why host names are handled in this seemingly complicated way, with two name resolutions including a reverse lookup of the client's IP address. This complicates use of the feature in case the client's reverse DNS entry is not set up or yields some undesirable host name. It is done primarily for efficiency: this way, a connection attempt requires at most two resolver lookups, one reverse and one forward. If there is a resolver problem with some address, it becomes only that client's problem. A hypothetical alternative implementation that only did forward lookups would have to resolve every host name mentioned in pg_hba.conf during every connection attempt. That could be quite slow if many names are listed. And if there is a resolver problem with one of the host names, it becomes everyone's problem.

Also, a reverse lookup is necessary to implement the suffix matching feature, because the actual client host name needs to be known in order to match it against the pattern.

Note that this behavior is consistent with other popular implementations of host name-based access control, such as the Apache HTTP Server and TCP Wrappers.

IP-address
IP-mask

These two fields can be used as an alternative to the IP-address/mask-length notation. Instead of specifying the mask length, the actual mask is specified in a separate column. For example, 255.0.0.0 represents an IPv4 CIDR mask length of 8, and 255.255.255.255 represents a CIDR mask length of 32.

These fields do not apply to local records.

auth-method

Specifies the authentication method to use when a connection matches this record. The possible choices are summarized here; details are in Section 21.3.

trust

Allow the connection unconditionally. This method allows anyone that can connect to the PostgreSQL database server to login as any PostgreSQL user they wish, without the need for a password or any other authentication. See Section 21.4 for details.

reject

Reject the connection unconditionally. This is useful for “filtering out” certain hosts from a group, for example a reject line could block a specific host from connecting, while a later line allows the remaining hosts in a specific network to connect.

scram-sha-256

Perform SCRAM-SHA-256 authentication to verify the user's password. See Section 21.5 for details.

md5

Perform SCRAM-SHA-256 or MD5 authentication to verify the user's password. See Section 21.5 for details.

password

Require the client to supply an unencrypted password for authentication. Since the password is sent in clear text over the network, this should not be used on untrusted networks. See Section 21.5 for details.

gss

Use GSSAPI to authenticate the user. This is only available for TCP/IP connections. See Section 21.6 for details. It can be used in conjunction with GSSAPI encryption.

sspi

Use SSPI to authenticate the user. This is only available on Windows. See Section 21.7 for details.

ident

Obtain the operating system user name of the client by contacting the ident server on the client and check if it matches the requested database user name. Ident authentication can only be used on TCP/IP connections. When specified for local connections, peer authentication will be used instead. See Section 21.8 for details.

peer

Obtain the client's operating system user name from the operating system and check if it matches the requested database user name. This is only available for local connections. See Section 21.9 for details.

ldap

Authenticate using an LDAP server. See Section 21.10 for details.

radius

Authenticate using a RADIUS server. See Section 21.11 for details.

cert

Authenticate using SSL client certificates. See Section 21.12 for details.

pam

Authenticate using the Pluggable Authentication Modules (PAM) service provided by the operating system. See Section 21.13 for details.

bsd

Authenticate using the BSD Authentication service provided by the operating system. See Section 21.14 for details.

auth-options

After the auth-method field, there can be field(s) of the form name=value that specify options for the authentication method. Details about which options are available for which authentication methods appear below.

In addition to the method-specific options listed below, there is a method-independent authentication option clientcert, which can be specified in any hostssl record. This option can be set to verify-ca or verify-full. Both options require the client to present a valid (trusted) SSL certificate, while verify-full additionally enforces that the cn (Common Name) in the certificate matches the username or an applicable mapping. This behavior is similar to the cert authentication method (see Section 21.12) but enables pairing the verification of client certificates with any authentication method that supports hostssl entries.

On any record using client certificate authentication (i.e. one using the cert authentication method or one using the clientcert option), you can specify which part of the client certificate credentials to match using the clientname option. This option can have one of two values. If you specify clientname=CN, which is the default, the username is matched against the certificate's Common Name (CN). If instead you specify clientname=DN the username is matched against the entire Distinguished Name (DN) of the certificate. This option is probably best used in conjunction with a username map. The comparison is done with the DN in RFC 2253 format. To see the DN of a client certificate in this format, do

openssl x509 -in myclient.crt -noout --subject -nameopt RFC2253 | sed "s/^subject=//"

Care needs to be taken when using this option, especially when using regular expression matching against the DN.

Files included by @ constructs are read as lists of names, which can be separated by either whitespace or commas. Comments are introduced by #, just as in pg_hba.conf, and nested @ constructs are allowed. Unless the file name following @ is an absolute path, it is taken to be relative to the directory containing the referencing file.

Since the pg_hba.conf records are examined sequentially for each connection attempt, the order of the records is significant. Typically, earlier records will have tight connection match parameters and weaker authentication methods, while later records will have looser match parameters and stronger authentication methods. For example, one might wish to use trust authentication for local TCP/IP connections but require a password for remote TCP/IP connections. In this case a record specifying trust authentication for connections from 127.0.0.1 would appear before a record specifying password authentication for a wider range of allowed client IP addresses.

The pg_hba.conf file is read on start-up and when the main server process receives a SIGHUP signal. If you edit the file on an active system, you will need to signal the postmaster (using pg_ctl reload, calling the SQL function pg_reload_conf(), or using kill -HUP) to make it re-read the file.

Note

The preceding statement is not true on Microsoft Windows: there, any changes in the pg_hba.conf file are immediately applied by subsequent new connections.

The system view pg_hba_file_rules can be helpful for pre-testing changes to the pg_hba.conf file, or for diagnosing problems if loading of the file did not have the desired effects. Rows in the view with non-null error fields indicate problems in the corresponding lines of the file.

Tip

To connect to a particular database, a user must not only pass the pg_hba.conf checks, but must have the CONNECT privilege for the database. If you wish to restrict which users can connect to which databases, it's usually easier to control this by granting/revoking CONNECT privilege than to put the rules in pg_hba.conf entries.

Some examples of pg_hba.conf entries are shown in Example 21.1. See the next section for details on the different authentication methods.

Example 21.1. Example pg_hba.conf Entries

# Allow any user on the local system to connect to any database with
# any database user name using Unix-domain sockets (the default for local
# connections).
#
# TYPE  DATABASE        USER            ADDRESS                 METHOD
local   all             all                                     trust

# The same using local loopback TCP/IP connections.
#
# TYPE  DATABASE        USER            ADDRESS                 METHOD
host    all             all             127.0.0.1/32            trust

# The same as the previous line, but using a separate netmask column
#
# TYPE  DATABASE        USER            IP-ADDRESS      IP-MASK             METHOD
host    all             all             127.0.0.1       255.255.255.255     trust

# The same over IPv6.
#
# TYPE  DATABASE        USER            ADDRESS                 METHOD
host    all             all             ::1/128                 trust

# The same using a host name (would typically cover both IPv4 and IPv6).
#
# TYPE  DATABASE        USER            ADDRESS                 METHOD
host    all             all             localhost               trust

# Allow any user from any host with IP address 192.168.93.x to connect
# to database "postgres" as the same user name that ident reports for
# the connection (typically the operating system user name).
#
# TYPE  DATABASE        USER            ADDRESS                 METHOD
host    postgres        all             192.168.93.0/24         ident

# Allow any user from host 192.168.12.10 to connect to database
# "postgres" if the user's password is correctly supplied.
#
# TYPE  DATABASE        USER            ADDRESS                 METHOD
host    postgres        all             192.168.12.10/32        scram-sha-256

# Allow any user from hosts in the example.com domain to connect to
# any database if the user's password is correctly supplied.
#
# Require SCRAM authentication for most users, but make an exception
# for user 'mike', who uses an older client that doesn't support SCRAM
# authentication.
#
# TYPE  DATABASE        USER            ADDRESS                 METHOD
host    all             mike            .example.com            md5
host    all             all             .example.com            scram-sha-256

# In the absence of preceding "host" lines, these three lines will
# reject all connections from 192.168.54.1 (since that entry will be
# matched first), but allow GSSAPI-encrypted connections from anywhere else
# on the Internet.  The zero mask causes no bits of the host IP address to
# be considered, so it matches any host.  Unencrypted GSSAPI connections
# (which "fall through" to the third line since "hostgssenc" only matches
# encrypted GSSAPI connections) are allowed, but only from 192.168.12.10.
#
# TYPE  DATABASE        USER            ADDRESS                 METHOD
host    all             all             192.168.54.1/32         reject
hostgssenc all          all             0.0.0.0/0               gss
host    all             all             192.168.12.10/32        gss

# Allow users from 192.168.x.x hosts to connect to any database, if
# they pass the ident check.  If, for example, ident says the user is
# "bryanh" and he requests to connect as PostgreSQL user "guest1", the
# connection is allowed if there is an entry in pg_ident.conf for map
# "omicron" that says "bryanh" is allowed to connect as "guest1".
#
# TYPE  DATABASE        USER            ADDRESS                 METHOD
host    all             all             192.168.0.0/16          ident map=omicron

# If these are the only three lines for local connections, they will
# allow local users to connect only to their own databases (databases
# with the same name as their database user name) except for administrators
# and members of role "support", who can connect to all databases.  The file
# $PGDATA/admins contains a list of names of administrators.  Passwords
# are required in all cases.
#
# TYPE  DATABASE        USER            ADDRESS                 METHOD
local   sameuser        all                                     md5
local   all             @admins                                 md5
local   all             +support                                md5

# The last two lines above can be combined into a single line:
local   all             @admins,+support                        md5

# The database column can also use lists and file names:
local   db1,db2,@demodbs  all                                   md5

Prev | Up | Next
Chapter 21. Client Authentication | Home | 21.2. User Name Maps


21.2. User Name Maps
Prev Up Chapter 21. Client Authentication Home Next

21.2. User Name Maps

When using an external authentication system such as Ident or GSSAPI, the name of the operating system user that initiated the connection might not be the same as the database user (role) that is to be used. In this case, a user name map can be applied to map the operating system user name to a database user. To use user name mapping, specify map=map-name in the options field in pg_hba.conf. This option is supported for all authentication methods that receive external user names. Since different mappings might be needed for different connections, the name of the map to be used is specified in the map-name parameter in pg_hba.conf to indicate which map to use for each individual connection.

User name maps are defined in the ident map file, which by default is named pg_ident.conf and is stored in the cluster's data directory. (It is possible to place the map file elsewhere, however; see the ident_file configuration parameter.) The ident map file contains lines of the general form:

map-name system-username database-username

Comments, whitespace and line continuations are handled in the same way as in pg_hba.conf. The map-name is an arbitrary name that will be used to refer to this mapping in pg_hba.conf. The other two fields specify an operating system user name and a matching database user name. The same map-name can be used repeatedly to specify multiple user-mappings within a single map.

There is no restriction regarding how many database users a given operating system user can correspond to, nor vice versa. Thus, entries in a map should be thought of as meaning “this operating system user is allowed to connect as this database user”, rather than implying that they are equivalent. The connection will be allowed if there is any map entry that pairs the user name obtained from the external authentication system with the database user name that the user has requested to connect as.

If the system-username field starts with a slash (/), the remainder of the field is treated as a regular expression. (See Section 9.7.3.1 for details of PostgreSQL's regular expression syntax.) The regular expression can include a single capture, or parenthesized subexpression, which can then be referenced in the database-username field as \1 (backslash-one). This allows the mapping of multiple user names in a single line, which is particularly useful for simple syntax substitutions. For example, these entries

mymap   /^(.*)@mydomain\.com$      \1
mymap   /^(.*)@otherdomain\.com$   guest

will remove the domain part for users with system user names that end with @mydomain.com, and allow any user whose system name ends with @otherdomain.com to log in as guest.

Tip

Keep in mind that by default, a regular expression can match just part of a string. It's usually wise to use ^ and $, as shown in the above example, to force the match to be to the entire system user name.

The pg_ident.conf file is read on start-up and when the main server process receives a SIGHUP signal. If you edit the file on an active system, you will need to signal the postmaster (using pg_ctl reload, calling the SQL function pg_reload_conf(), or using kill -HUP) to make it re-read the file.

A pg_ident.conf file that could be used in conjunction with the pg_hba.conf file in Example 21.1 is shown in Example 21.2. In this example, anyone logged in to a machine on the 192.168 network that does not have the operating system user name bryanh, ann, or robert would not be granted access. Unix user robert would only be allowed access when he tries to connect as PostgreSQL user bob, not as robert or anyone else. ann would only be allowed to connect as ann. User bryanh would be allowed to connect as either bryanh or as guest1.

Example 21.2. An Example pg_ident.conf File

# MAPNAME       SYSTEM-USERNAME         PG-USERNAME

omicron         bryanh                  bryanh
omicron         ann                     ann
# bob has user name robert on these machines
omicron         robert                  bob
# bryanh can also connect as guest1
omicron         bryanh                  guest1

Prev | Up | Next
21.1. The pg_hba.conf File | Home | 21.3. Authentication Methods


21.3. Authentication Methods
Prev Up Chapter 21. Client Authentication Home Next

21.3. Authentication Methods

PostgreSQL provides various methods for authenticating users:

  • Trust authentication, which simply trusts that users are who they say they are.

  • Password authentication, which requires that users send a password.

  • GSSAPI authentication, which relies on a GSSAPI-compatible security library. Typically this is used to access an authentication server such as a Kerberos or Microsoft Active Directory server.

  • SSPI authentication, which uses a Windows-specific protocol similar to GSSAPI.

  • Ident authentication, which relies on an “Identification Protocol” (RFC 1413) service on the client's machine. (On local Unix-socket connections, this is treated as peer authentication.)

  • Peer authentication, which relies on operating system facilities to identify the process at the other end of a local connection. This is not supported for remote connections.

  • LDAP authentication, which relies on an LDAP authentication server.

  • RADIUS authentication, which relies on a RADIUS authentication server.

  • Certificate authentication, which requires an SSL connection and authenticates users by checking the SSL certificate they send.

  • PAM authentication, which relies on a PAM (Pluggable Authentication Modules) library.

  • BSD authentication, which relies on the BSD Authentication framework (currently available only on OpenBSD).

Peer authentication is usually recommendable for local connections, though trust authentication might be sufficient in some circumstances. Password authentication is the easiest choice for remote connections. All the other options require some kind of external security infrastructure (usually an authentication server or a certificate authority for issuing SSL certificates), or are platform-specific.

The following sections describe each of these authentication methods in more detail.

Prev | Up | Next
21.2. User Name Maps | Home | 21.4. Trust Authentication


21.4. Trust Authentication
Prev Up Chapter 21. Client Authentication Home Next

21.4. Trust Authentication

When trust authentication is specified, PostgreSQL assumes that anyone who can connect to the server is authorized to access the database with whatever database user name they specify (even superuser names). Of course, restrictions made in the database and user columns still apply. This method should only be used when there is adequate operating-system-level protection on connections to the server.

trust authentication is appropriate and very convenient for local connections on a single-user workstation. It is usually not appropriate by itself on a multiuser machine. However, you might be able to use trust even on a multiuser machine, if you restrict access to the server's Unix-domain socket file using file-system permissions. To do this, set the unix_socket_permissions (and possibly unix_socket_group) configuration parameters as described in Section 20.3. Or you could set the unix_socket_directories configuration parameter to place the socket file in a suitably restricted directory.

Setting file-system permissions only helps for Unix-socket connections. Local TCP/IP connections are not restricted by file-system permissions. Therefore, if you want to use file-system permissions for local security, remove the host ... 127.0.0.1 ... line from pg_hba.conf, or change it to a non-trust authentication method.

trust authentication is only suitable for TCP/IP connections if you trust every user on every machine that is allowed to connect to the server by the pg_hba.conf lines that specify trust. It is seldom reasonable to use trust for any TCP/IP connections other than those from localhost (127.0.0.1).

Prev | Up | Next
21.3. Authentication Methods | Home | 21.5. Password Authentication


21.5. Password Authentication
Prev Up Chapter 21. Client Authentication Home Next

21.5. Password Authentication

There are several password-based authentication methods. These methods operate similarly but differ in how the users' passwords are stored on the server and how the password provided by a client is sent across the connection.

scram-sha-256

The method scram-sha-256 performs SCRAM-SHA-256 authentication, as described in RFC 7677. It is a challenge-response scheme that prevents password sniffing on untrusted connections and supports storing passwords on the server in a cryptographically hashed form that is thought to be secure.

This is the most secure of the currently provided methods, but it is not supported by older client libraries.

md5

The method md5 uses a custom less secure challenge-response mechanism. It prevents password sniffing and avoids storing passwords on the server in plain text but provides no protection if an attacker manages to steal the password hash from the server. Also, the MD5 hash algorithm is nowadays no longer considered secure against determined attacks.

The md5 method cannot be used with the db_user_namespace feature.

To ease transition from the md5 method to the newer SCRAM method, if md5 is specified as a method in pg_hba.conf but the user's password on the server is encrypted for SCRAM (see below), then SCRAM-based authentication will automatically be chosen instead.

password

The method password sends the password in clear-text and is therefore vulnerable to password “sniffing” attacks. It should always be avoided if possible. If the connection is protected by SSL encryption then password can be used safely, though. (Though SSL certificate authentication might be a better choice if one is depending on using SSL).

PostgreSQL database passwords are separate from operating system user passwords. The password for each database user is stored in the pg_authid system catalog. Passwords can be managed with the SQL commands CREATE ROLE and ALTER ROLE, e.g., CREATE ROLE foo WITH LOGIN PASSWORD 'secret', or the psql command \password. If no password has been set up for a user, the stored password is null and password authentication will always fail for that user.

The availability of the different password-based authentication methods depends on how a user's password on the server is encrypted (or hashed, more accurately). This is controlled by the configuration parameter password_encryption at the time the password is set. If a password was encrypted using the scram-sha-256 setting, then it can be used for the authentication methods scram-sha-256 and password (but password transmission will be in plain text in the latter case). The authentication method specification md5 will automatically switch to using the scram-sha-256 method in this case, as explained above, so it will also work. If a password was encrypted using the md5 setting, then it can be used only for the md5 and password authentication method specifications (again, with the password transmitted in plain text in the latter case). (Previous PostgreSQL releases supported storing the password on the server in plain text. This is no longer possible.) To check the currently stored password hashes, see the system catalog pg_authid.

To upgrade an existing installation from md5 to scram-sha-256, after having ensured that all client libraries in use are new enough to support SCRAM, set password_encryption = 'scram-sha-256' in postgresql.conf, make all users set new passwords, and change the authentication method specifications in pg_hba.conf to scram-sha-256.

Prev | Up | Next
21.4. Trust Authentication | Home | 21.6. GSSAPI Authentication


21.6. GSSAPI Authentication
Prev Up Chapter 21. Client Authentication Home Next

21.6. GSSAPI Authentication

GSSAPI is an industry-standard protocol for secure authentication defined in RFC 2743. PostgreSQL supports GSSAPI for authentication, communications encryption, or both. GSSAPI provides automatic authentication (single sign-on) for systems that support it. The authentication itself is secure. If GSSAPI encryption or SSL encryption is used, the data sent along the database connection will be encrypted; otherwise, it will not.

GSSAPI support has to be enabled when PostgreSQL is built; see Chapter 17 for more information.

When GSSAPI uses Kerberos, it uses a standard service principal (authentication identity) name in the format servicename/hostname@realm. The principal name used by a particular installation is not encoded in the PostgreSQL server in any way; rather it is specified in the keytab file that the server reads to determine its identity. If multiple principals are listed in the keytab file, the server will accept any one of them. The server's realm name is the preferred realm specified in the Kerberos configuration file(s) accessible to the server.

When connecting, the client must know the principal name of the server it intends to connect to. The servicename part of the principal is ordinarily postgres, but another value can be selected via libpq's krbsrvname connection parameter. The hostname part is the fully qualified host name that libpq is told to connect to. The realm name is the preferred realm specified in the Kerberos configuration file(s) accessible to the client.

The client will also have a principal name for its own identity (and it must have a valid ticket for this principal). To use GSSAPI for authentication, the client principal must be associated with a PostgreSQL database user name. The pg_ident.conf configuration file can be used to map principals to user names; for example, pgusername@realm could be mapped to just pgusername. Alternatively, you can use the full username@realm principal as the role name in PostgreSQL without any mapping.

PostgreSQL also supports mapping client principals to user names by just stripping the realm from the principal. This method is supported for backwards compatibility and is strongly discouraged as it is then impossible to distinguish different users with the same user name but coming from different realms. To enable this, set include_realm to 0. For simple single-realm installations, doing that combined with setting the krb_realm parameter (which checks that the principal's realm matches exactly what is in the krb_realm parameter) is still secure; but this is a less capable approach compared to specifying an explicit mapping in pg_ident.conf.

The location of the server's keytab file is specified by the krb_server_keyfile configuration parameter. For security reasons, it is recommended to use a separate keytab just for the PostgreSQL server rather than allowing the server to read the system keytab file. Make sure that your server keytab file is readable (and preferably only readable, not writable) by the PostgreSQL server account. (See also Section 19.1.)

The keytab file is generated using the Kerberos software; see the Kerberos documentation for details. The following example shows doing this using the kadmin tool of MIT-compatible Kerberos 5 implementations:

kadmin% addprinc -randkey postgres/server.my.domain.org
kadmin% ktadd -k krb5.keytab postgres/server.my.domain.org

The following authentication options are supported for the GSSAPI authentication method:

include_realm

If set to 0, the realm name from the authenticated user principal is stripped off before being passed through the user name mapping (Section 21.2). This is discouraged and is primarily available for backwards compatibility, as it is not secure in multi-realm environments unless krb_realm is also used. It is recommended to leave include_realm set to the default (1) and to provide an explicit mapping in pg_ident.conf to convert principal names to PostgreSQL user names.

map

Allows mapping from client principals to database user names. See Section 21.2 for details. For a GSSAPI/Kerberos principal, such as username@EXAMPLE.COM (or, less commonly, username/hostbased@EXAMPLE.COM), the user name used for mapping is username@EXAMPLE.COM (or username/hostbased@EXAMPLE.COM, respectively), unless include_realm has been set to 0, in which case username (or username/hostbased) is what is seen as the system user name when mapping.

krb_realm

Sets the realm to match user principal names against. If this parameter is set, only users of that realm will be accepted. If it is not set, users of any realm can connect, subject to whatever user name mapping is done.

In addition to these settings, which can be different for different pg_hba.conf entries, there is the server-wide krb_caseins_users configuration parameter. If that is set to true, client principals are matched to user map entries case-insensitively. krb_realm, if set, is also matched case-insensitively.

Prev | Up | Next
21.5. Password Authentication | Home | 21.7. SSPI Authentication


21.8. Ident Authentication
Prev Up Chapter 21. Client Authentication Home Next

21.8. Ident Authentication

The ident authentication method works by obtaining the client's operating system user name from an ident server and using it as the allowed database user name (with an optional user name mapping). This is only supported on TCP/IP connections.

Note

When ident is specified for a local (non-TCP/IP) connection, peer authentication (see Section 21.9) will be used instead.

The following configuration options are supported for ident:

map

Allows for mapping between system and database user names. See Section 21.2 for details.

The “Identification Protocol” is described in RFC 1413. Virtually every Unix-like operating system ships with an ident server that listens on TCP port 113 by default. The basic functionality of an ident server is to answer questions like “What user initiated the connection that goes out of your port X and connects to my port Y?”. Since PostgreSQL knows both X and Y when a physical connection is established, it can interrogate the ident server on the host of the connecting client and can theoretically determine the operating system user for any given connection.

The drawback of this procedure is that it depends on the integrity of the client: if the client machine is untrusted or compromised, an attacker could run just about any program on port 113 and return any user name they choose. This authentication method is therefore only appropriate for closed networks where each client machine is under tight control and where the database and system administrators operate in close contact. In other words, you must trust the machine running the ident server. Heed the warning:

The Identification Protocol is not intended as an authorization or access control protocol.

--RFC 1413

Some ident servers have a nonstandard option that causes the returned user name to be encrypted, using a key that only the originating machine's administrator knows. This option must not be used when using the ident server with PostgreSQL, since PostgreSQL does not have any way to decrypt the returned string to determine the actual user name.

Prev | Up | Next
21.7. SSPI Authentication | Home | 21.9. Peer Authentication


21.9. Peer Authentication
Prev Up Chapter 21. Client Authentication Home Next

21.9. Peer Authentication

The peer authentication method works by obtaining the client's operating system user name from the kernel and using it as the allowed database user name (with optional user name mapping). This method is only supported on local connections.

The following configuration options are supported for peer:

map

Allows for mapping between system and database user names. See Section 21.2 for details.

Peer authentication is only available on operating systems providing the getpeereid() function, the SO_PEERCRED socket parameter, or similar mechanisms. Currently that includes Linux, most flavors of BSD including macOS, and Solaris.

Prev | Up | Next
21.8. Ident Authentication | Home | 21.10. LDAP Authentication


21.10. LDAP Authentication
Prev Up Chapter 21. Client Authentication Home Next

21.10. LDAP Authentication

This authentication method operates similarly to password except that it uses LDAP as the password verification method. LDAP is used only to validate the user name/password pairs. Therefore the user must already exist in the database before LDAP can be used for authentication.

LDAP authentication can operate in two modes. In the first mode, which we will call the simple bind mode, the server will bind to the distinguished name constructed as prefix username suffix. Typically, the prefix parameter is used to specify cn=, or DOMAIN\ in an Active Directory environment. suffix is used to specify the remaining part of the DN in a non-Active Directory environment.

In the second mode, which we will call the search+bind mode, the server first binds to the LDAP directory with a fixed user name and password, specified with ldapbinddn and ldapbindpasswd, and performs a search for the user trying to log in to the database. If no user and password is configured, an anonymous bind will be attempted to the directory. The search will be performed over the subtree at ldapbasedn, and will try to do an exact match of the attribute specified in ldapsearchattribute. Once the user has been found in this search, the server disconnects and re-binds to the directory as this user, using the password specified by the client, to verify that the login is correct. This mode is the same as that used by LDAP authentication schemes in other software, such as Apache mod_authnz_ldap and pam_ldap. This method allows for significantly more flexibility in where the user objects are located in the directory, but will cause two separate connections to the LDAP server to be made.

The following configuration options are used in both modes:

ldapserver

Names or IP addresses of LDAP servers to connect to. Multiple servers may be specified, separated by spaces.

ldapport

Port number on LDAP server to connect to. If no port is specified, the LDAP library's default port setting will be used.

ldapscheme

Set to ldaps to use LDAPS. This is a non-standard way of using LDAP over SSL, supported by some LDAP server implementations. See also the ldaptls option for an alternative.

ldaptls

Set to 1 to make the connection between PostgreSQL and the LDAP server use TLS encryption. This uses the StartTLS operation per RFC 4513. See also the ldapscheme option for an alternative.

Note that using ldapscheme or ldaptls only encrypts the traffic between the PostgreSQL server and the LDAP server. The connection between the PostgreSQL server and the PostgreSQL client will still be unencrypted unless SSL is used there as well.

The following options are used in simple bind mode only:

ldapprefix

String to prepend to the user name when forming the DN to bind as, when doing simple bind authentication.

ldapsuffix

String to append to the user name when forming the DN to bind as, when doing simple bind authentication.

The following options are used in search+bind mode only:

ldapbasedn

Root DN to begin the search for the user in, when doing search+bind authentication.

ldapbinddn

DN of user to bind to the directory with to perform the search when doing search+bind authentication.

ldapbindpasswd

Password for user to bind to the directory with to perform the search when doing search+bind authentication.

ldapsearchattribute

Attribute to match against the user name in the search when doing search+bind authentication. If no attribute is specified, the uid attribute will be used.

ldapsearchfilter

The search filter to use when doing search+bind authentication. Occurrences of $username will be replaced with the user name. This allows for more flexible search filters than ldapsearchattribute.

ldapurl

An RFC 4516 LDAP URL. This is an alternative way to write some of the other LDAP options in a more compact and standard form. The format is

ldap[s]://host[:port]/basedn[?[attribute][?[scope][?[filter]]]]

scope must be one of base, one, sub, typically the last. (The default is base, which is normally not useful in this application.) attribute can nominate a single attribute, in which case it is used as a value for ldapsearchattribute. If attribute is empty then filter can be used as a value for ldapsearchfilter.

The URL scheme ldaps chooses the LDAPS method for making LDAP connections over SSL, equivalent to using ldapscheme=ldaps. To use encrypted LDAP connections using the StartTLS operation, use the normal URL scheme ldap and specify the ldaptls option in addition to ldapurl.

For non-anonymous binds, ldapbinddn and ldapbindpasswd must be specified as separate options.

LDAP URLs are currently only supported with OpenLDAP, not on Windows.

It is an error to mix configuration options for simple bind with options for search+bind.

When using search+bind mode, the search can be performed using a single attribute specified with ldapsearchattribute, or using a custom search filter specified with ldapsearchfilter. Specifying ldapsearchattribute=foo is equivalent to specifying ldapsearchfilter="(foo=$username)". If neither option is specified the default is ldapsearchattribute=uid.

If PostgreSQL was compiled with OpenLDAP as the LDAP client library, the ldapserver setting may be omitted. In that case, a list of host names and ports is looked up via RFC 2782 DNS SRV records. The name _ldap._tcp.DOMAIN is looked up, where DOMAIN is extracted from ldapbasedn.

Here is an example for a simple-bind LDAP configuration:

host ... ldap ldapserver=ldap.example.net ldapprefix="cn=" ldapsuffix=", dc=example, dc=net"

When a connection to the database server as database user someuser is requested, PostgreSQL will attempt to bind to the LDAP server using the DN cn=someuser, dc=example, dc=net and the password provided by the client. If that connection succeeds, the database access is granted.

Here is an example for a search+bind configuration:

host ... ldap ldapserver=ldap.example.net ldapbasedn="dc=example, dc=net" ldapsearchattribute=uid

When a connection to the database server as database user someuser is requested, PostgreSQL will attempt to bind anonymously (since ldapbinddn was not specified) to the LDAP server, perform a search for (uid=someuser) under the specified base DN. If an entry is found, it will then attempt to bind using that found information and the password supplied by the client. If that second connection succeeds, the database access is granted.

Here is the same search+bind configuration written as a URL:

host ... ldap ldapurl="ldap://ldap.example.net/dc=example,dc=net?uid?sub"

Some other software that supports authentication against LDAP uses the same URL format, so it will be easier to share the configuration.

Here is an example for a search+bind configuration that uses ldapsearchfilter instead of ldapsearchattribute to allow authentication by user ID or email address:

host ... ldap ldapserver=ldap.example.net ldapbasedn="dc=example, dc=net" ldapsearchfilter="(|(uid=$username)(mail=$username))"

Here is an example for a search+bind configuration that uses DNS SRV discovery to find the host name(s) and port(s) for the LDAP service for the domain name example.net:

host ... ldap ldapbasedn="dc=example,dc=net"

Tip

Since LDAP often uses commas and spaces to separate the different parts of a DN, it is often necessary to use double-quoted parameter values when configuring LDAP options, as shown in the examples.

Prev | Up | Next
21.9. Peer Authentication | Home | 21.11. RADIUS Authentication


21.11. RADIUS Authentication
Prev Up Chapter 21. Client Authentication Home Next

21.11. RADIUS Authentication

This authentication method operates similarly to password except that it uses RADIUS as the password verification method. RADIUS is used only to validate the user name/password pairs. Therefore the user must already exist in the database before RADIUS can be used for authentication.

When using RADIUS authentication, an Access Request message will be sent to the configured RADIUS server. This request will be of type Authenticate Only, and include parameters for user name, password (encrypted) and NAS Identifier. The request will be encrypted using a secret shared with the server. The RADIUS server will respond to this request with either Access Accept or Access Reject. There is no support for RADIUS accounting.

Multiple RADIUS servers can be specified, in which case they will be tried sequentially. If a negative response is received from a server, the authentication will fail. If no response is received, the next server in the list will be tried. To specify multiple servers, separate the server names with commas and surround the list with double quotes. If multiple servers are specified, the other RADIUS options can also be given as comma-separated lists, to provide individual values for each server. They can also be specified as a single value, in which case that value will apply to all servers.

The following configuration options are supported for RADIUS:

radiusservers

The DNS names or IP addresses of the RADIUS servers to connect to. This parameter is required.

radiussecrets

The shared secrets used when talking securely to the RADIUS servers. This must have exactly the same value on the PostgreSQL and RADIUS servers. It is recommended that this be a string of at least 16 characters. This parameter is required.

Note

The encryption vector used will only be cryptographically strong if PostgreSQL is built with support for OpenSSL. In other cases, the transmission to the RADIUS server should only be considered obfuscated, not secured, and external security measures should be applied if necessary.

radiusports

The port numbers to connect to on the RADIUS servers. If no port is specified, the default RADIUS port (1812) will be used.

radiusidentifiers

The strings to be used as NAS Identifier in the RADIUS requests. This parameter can be used, for example, to identify which database cluster the user is attempting to connect to, which can be useful for policy matching on the RADIUS server. If no identifier is specified, the default postgresql will be used.

If it is necessary to have a comma or whitespace in a RADIUS parameter value, that can be done by putting double quotes around the value, but it is tedious because two layers of double-quoting are now required. An example of putting whitespace into RADIUS secret strings is:

host ... radius radiusservers="server1,server2" radiussecrets="""secret one"",""secret two"""

Prev | Up | Next
21.10. LDAP Authentication | Home | 21.12. Certificate Authentication


21.12. Certificate Authentication
Prev Up Chapter 21. Client Authentication Home Next

21.12. Certificate Authentication

This authentication method uses SSL client certificates to perform authentication. It is therefore only available for SSL connections. When using this authentication method, the server will require that the client provide a valid, trusted certificate. No password prompt will be sent to the client. The cn (Common Name) attribute of the certificate will be compared to the requested database user name, and if they match the login will be allowed. User name mapping can be used to allow cn to be different from the database user name.

The following configuration options are supported for SSL certificate authentication:

map

Allows for mapping between system and database user names. See Section 21.2 for details.

It is redundant to use the clientcert option with cert authentication because cert authentication is effectively trust authentication with clientcert=verify-full.

Prev | Up | Next
21.11. RADIUS Authentication | Home | 21.13. PAM Authentication


21.13. PAM Authentication
Prev Up Chapter 21. Client Authentication Home Next

21.13. PAM Authentication

This authentication method operates similarly to password except that it uses PAM (Pluggable Authentication Modules) as the authentication mechanism. The default PAM service name is postgresql. PAM is used only to validate user name/password pairs and optionally the connected remote host name or IP address. Therefore the user must already exist in the database before PAM can be used for authentication. For more information about PAM, please read the Linux-PAM Page.

The following configuration options are supported for PAM:

pamservice

PAM service name.

pam_use_hostname

Determines whether the remote IP address or the host name is provided to PAM modules through the PAM_RHOST item. By default, the IP address is used. Set this option to 1 to use the resolved host name instead. Host name resolution can lead to login delays. (Most PAM configurations don't use this information, so it is only necessary to consider this setting if a PAM configuration was specifically created to make use of it.)

Note

If PAM is set up to read /etc/shadow, authentication will fail because the PostgreSQL server is started by a non-root user. However, this is not an issue when PAM is configured to use LDAP or other authentication methods.

Prev | Up | Next
21.12. Certificate Authentication | Home | 21.14. BSD Authentication


21.14. BSD Authentication
Prev Up Chapter 21. Client Authentication Home Next

21.14. BSD Authentication

This authentication method operates similarly to password except that it uses BSD Authentication to verify the password. BSD Authentication is used only to validate user name/password pairs. Therefore the user's role must already exist in the database before BSD Authentication can be used for authentication. The BSD Authentication framework is currently only available on OpenBSD.

BSD Authentication in PostgreSQL uses the auth-postgresql login type and authenticates with the postgresql login class if that's defined in login.conf. By default that login class does not exist, and PostgreSQL will use the default login class.

Note

To use BSD Authentication, the PostgreSQL user account (that is, the operating system user running the server) must first be added to the auth group. The auth group exists by default on OpenBSD systems.

Prev | Up | Next
21.13. PAM Authentication | Home | 21.15. Authentication Problems


21.15. Authentication Problems
Prev Up Chapter 21. Client Authentication Home Next

21.15. Authentication Problems

Authentication failures and related problems generally manifest themselves through error messages like the following:

FATAL:  no pg_hba.conf entry for host "123.123.123.123", user "andym", database "testdb"

This is what you are most likely to get if you succeed in contacting the server, but it does not want to talk to you. As the message suggests, the server refused the connection request because it found no matching entry in its pg_hba.conf configuration file.

FATAL:  password authentication failed for user "andym"

Messages like this indicate that you contacted the server, and it is willing to talk to you, but not until you pass the authorization method specified in the pg_hba.conf file. Check the password you are providing, or check your Kerberos or ident software if the complaint mentions one of those authentication types.

FATAL:  user "andym" does not exist

The indicated database user name was not found.

FATAL:  database "testdb" does not exist

The database you are trying to connect to does not exist. Note that if you do not specify a database name, it defaults to the database user name, which might or might not be the right thing.

Tip

The server log might contain more information about an authentication failure than is reported to the client. If you are confused about the reason for a failure, check the server log.

Prev | Up | Next
21.14. BSD Authentication | Home | Chapter 22. Database Roles


22.1. Database Roles
Prev Up Chapter 22. Database Roles Home Next

22.1. Database Roles

Database roles are conceptually completely separate from operating system users. In practice it might be convenient to maintain a correspondence, but this is not required. Database roles are global across a database cluster installation (and not per individual database). To create a role use the CREATE ROLE SQL command:

CREATE ROLE name;

name follows the rules for SQL identifiers: either unadorned without special characters, or double-quoted. (In practice, you will usually want to add additional options, such as LOGIN, to the command. More details appear below.) To remove an existing role, use the analogous DROP ROLE command:

DROP ROLE name;

For convenience, the programs createuser and dropuser are provided as wrappers around these SQL commands that can be called from the shell command line:

createuser name
dropuser name

To determine the set of existing roles, examine the pg_roles system catalog, for example

SELECT rolname FROM pg_roles;

The psql program's \du meta-command is also useful for listing the existing roles.

In order to bootstrap the database system, a freshly initialized system always contains one predefined role. This role is always a “superuser”, and by default (unless altered when running initdb) it will have the same name as the operating system user that initialized the database cluster. Customarily, this role will be named postgres. In order to create more roles you first have to connect as this initial role.

Every connection to the database server is made using the name of some particular role, and this role determines the initial access privileges for commands issued in that connection. The role name to use for a particular database connection is indicated by the client that is initiating the connection request in an application-specific fashion. For example, the psql program uses the -U command line option to indicate the role to connect as. Many applications assume the name of the current operating system user by default (including createuser and psql). Therefore it is often convenient to maintain a naming correspondence between roles and operating system users.

The set of database roles a given client connection can connect as is determined by the client authentication setup, as explained in Chapter 21. (Thus, a client is not limited to connect as the role matching its operating system user, just as a person's login name need not match his or her real name.) Since the role identity determines the set of privileges available to a connected client, it is important to carefully configure privileges when setting up a multiuser environment.

Prev | Up | Next
Chapter 22. Database Roles | Home | 22.2. Role Attributes


22.5. Predefined Roles
Prev Up Chapter 22. Database Roles Home Next

22.5. Predefined Roles

PostgreSQL provides a set of predefined roles that provide access to certain, commonly needed, privileged capabilities and information. Administrators (including roles that have the CREATEROLE privilege) can GRANT these roles to users and/or other roles in their environment, providing those users with access to the specified capabilities and information.

The predefined roles are described in Table 22.1. Note that the specific permissions for each of the roles may change in the future as additional capabilities are added. Administrators should monitor the release notes for changes.

Table 22.1. Predefined Roles

Role | Allowed Access
pg_read_all_data | Read all data (tables, views, sequences), as if having SELECT rights on those objects, and USAGE rights on all schemas, even without having it explicitly. This role does not have the role attribute BYPASSRLS set. If RLS is being used, an administrator may wish to set BYPASSRLS on roles which this role is GRANTed to.
pg_write_all_data | Write all data (tables, views, sequences), as if having INSERT, UPDATE, and DELETE rights on those objects, and USAGE rights on all schemas, even without having it explicitly. This role does not have the role attribute BYPASSRLS set. If RLS is being used, an administrator may wish to set BYPASSRLS on roles which this role is GRANTed to.
pg_read_all_settings | Read all configuration variables, even those normally visible only to superusers.
pg_read_all_stats | Read all pg_stat_* views and use various statistics related extensions, even those normally visible only to superusers.
pg_stat_scan_tables | Execute monitoring functions that may take ACCESS SHARE locks on tables, potentially for a long time.
pg_monitor | Read/execute various monitoring views and functions. This role is a member of pg_read_all_settings, pg_read_all_stats and pg_stat_scan_tables.
pg_database_owner | None. Membership consists, implicitly, of the current database owner.
pg_signal_backend | Signal another backend to cancel a query or terminate its session.
pg_read_server_files | Allow reading files from any location the database can access on the server with COPY and other file-access functions.
pg_write_server_files | Allow writing to files in any location the database can access on the server with COPY and other file-access functions.
pg_execute_server_program | Allow executing programs on the database server as the user the database runs as with COPY and other functions which allow executing a server-side program.

The pg_monitor, pg_read_all_settings, pg_read_all_stats and pg_stat_scan_tables roles are intended to allow administrators to easily configure a role for the purpose of monitoring the database server. They grant a set of common privileges allowing the role to read various useful configuration settings, statistics and other system information normally restricted to superusers.

The pg_database_owner role has one implicit, situation-dependent member, namely the owner of the current database. The role conveys no rights at first. Like any role, it can own objects or receive grants of access privileges. Consequently, once pg_database_owner has rights within a template database, each owner of a database instantiated from that template will exercise those rights. pg_database_owner cannot be a member of any role, and it cannot have non-implicit members.

The pg_signal_backend role is intended to allow administrators to enable trusted, but non-superuser, roles to send signals to other backends. Currently this role enables sending of signals for canceling a query on another backend or terminating its session. A user granted this role cannot however send signals to a backend owned by a superuser. See Section 9.27.2.

The pg_read_server_files, pg_write_server_files and pg_execute_server_program roles are intended to allow administrators to have trusted, but non-superuser, roles which are able to access files and run programs on the database server as the user the database runs as. As these roles are able to access any file on the server file system, they bypass all database-level permission checks when accessing files directly and they could be used to gain superuser-level access, therefore great care should be taken when granting these roles to users.

Care should be taken when granting these roles to ensure they are only used where needed and with the understanding that these roles grant access to privileged information.

Administrators can grant access to these roles to users using the GRANT command, for example:

GRANT pg_signal_backend TO admin_user;

Prev | Up | Next
22.4. Dropping Roles | Home | 22.6. Function Security


22.6. Function Security
Prev Up Chapter 22. Database Roles Home Next

22.6. Function Security

Functions, triggers and row-level security policies allow users to insert code into the backend server that other users might execute unintentionally. Hence, these mechanisms permit users to “Trojan horse” others with relative ease. The strongest protection is tight control over who can define objects. Where that is infeasible, write queries referring only to objects having trusted owners. Remove from search_path the public schema and any other schemas that permit untrusted users to create objects.

Functions run inside the backend server process with the operating system permissions of the database server daemon. If the programming language used for the function allows unchecked memory accesses, it is possible to change the server's internal data structures. Hence, among many other things, such functions can circumvent any system access controls. Function languages that allow such access are considered “untrusted”, and PostgreSQL allows only superusers to create functions written in those languages.

Prev | Up | Next
22.5. Predefined Roles | Home | Chapter 23. Managing Databases


Chapter 23. Managing Databases
Prev Up Part III. Server Administration Home Next

Chapter 23. Managing Databases

Table of Contents

23.1. Overview
23.2. Creating a Database
23.3. Template Databases
23.4. Database Configuration
23.5. Destroying a Database
23.6. Tablespaces

Every instance of a running PostgreSQL server manages one or more databases. Databases are therefore the topmost hierarchical level for organizing SQL objects (“database objects”). This chapter describes the properties of databases, and how to create, manage, and destroy them.

Prev | Up | Next
22.6. Function Security | Home | 23.1. Overview


23.1. Overview
Prev Up Chapter 23. Managing Databases Home Next

23.1. Overview

A small number of objects, like role, database, and tablespace names, are defined at the cluster level and stored in the pg_global tablespace. Inside the cluster are multiple databases, which are isolated from each other but can access cluster-level objects. Inside each database are multiple schemas, which contain objects like tables and functions. So the full hierarchy is: cluster, database, schema, table (or some other kind of object, such as a function).

When connecting to the database server, a client must specify the database name in its connection request. It is not possible to access more than one database per connection. However, clients can open multiple connections to the same database, or different databases. Database-level security has two components: access control (see Section 21.1), managed at the connection level, and authorization control (see Section 5.7), managed via the grant system. Foreign data wrappers (see postgres_fdw) allow for objects within one database to act as proxies for objects in other database or clusters. The older dblink module (see dblink) provides a similar capability. By default, all users can connect to all databases using all connection methods.

If one PostgreSQL server cluster is planned to contain unrelated projects or users that should be, for the most part, unaware of each other, it is recommended to put them into separate databases and adjust authorizations and access controls accordingly. If the projects or users are interrelated, and thus should be able to use each other's resources, they should be put in the same database but probably into separate schemas; this provides a modular structure with namespace isolation and authorization control. More information about managing schemas is in Section 5.9.

While multiple databases can be created within a single cluster, it is advised to consider carefully whether the benefits outweigh the risks and limitations. In particular, the impact that having a shared WAL (see Chapter 30) has on backup and recovery options. While individual databases in the cluster are isolated when considered from the user's perspective, they are closely bound from the database administrator's point-of-view.

Databases are created with the CREATE DATABASE command (see Section 23.2) and destroyed with the DROP DATABASE command (see Section 23.5). To determine the set of existing databases, examine the pg_database system catalog, for example

SELECT datname FROM pg_database;

The psql program's \l meta-command and -l command-line option are also useful for listing the existing databases.

Note

The SQL standard calls databases “catalogs”, but there is no difference in practice.

Prev | Up | Next
Chapter 23. Managing Databases | Home | 23.2. Creating a Database


23.2. Creating a Database
Prev Up Chapter 23. Managing Databases Home Next

23.2. Creating a Database

In order to create a database, the PostgreSQL server must be up and running (see Section 19.3).

Databases are created with the SQL command CREATE DATABASE:

CREATE DATABASE name;

where name follows the usual rules for SQL identifiers. The current role automatically becomes the owner of the new database. It is the privilege of the owner of a database to remove it later (which also removes all the objects in it, even if they have a different owner).

The creation of databases is a restricted operation. See Section 22.2 for how to grant permission.

Since you need to be connected to the database server in order to execute the CREATE DATABASE command, the question remains how the first database at any given site can be created. The first database is always created by the initdb command when the data storage area is initialized. (See Section 19.2.) This database is called postgres. So to create the first “ordinary” database you can connect to postgres.

A second database, template1, is also created during database cluster initialization. Whenever a new database is created within the cluster, template1 is essentially cloned. This means that any changes you make in template1 are propagated to all subsequently created databases. Because of this, avoid creating objects in template1 unless you want them propagated to every newly created database. More details appear in Section 23.3.

As a convenience, there is a program you can execute from the shell to create new databases, createdb.

createdb dbname

createdb does no magic. It connects to the postgres database and issues the CREATE DATABASE command, exactly as described above. The createdb reference page contains the invocation details. Note that createdb without any arguments will create a database with the current user name.

Note

Chapter 21 contains information about how to restrict who can connect to a given database.

Sometimes you want to create a database for someone else, and have them become the owner of the new database, so they can configure and manage it themselves. To achieve that, use one of the following commands:

CREATE DATABASE dbname OWNER rolename;

from the SQL environment, or:

createdb -O rolename dbname

from the shell. Only the superuser is allowed to create a database for someone else (that is, for a role you are not a member of).

Prev | Up | Next
23.1. Overview | Home | 23.3. Template Databases


23.3. Template Databases
Prev Up Chapter 23. Managing Databases Home Next

23.3. Template Databases

CREATE DATABASE actually works by copying an existing database. By default, it copies the standard system database named template1. Thus that database is the “template” from which new databases are made. If you add objects to template1, these objects will be copied into subsequently created user databases. This behavior allows site-local modifications to the standard set of objects in databases. For example, if you install the procedural language PL/Perl in template1, it will automatically be available in user databases without any extra action being taken when those databases are created.

There is a second standard system database named template0. This database contains the same data as the initial contents of template1, that is, only the standard objects predefined by your version of PostgreSQL. template0 should never be changed after the database cluster has been initialized. By instructing CREATE DATABASE to copy template0 instead of template1, you can create a “pristine” user database (one where no user-defined objects exist and where the system objects have not been altered) that contains none of the site-local additions in template1. This is particularly handy when restoring a pg_dump dump: the dump script should be restored in a pristine database to ensure that one recreates the correct contents of the dumped database, without conflicting with objects that might have been added to template1 later on.

Another common reason for copying template0 instead of template1 is that new encoding and locale settings can be specified when copying template0, whereas a copy of template1 must use the same settings it does. This is because template1 might contain encoding-specific or locale-specific data, while template0 is known not to.

To create a database by copying template0, use:

CREATE DATABASE dbname TEMPLATE template0;

from the SQL environment, or:

createdb -T template0 dbname

from the shell.

It is possible to create additional template databases, and indeed one can copy any database in a cluster by specifying its name as the template for CREATE DATABASE. It is important to understand, however, that this is not (yet) intended as a general-purpose “COPY DATABASE” facility. The principal limitation is that no other sessions can be connected to the source database while it is being copied. CREATE DATABASE will fail if any other connection exists when it starts; during the copy operation, new connections to the source database are prevented.

Two useful flags exist in pg_database for each database: the columns datistemplate and datallowconn. datistemplate can be set to indicate that a database is intended as a template for CREATE DATABASE. If this flag is set, the database can be cloned by any user with CREATEDB privileges; if it is not set, only superusers and the owner of the database can clone it. If datallowconn is false, then no new connections to that database will be allowed (but existing sessions are not terminated simply by setting the flag false). The template0 database is normally marked datallowconn = false to prevent its modification. Both template0 and template1 should always be marked with datistemplate = true.

Note

template1 and template0 do not have any special status beyond the fact that the name template1 is the default source database name for CREATE DATABASE. For example, one could drop template1 and recreate it from template0 without any ill effects. This course of action might be advisable if one has carelessly added a bunch of junk in template1. (To delete template1, it must have pg_database.datistemplate = false.)

The postgres database is also created when a database cluster is initialized. This database is meant as a default database for users and applications to connect to. It is simply a copy of template1 and can be dropped and recreated if necessary.

Prev | Up | Next
23.2. Creating a Database | Home | 23.4. Database Configuration


23.4. Database Configuration
Prev Up Chapter 23. Managing Databases Home Next

23.4. Database Configuration

Recall from Chapter 20 that the PostgreSQL server provides a large number of run-time configuration variables. You can set database-specific default values for many of these settings.

For example, if for some reason you want to disable the GEQO optimizer for a given database, you'd ordinarily have to either disable it for all databases or make sure that every connecting client is careful to issue SET geqo TO off. To make this setting the default within a particular database, you can execute the command:

ALTER DATABASE mydb SET geqo TO off;

This will save the setting (but not set it immediately). In subsequent connections to this database it will appear as though SET geqo TO off; had been executed just before the session started. Note that users can still alter this setting during their sessions; it will only be the default. To undo any such setting, use ALTER DATABASE dbname RESET varname.

Prev | Up | Next
23.3. Template Databases | Home | 23.5. Destroying a Database


23.5. Destroying a Database
Prev Up Chapter 23. Managing Databases Home Next

23.5. Destroying a Database

Databases are destroyed with the command DROP DATABASE:

DROP DATABASE name;

Only the owner of the database, or a superuser, can drop a database. Dropping a database removes all objects that were contained within the database. The destruction of a database cannot be undone.

You cannot execute the DROP DATABASE command while connected to the victim database. You can, however, be connected to any other database, including the template1 database. template1 would be the only option for dropping the last user database of a given cluster.

For convenience, there is also a shell program to drop databases, dropdb:

dropdb dbname

(Unlike createdb, it is not the default action to drop the database with the current user name.)

Prev | Up | Next
23.4. Database Configuration | Home | 23.6. Tablespaces


23.6. Tablespaces
Prev Up Chapter 23. Managing Databases Home Next

23.6. Tablespaces

Tablespaces in PostgreSQL allow database administrators to define locations in the file system where the files representing database objects can be stored. Once created, a tablespace can be referred to by name when creating database objects.

By using tablespaces, an administrator can control the disk layout of a PostgreSQL installation. This is useful in at least two ways. First, if the partition or volume on which the cluster was initialized runs out of space and cannot be extended, a tablespace can be created on a different partition and used until the system can be reconfigured.

Second, tablespaces allow an administrator to use knowledge of the usage pattern of database objects to optimize performance. For example, an index which is very heavily used can be placed on a very fast, highly available disk, such as an expensive solid state device. At the same time a table storing archived data which is rarely used or not performance critical could be stored on a less expensive, slower disk system.

Warning

Even though located outside the main PostgreSQL data directory, tablespaces are an integral part of the database cluster and cannot be treated as an autonomous collection of data files. They are dependent on metadata contained in the main data directory, and therefore cannot be attached to a different database cluster or backed up individually. Similarly, if you lose a tablespace (file deletion, disk failure, etc), the database cluster might become unreadable or unable to start. Placing a tablespace on a temporary file system like a RAM disk risks the reliability of the entire cluster.

To define a tablespace, use the CREATE TABLESPACE command, for example::

CREATE TABLESPACE fastspace LOCATION '/ssd1/postgresql/data';

The location must be an existing, empty directory that is owned by the PostgreSQL operating system user. All objects subsequently created within the tablespace will be stored in files underneath this directory. The location must not be on removable or transient storage, as the cluster might fail to function if the tablespace is missing or lost.

Note

There is usually not much point in making more than one tablespace per logical file system, since you cannot control the location of individual files within a logical file system. However, PostgreSQL does not enforce any such limitation, and indeed it is not directly aware of the file system boundaries on your system. It just stores files in the directories you tell it to use.

Creation of the tablespace itself must be done as a database superuser, but after that you can allow ordinary database users to use it. To do that, grant them the CREATE privilege on it.

Tables, indexes, and entire databases can be assigned to particular tablespaces. To do so, a user with the CREATE privilege on a given tablespace must pass the tablespace name as a parameter to the relevant command. For example, the following creates a table in the tablespace space1:

CREATE TABLE foo(i int) TABLESPACE space1;

Alternatively, use the default_tablespace parameter:

SET default_tablespace = space1;
CREATE TABLE foo(i int);

When default_tablespace is set to anything but an empty string, it supplies an implicit TABLESPACE clause for CREATE TABLE and CREATE INDEX commands that do not have an explicit one.

There is also a temp_tablespaces parameter, which determines the placement of temporary tables and indexes, as well as temporary files that are used for purposes such as sorting large data sets. This can be a list of tablespace names, rather than only one, so that the load associated with temporary objects can be spread over multiple tablespaces. A random member of the list is picked each time a temporary object is to be created.

The tablespace associated with a database is used to store the system catalogs of that database. Furthermore, it is the default tablespace used for tables, indexes, and temporary files created within the database, if no TABLESPACE clause is given and no other selection is specified by default_tablespace or temp_tablespaces (as appropriate). If a database is created without specifying a tablespace for it, it uses the same tablespace as the template database it is copied from.

Two tablespaces are automatically created when the database cluster is initialized. The pg_global tablespace is used for shared system catalogs. The pg_default tablespace is the default tablespace of the template1 and template0 databases (and, therefore, will be the default tablespace for other databases as well, unless overridden by a TABLESPACE clause in CREATE DATABASE).

Once created, a tablespace can be used from any database, provided the requesting user has sufficient privilege. This means that a tablespace cannot be dropped until all objects in all databases using the tablespace have been removed.

To remove an empty tablespace, use the DROP TABLESPACE command.

To determine the set of existing tablespaces, examine the pg_tablespace system catalog, for example

SELECT spcname FROM pg_tablespace;

The psql program's \db meta-command is also useful for listing the existing tablespaces.

PostgreSQL makes use of symbolic links to simplify the implementation of tablespaces. This means that tablespaces can be used only on systems that support symbolic links.

The directory $PGDATA/pg_tblspc contains symbolic links that point to each of the non-built-in tablespaces defined in the cluster. Although not recommended, it is possible to adjust the tablespace layout by hand by redefining these links. Under no circumstances perform this operation while the server is running. Note that in PostgreSQL 9.1 and earlier you will also need to update the pg_tablespace catalog with the new locations. (If you do not, pg_dump will continue to output the old tablespace locations.)

Prev | Up | Next
23.5. Destroying a Database | Home | Chapter 24. Localization


Chapter 24. Localization
Prev Up Part III. Server Administration Home Next

Chapter 24. Localization

Table of Contents

24.1. Locale Support
24.1.1. Overview
24.1.2. Behavior
24.1.3. Problems
24.2. Collation Support
24.2.1. Concepts
24.2.2. Managing Collations
24.3. Character Set Support
24.3.1. Supported Character Sets
24.3.2. Setting the Character Set
24.3.3. Automatic Character Set Conversion Between Server and Client
24.3.4. Available Character Set Conversions
24.3.5. Further Reading

This chapter describes the available localization features from the point of view of the administrator. PostgreSQL supports two localization facilities:

  • Using the locale features of the operating system to provide locale-specific collation order, number formatting, translated messages, and other aspects. This is covered in Section 24.1 and Section 24.2.

  • Providing a number of different character sets to support storing text in all kinds of languages, and providing character set translation between client and server. This is covered in Section 24.3.

Prev | Up | Next
23.6. Tablespaces | Home | 24.1. Locale Support


24.1. Locale Support
Prev Up Chapter 24. Localization Home Next

24.1. Locale Support

24.1.1. Overview
24.1.2. Behavior
24.1.3. Problems

Locale support refers to an application respecting cultural preferences regarding alphabets, sorting, number formatting, etc. PostgreSQL uses the standard ISO C and POSIX locale facilities provided by the server operating system. For additional information refer to the documentation of your system.

24.1.1. Overview

Locale support is automatically initialized when a database cluster is created using initdb. initdb will initialize the database cluster with the locale setting of its execution environment by default, so if your system is already set to use the locale that you want in your database cluster then there is nothing else you need to do. If you want to use a different locale (or you are not sure which locale your system is set to), you can instruct initdb exactly which locale to use by specifying the --locale option. For example:

initdb --locale=sv_SE

This example for Unix systems sets the locale to Swedish (sv) as spoken in Sweden (SE). Other possibilities might include en_US (U.S. English) and fr_CA (French Canadian). If more than one character set can be used for a locale then the specifications can take the form language_territory.codeset. For example, fr_BE.UTF-8 represents the French language (fr) as spoken in Belgium (BE), with a UTF-8 character set encoding.

What locales are available on your system under what names depends on what was provided by the operating system vendor and what was installed. On most Unix systems, the command locale -a will provide a list of available locales. Windows uses more verbose locale names, such as German_Germany or Swedish_Sweden.1252, but the principles are the same.

Occasionally it is useful to mix rules from several locales, e.g., use English collation rules but Spanish messages. To support that, a set of locale subcategories exist that control only certain aspects of the localization rules:

LC_COLLATE | String sort order
LC_CTYPE | Character classification (What is a letter? Its upper-case equivalent?)
LC_MESSAGES | Language of messages
LC_MONETARY | Formatting of currency amounts
LC_NUMERIC | Formatting of numbers
LC_TIME | Formatting of dates and times

The category names translate into names of initdb options to override the locale choice for a specific category. For instance, to set the locale to French Canadian, but use U.S. rules for formatting currency, use initdb --locale=fr_CA --lc-monetary=en_US.

If you want the system to behave as if it had no locale support, use the special locale name C, or equivalently POSIX.

Some locale categories must have their values fixed when the database is created. You can use different settings for different databases, but once a database is created, you cannot change them for that database anymore. LC_COLLATE and LC_CTYPE are these categories. They affect the sort order of indexes, so they must be kept fixed, or indexes on text columns would become corrupt. (But you can alleviate this restriction using collations, as discussed in Section 24.2.) The default values for these categories are determined when initdb is run, and those values are used when new databases are created, unless specified otherwise in the CREATE DATABASE command.

The other locale categories can be changed whenever desired by setting the server configuration parameters that have the same name as the locale categories (see Section 20.11.2 for details). The values that are chosen by initdb are actually only written into the configuration file postgresql.conf to serve as defaults when the server is started. If you remove these assignments from postgresql.conf then the server will inherit the settings from its execution environment.

Note that the locale behavior of the server is determined by the environment variables seen by the server, not by the environment of any client. Therefore, be careful to configure the correct locale settings before starting the server. A consequence of this is that if client and server are set up in different locales, messages might appear in different languages depending on where they originated.

Note

When we speak of inheriting the locale from the execution environment, this means the following on most operating systems: For a given locale category, say the collation, the following environment variables are consulted in this order until one is found to be set: LC_ALL, LC_COLLATE (or the variable corresponding to the respective category), LANG. If none of these environment variables are set then the locale defaults to C.

Some message localization libraries also look at the environment variable LANGUAGE which overrides all other locale settings for the purpose of setting the language of messages. If in doubt, please refer to the documentation of your operating system, in particular the documentation about gettext.

To enable messages to be translated to the user's preferred language, NLS must have been selected at build time (configure --enable-nls). All other locale support is built in automatically.

24.1.2. Behavior

The locale settings influence the following SQL features:

  • Sort order in queries using ORDER BY or the standard comparison operators on textual data

  • The upper, lower, and initcap functions

  • Pattern matching operators (LIKE, SIMILAR TO, and POSIX-style regular expressions); locales affect both case insensitive matching and the classification of characters by character-class regular expressions

  • The to_char family of functions

  • The ability to use indexes with LIKE clauses

The drawback of using locales other than C or POSIX in PostgreSQL is its performance impact. It slows character handling and prevents ordinary indexes from being used by LIKE. For this reason use locales only if you actually need them.

As a workaround to allow PostgreSQL to use indexes with LIKE clauses under a non-C locale, several custom operator classes exist. These allow the creation of an index that performs a strict character-by-character comparison, ignoring locale comparison rules. Refer to Section 11.10 for more information. Another approach is to create indexes using the C collation, as discussed in Section 24.2.

24.1.3. Problems

If locale support doesn't work according to the explanation above, check that the locale support in your operating system is correctly configured. To check what locales are installed on your system, you can use the command locale -a if your operating system provides it.

Check that PostgreSQL is actually using the locale that you think it is. The LC_COLLATE and LC_CTYPE settings are determined when a database is created, and cannot be changed except by creating a new database. Other locale settings including LC_MESSAGES and LC_MONETARY are initially determined by the environment the server is started in, but can be changed on-the-fly. You can check the active locale settings using the SHOW command.

The directory src/test/locale in the source distribution contains a test suite for PostgreSQL's locale support.

Client applications that handle server-side errors by parsing the text of the error message will obviously have problems when the server's messages are in a different language. Authors of such applications are advised to make use of the error code scheme instead.

Maintaining catalogs of message translations requires the on-going efforts of many volunteers that want to see PostgreSQL speak their preferred language well. If messages in your language are currently not available or not fully translated, your assistance would be appreciated. If you want to help, refer to Chapter 55 or write to the developers' mailing list.

Prev | Up | Next
Chapter 24. Localization | Home | 24.2. Collation Support


24.2. Collation Support
Prev Up Chapter 24. Localization Home Next

24.2. Collation Support

24.2.1. Concepts
24.2.2. Managing Collations

The collation feature allows specifying the sort order and character classification behavior of data per-column, or even per-operation. This alleviates the restriction that the LC_COLLATE and LC_CTYPE settings of a database cannot be changed after its creation.

24.2.1. Concepts

Conceptually, every expression of a collatable data type has a collation. (The built-in collatable data types are text, varchar, and char. User-defined base types can also be marked collatable, and of course a domain over a collatable data type is collatable.) If the expression is a column reference, the collation of the expression is the defined collation of the column. If the expression is a constant, the collation is the default collation of the data type of the constant. The collation of a more complex expression is derived from the collations of its inputs, as described below.

The collation of an expression can be the “default” collation, which means the locale settings defined for the database. It is also possible for an expression's collation to be indeterminate. In such cases, ordering operations and other operations that need to know the collation will fail.

When the database system has to perform an ordering or a character classification, it uses the collation of the input expression. This happens, for example, with ORDER BY clauses and function or operator calls such as <. The collation to apply for an ORDER BY clause is simply the collation of the sort key. The collation to apply for a function or operator call is derived from the arguments, as described below. In addition to comparison operators, collations are taken into account by functions that convert between lower and upper case letters, such as lower, upper, and initcap; by pattern matching operators; and by to_char and related functions.

For a function or operator call, the collation that is derived by examining the argument collations is used at run time for performing the specified operation. If the result of the function or operator call is of a collatable data type, the collation is also used at parse time as the defined collation of the function or operator expression, in case there is a surrounding expression that requires knowledge of its collation.

The collation derivation of an expression can be implicit or explicit. This distinction affects how collations are combined when multiple different collations appear in an expression. An explicit collation derivation occurs when a COLLATE clause is used; all other collation derivations are implicit. When multiple collations need to be combined, for example in a function call, the following rules are used:

  1. If any input expression has an explicit collation derivation, then all explicitly derived collations among the input expressions must be the same, otherwise an error is raised. If any explicitly derived collation is present, that is the result of the collation combination.

  2. Otherwise, all input expressions must have the same implicit collation derivation or the default collation. If any non-default collation is present, that is the result of the collation combination. Otherwise, the result is the default collation.

  3. If there are conflicting non-default implicit collations among the input expressions, then the combination is deemed to have indeterminate collation. This is not an error condition unless the particular function being invoked requires knowledge of the collation it should apply. If it does, an error will be raised at run-time.

For example, consider this table definition:

CREATE TABLE test1 (
    a text COLLATE "de_DE",
    b text COLLATE "es_ES",
    ...
);

Then in

SELECT a < 'foo' FROM test1;

the < comparison is performed according to de_DE rules, because the expression combines an implicitly derived collation with the default collation. But in

SELECT a < ('foo' COLLATE "fr_FR") FROM test1;

the comparison is performed using fr_FR rules, because the explicit collation derivation overrides the implicit one. Furthermore, given

SELECT a < b FROM test1;

the parser cannot determine which collation to apply, since the a and b columns have conflicting implicit collations. Since the < operator does need to know which collation to use, this will result in an error. The error can be resolved by attaching an explicit collation specifier to either input expression, thus:

SELECT a < b COLLATE "de_DE" FROM test1;

or equivalently

SELECT a COLLATE "de_DE" < b FROM test1;

On the other hand, the structurally similar case

SELECT a || b FROM test1;

does not result in an error, because the || operator does not care about collations: its result is the same regardless of the collation.

The collation assigned to a function or operator's combined input expressions is also considered to apply to the function or operator's result, if the function or operator delivers a result of a collatable data type. So, in

SELECT * FROM test1 ORDER BY a || 'foo';

the ordering will be done according to de_DE rules. But this query:

SELECT * FROM test1 ORDER BY a || b;

results in an error, because even though the || operator doesn't need to know a collation, the ORDER BY clause does. As before, the conflict can be resolved with an explicit collation specifier:

SELECT * FROM test1 ORDER BY a || b COLLATE "fr_FR";

24.2.2. Managing Collations

A collation is an SQL schema object that maps an SQL name to locales provided by libraries installed in the operating system. A collation definition has a provider that specifies which library supplies the locale data. One standard provider name is libc, which uses the locales provided by the operating system C library. These are the locales that most tools provided by the operating system use. Another provider is icu, which uses the external ICU library. ICU locales can only be used if support for ICU was configured when PostgreSQL was built.

A collation object provided by libc maps to a combination of LC_COLLATE and LC_CTYPE settings, as accepted by the setlocale() system library call. (As the name would suggest, the main purpose of a collation is to set LC_COLLATE, which controls the sort order. But it is rarely necessary in practice to have an LC_CTYPE setting that is different from LC_COLLATE, so it is more convenient to collect these under one concept than to create another infrastructure for setting LC_CTYPE per expression.) Also, a libc collation is tied to a character set encoding (see Section 24.3). The same collation name may exist for different encodings.

A collation object provided by icu maps to a named collator provided by the ICU library. ICU does not support separate “collate” and “ctype” settings, so they are always the same. Also, ICU collations are independent of the encoding, so there is always only one ICU collation of a given name in a database.

24.2.2.1. Standard Collations

On all platforms, the collations named default, C, and POSIX are available. Additional collations may be available depending on operating system support. The default collation selects the LC_COLLATE and LC_CTYPE values specified at database creation time. The C and POSIX collations both specify “traditional C” behavior, in which only the ASCII letters “A” through “Z” are treated as letters, and sorting is done strictly by character code byte values.

Additionally, the SQL standard collation name ucs_basic is available for encoding UTF8. It is equivalent to C and sorts by Unicode code point.

24.2.2.2. Predefined Collations

If the operating system provides support for using multiple locales within a single program (newlocale and related functions), or if support for ICU is configured, then when a database cluster is initialized, initdb populates the system catalog pg_collation with collations based on all the locales it finds in the operating system at the time.

To inspect the currently available locales, use the query SELECT * FROM pg_collation, or the command \dOS+ in psql.

24.2.2.2.1. libc Collations

For example, the operating system might provide a locale named de_DE.utf8. initdb would then create a collation named de_DE.utf8 for encoding UTF8 that has both LC_COLLATE and LC_CTYPE set to de_DE.utf8. It will also create a collation with the .utf8 tag stripped off the name. So you could also use the collation under the name de_DE, which is less cumbersome to write and makes the name less encoding-dependent. Note that, nevertheless, the initial set of collation names is platform-dependent.

The default set of collations provided by libc map directly to the locales installed in the operating system, which can be listed using the command locale -a. In case a libc collation is needed that has different values for LC_COLLATE and LC_CTYPE, or if new locales are installed in the operating system after the database system was initialized, then a new collation may be created using the CREATE COLLATION command. New operating system locales can also be imported en masse using the pg_import_system_collations() function.

Within any particular database, only collations that use that database's encoding are of interest. Other entries in pg_collation are ignored. Thus, a stripped collation name such as de_DE can be considered unique within a given database even though it would not be unique globally. Use of the stripped collation names is recommended, since it will make one fewer thing you need to change if you decide to change to another database encoding. Note however that the default, C, and POSIX collations can be used regardless of the database encoding.

PostgreSQL considers distinct collation objects to be incompatible even when they have identical properties. Thus for example,

SELECT a COLLATE "C" < b COLLATE "POSIX" FROM test1;

will draw an error even though the C and POSIX collations have identical behaviors. Mixing stripped and non-stripped collation names is therefore not recommended.

24.2.2.2.2. ICU Collations

With ICU, it is not sensible to enumerate all possible locale names. ICU uses a particular naming system for locales, but there are many more ways to name a locale than there are actually distinct locales. initdb uses the ICU APIs to extract a set of distinct locales to populate the initial set of collations. Collations provided by ICU are created in the SQL environment with names in BCP 47 language tag format, with a “private use” extension -x-icu appended, to distinguish them from libc locales.

Here are some example collations that might be created:

de-x-icu

German collation, default variant

de-AT-x-icu

German collation for Austria, default variant

(There are also, say, de-DE-x-icu or de-CH-x-icu, but as of this writing, they are equivalent to de-x-icu.)

und-x-icu (for “undefined”)

ICU “root” collation. Use this to get a reasonable language-agnostic sort order.

Some (less frequently used) encodings are not supported by ICU. When the database encoding is one of these, ICU collation entries in pg_collation are ignored. Attempting to use one will draw an error along the lines of “collation "de-x-icu" for encoding "WIN874" does not exist”.

24.2.2.3. Creating New Collation Objects

If the standard and predefined collations are not sufficient, users can create their own collation objects using the SQL command CREATE COLLATION.

The standard and predefined collations are in the schema pg_catalog, like all predefined objects. User-defined collations should be created in user schemas. This also ensures that they are saved by pg_dump.

24.2.2.3.1. libc Collations

New libc collations can be created like this:

CREATE COLLATION german (provider = libc, locale = 'de_DE');

The exact values that are acceptable for the locale clause in this command depend on the operating system. On Unix-like systems, the command locale -a will show a list.

Since the predefined libc collations already include all collations defined in the operating system when the database instance is initialized, it is not often necessary to manually create new ones. Reasons might be if a different naming system is desired (in which case see also Section 24.2.2.3.3) or if the operating system has been upgraded to provide new locale definitions (in which case see also pg_import_system_collations()).

24.2.2.3.2. ICU Collations

ICU allows collations to be customized beyond the basic language+country set that is preloaded by initdb. Users are encouraged to define their own collation objects that make use of these facilities to suit the sorting behavior to their requirements. See https://unicode-org.github.io/icu/userguide/locale/ and https://unicode-org.github.io/icu/userguide/collation/api.html for information on ICU locale naming. The set of acceptable names and attributes depends on the particular ICU version.

Here are some examples:

CREATE COLLATION "de-u-co-phonebk-x-icu" (provider = icu, locale = 'de-u-co-phonebk');
CREATE COLLATION "de-u-co-phonebk-x-icu" (provider = icu, locale = 'de@collation=phonebook');

German collation with phone book collation type

The first example selects the ICU locale using a “language tag” per BCP 47. The second example uses the traditional ICU-specific locale syntax. The first style is preferred going forward, but it is not supported by older ICU versions.

Note that you can name the collation objects in the SQL environment anything you want. In this example, we follow the naming style that the predefined collations use, which in turn also follow BCP 47, but that is not required for user-defined collations.

CREATE COLLATION "und-u-co-emoji-x-icu" (provider = icu, locale = 'und-u-co-emoji');
CREATE COLLATION "und-u-co-emoji-x-icu" (provider = icu, locale = '@collation=emoji');

Root collation with Emoji collation type, per Unicode Technical Standard #51

Observe how in the traditional ICU locale naming system, the root locale is selected by an empty string.

CREATE COLLATION latinlast (provider = icu, locale = 'en-u-kr-grek-latn');
CREATE COLLATION latinlast (provider = icu, locale = 'en@colReorder=grek-latn');

Sort Greek letters before Latin ones. (The default is Latin before Greek.)

CREATE COLLATION upperfirst (provider = icu, locale = 'en-u-kf-upper');
CREATE COLLATION upperfirst (provider = icu, locale = 'en@colCaseFirst=upper');

Sort upper-case letters before lower-case letters. (The default is lower-case letters first.)

CREATE COLLATION special (provider = icu, locale = 'en-u-kf-upper-kr-grek-latn');
CREATE COLLATION special (provider = icu, locale = 'en@colCaseFirst=upper;colReorder=grek-latn');

Combines both of the above options.

CREATE COLLATION numeric (provider = icu, locale = 'en-u-kn-true');
CREATE COLLATION numeric (provider = icu, locale = 'en@colNumeric=yes');

Numeric ordering, sorts sequences of digits by their numeric value, for example: A-21 < A-123 (also known as natural sort).

See Unicode Technical Standard #35 and BCP 47 for details. The list of possible collation types (co subtag) can be found in the CLDR repository.

Note that while this system allows creating collations that “ignore case” or “ignore accents” or similar (using the ks key), in order for such collations to act in a truly case- or accent-insensitive manner, they also need to be declared as not deterministic in CREATE COLLATION; see Section 24.2.2.4. Otherwise, any strings that compare equal according to the collation but are not byte-wise equal will be sorted according to their byte values.

Note

By design, ICU will accept almost any string as a locale name and match it to the closest locale it can provide, using the fallback procedure described in its documentation. Thus, there will be no direct feedback if a collation specification is composed using features that the given ICU installation does not actually support. It is therefore recommended to create application-level test cases to check that the collation definitions satisfy one's requirements.

24.2.2.3.3. Copying Collations

The command CREATE COLLATION can also be used to create a new collation from an existing collation, which can be useful to be able to use operating-system-independent collation names in applications, create compatibility names, or use an ICU-provided collation under a more readable name. For example:

CREATE COLLATION german FROM "de_DE";
CREATE COLLATION french FROM "fr-x-icu";

24.2.2.4. Nondeterministic Collations

A collation is either deterministic or nondeterministic. A deterministic collation uses deterministic comparisons, which means that it considers strings to be equal only if they consist of the same byte sequence. Nondeterministic comparison may determine strings to be equal even if they consist of different bytes. Typical situations include case-insensitive comparison, accent-insensitive comparison, as well as comparison of strings in different Unicode normal forms. It is up to the collation provider to actually implement such insensitive comparisons; the deterministic flag only determines whether ties are to be broken using bytewise comparison. See also Unicode Technical Standard 10 for more information on the terminology.

To create a nondeterministic collation, specify the property deterministic = false to CREATE COLLATION, for example:

CREATE COLLATION ndcoll (provider = icu, locale = 'und', deterministic = false);

This example would use the standard Unicode collation in a nondeterministic way. In particular, this would allow strings in different normal forms to be compared correctly. More interesting examples make use of the ICU customization facilities explained above. For example:

CREATE COLLATION case_insensitive (provider = icu, locale = 'und-u-ks-level2', deterministic = false);
CREATE COLLATION ignore_accents (provider = icu, locale = 'und-u-ks-level1-kc-true', deterministic = false);

All standard and predefined collations are deterministic, all user-defined collations are deterministic by default. While nondeterministic collations give a more “correct” behavior, especially when considering the full power of Unicode and its many special cases, they also have some drawbacks. Foremost, their use leads to a performance penalty. Note, in particular, that B-tree cannot use deduplication with indexes that use a nondeterministic collation. Also, certain operations are not possible with nondeterministic collations, such as pattern matching operations. Therefore, they should be used only in cases where they are specifically wanted.

Tip

To deal with text in different Unicode normalization forms, it is also an option to use the functions/expressions normalize and is normalized to preprocess or check the strings, instead of using nondeterministic collations. There are different trade-offs for each approach.

Prev | Up | Next
24.1. Locale Support | Home | 24.3. Character Set Support


24.3. Character Set Support
Prev Up Chapter 24. Localization Home Next

24.3. Character Set Support

24.3.1. Supported Character Sets
24.3.2. Setting the Character Set
24.3.3. Automatic Character Set Conversion Between Server and Client
24.3.4. Available Character Set Conversions
24.3.5. Further Reading

The character set support in PostgreSQL allows you to store text in a variety of character sets (also called encodings), including single-byte character sets such as the ISO 8859 series and multiple-byte character sets such as EUC (Extended Unix Code), UTF-8, and Mule internal code. All supported character sets can be used transparently by clients, but a few are not supported for use within the server (that is, as a server-side encoding). The default character set is selected while initializing your PostgreSQL database cluster using initdb. It can be overridden when you create a database, so you can have multiple databases each with a different character set.

An important restriction, however, is that each database's character set must be compatible with the database's LC_CTYPE (character classification) and LC_COLLATE (string sort order) locale settings. For C or POSIX locale, any character set is allowed, but for other libc-provided locales there is only one character set that will work correctly. (On Windows, however, UTF-8 encoding can be used with any locale.) If you have ICU support configured, ICU-provided locales can be used with most but not all server-side encodings.

24.3.1. Supported Character Sets

Table 24.1 shows the character sets available for use in PostgreSQL.

Table 24.1. PostgreSQL Character Sets

Name | Description | Language | Server? | ICU? | Bytes/​Char | Aliases
BIG5 | Big Five | Traditional Chinese | No | No | 1–2 | WIN950, Windows950
EUC_CN | Extended UNIX Code-CN | Simplified Chinese | Yes | Yes | 1–3 |
EUC_JP | Extended UNIX Code-JP | Japanese | Yes | Yes | 1–3 |
EUC_JIS_2004 | Extended UNIX Code-JP, JIS X 0213 | Japanese | Yes | No | 1–3 |
EUC_KR | Extended UNIX Code-KR | Korean | Yes | Yes | 1–3 |
EUC_TW | Extended UNIX Code-TW | Traditional Chinese, Taiwanese | Yes | Yes | 1–3 |
GB18030 | National Standard | Chinese | No | No | 1–4 |
GBK | Extended National Standard | Simplified Chinese | No | No | 1–2 | WIN936, Windows936
ISO_8859_5 | ISO 8859-5, ECMA 113 | Latin/Cyrillic | Yes | Yes | 1 |
ISO_8859_6 | ISO 8859-6, ECMA 114 | Latin/Arabic | Yes | Yes | 1 |
ISO_8859_7 | ISO 8859-7, ECMA 118 | Latin/Greek | Yes | Yes | 1 |
ISO_8859_8 | ISO 8859-8, ECMA 121 | Latin/Hebrew | Yes | Yes | 1 |
JOHAB | JOHAB | Korean (Hangul) | No | No | 1–3 |
KOI8R | KOI8-R | Cyrillic (Russian) | Yes | Yes | 1 | KOI8
KOI8U | KOI8-U | Cyrillic (Ukrainian) | Yes | Yes | 1 |
LATIN1 | ISO 8859-1, ECMA 94 | Western European | Yes | Yes | 1 | ISO88591
LATIN2 | ISO 8859-2, ECMA 94 | Central European | Yes | Yes | 1 | ISO88592
LATIN3 | ISO 8859-3, ECMA 94 | South European | Yes | Yes | 1 | ISO88593
LATIN4 | ISO 8859-4, ECMA 94 | North European | Yes | Yes | 1 | ISO88594
LATIN5 | ISO 8859-9, ECMA 128 | Turkish | Yes | Yes | 1 | ISO88599
LATIN6 | ISO 8859-10, ECMA 144 | Nordic | Yes | Yes | 1 | ISO885910
LATIN7 | ISO 8859-13 | Baltic | Yes | Yes | 1 | ISO885913
LATIN8 | ISO 8859-14 | Celtic | Yes | Yes | 1 | ISO885914
LATIN9 | ISO 8859-15 | LATIN1 with Euro and accents | Yes | Yes | 1 | ISO885915
LATIN10 | ISO 8859-16, ASRO SR 14111 | Romanian | Yes | No | 1 | ISO885916
MULE_INTERNAL | Mule internal code | Multilingual Emacs | Yes | No | 1–4 |
SJIS | Shift JIS | Japanese | No | No | 1–2 | Mskanji, ShiftJIS, WIN932, Windows932
SHIFT_JIS_2004 | Shift JIS, JIS X 0213 | Japanese | No | No | 1–2 |
SQL_ASCII | unspecified (see text) | any | Yes | No | 1 |
UHC | Unified Hangul Code | Korean | No | No | 1–2 | WIN949, Windows949
UTF8 | Unicode, 8-bit | all | Yes | Yes | 1–4 | Unicode
WIN866 | Windows CP866 | Cyrillic | Yes | Yes | 1 | ALT
WIN874 | Windows CP874 | Thai | Yes | No | 1 |
WIN1250 | Windows CP1250 | Central European | Yes | Yes | 1 |
WIN1251 | Windows CP1251 | Cyrillic | Yes | Yes | 1 | WIN
WIN1252 | Windows CP1252 | Western European | Yes | Yes | 1 |
WIN1253 | Windows CP1253 | Greek | Yes | Yes | 1 |
WIN1254 | Windows CP1254 | Turkish | Yes | Yes | 1 |
WIN1255 | Windows CP1255 | Hebrew | Yes | Yes | 1 |
WIN1256 | Windows CP1256 | Arabic | Yes | Yes | 1 |
WIN1257 | Windows CP1257 | Baltic | Yes | Yes | 1 |
WIN1258 | Windows CP1258 | Vietnamese | Yes | Yes | 1 | ABC, TCVN, TCVN5712, VSCII

Not all client APIs support all the listed character sets. For example, the PostgreSQL JDBC driver does not support MULE_INTERNAL, LATIN6, LATIN8, and LATIN10.

The SQL_ASCII setting behaves considerably differently from the other settings. When the server character set is SQL_ASCII, the server interprets byte values 0–127 according to the ASCII standard, while byte values 128–255 are taken as uninterpreted characters. No encoding conversion will be done when the setting is SQL_ASCII. Thus, this setting is not so much a declaration that a specific encoding is in use, as a declaration of ignorance about the encoding. In most cases, if you are working with any non-ASCII data, it is unwise to use the SQL_ASCII setting because PostgreSQL will be unable to help you by converting or validating non-ASCII characters.

24.3.2. Setting the Character Set

initdb defines the default character set (encoding) for a PostgreSQL cluster. For example,

initdb -E EUC_JP

sets the default character set to EUC_JP (Extended Unix Code for Japanese). You can use --encoding instead of -E if you prefer longer option strings. If no -E or --encoding option is given, initdb attempts to determine the appropriate encoding to use based on the specified or default locale.

You can specify a non-default encoding at database creation time, provided that the encoding is compatible with the selected locale:

createdb -E EUC_KR -T template0 --lc-collate=ko_KR.euckr --lc-ctype=ko_KR.euckr korean

This will create a database named korean that uses the character set EUC_KR, and locale ko_KR. Another way to accomplish this is to use this SQL command:

CREATE DATABASE korean WITH ENCODING 'EUC_KR' LC_COLLATE='ko_KR.euckr' LC_CTYPE='ko_KR.euckr' TEMPLATE=template0;

Notice that the above commands specify copying the template0 database. When copying any other database, the encoding and locale settings cannot be changed from those of the source database, because that might result in corrupt data. For more information see Section 23.3.

The encoding for a database is stored in the system catalog pg_database. You can see it by using the psql -l option or the \l command.

$ psql -l
                                         List of databases
   Name    |  Owner   | Encoding  |  Collation  |    Ctype    |          Access Privileges
-----------+----------+-----------+-------------+-------------+-------------------------------------
 clocaledb | hlinnaka | SQL_ASCII | C           | C           |
 englishdb | hlinnaka | UTF8      | en_GB.UTF8  | en_GB.UTF8  |
 japanese  | hlinnaka | UTF8      | ja_JP.UTF8  | ja_JP.UTF8  |
 korean    | hlinnaka | EUC_KR    | ko_KR.euckr | ko_KR.euckr |
 postgres  | hlinnaka | UTF8      | fi_FI.UTF8  | fi_FI.UTF8  |
 template0 | hlinnaka | UTF8      | fi_FI.UTF8  | fi_FI.UTF8  | {=c/hlinnaka,hlinnaka=CTc/hlinnaka}
 template1 | hlinnaka | UTF8      | fi_FI.UTF8  | fi_FI.UTF8  | {=c/hlinnaka,hlinnaka=CTc/hlinnaka}
(7 rows)

Important

On most modern operating systems, PostgreSQL can determine which character set is implied by the LC_CTYPE setting, and it will enforce that only the matching database encoding is used. On older systems it is your responsibility to ensure that you use the encoding expected by the locale you have selected. A mistake in this area is likely to lead to strange behavior of locale-dependent operations such as sorting.

PostgreSQL will allow superusers to create databases with SQL_ASCII encoding even when LC_CTYPE is not C or POSIX. As noted above, SQL_ASCII does not enforce that the data stored in the database has any particular encoding, and so this choice poses risks of locale-dependent misbehavior. Using this combination of settings is deprecated and may someday be forbidden altogether.

24.3.3. Automatic Character Set Conversion Between Server and Client

PostgreSQL supports automatic character set conversion between server and client for many combinations of character sets (Section 24.3.4 shows which ones).

To enable automatic character set conversion, you have to tell PostgreSQL the character set (encoding) you would like to use in the client. There are several ways to accomplish this:

  • Using the \encoding command in psql. \encoding allows you to change client encoding on the fly. For example, to change the encoding to SJIS, type:

    \encoding SJIS

  • libpq (Section 34.11) has functions to control the client encoding.

  • Using SET client_encoding TO. Setting the client encoding can be done with this SQL command:

    SET CLIENT_ENCODING TO 'value';

    Also you can use the standard SQL syntax SET NAMES for this purpose:

    SET NAMES 'value';

    To query the current client encoding:

    SHOW client_encoding;

    To return to the default encoding:

    RESET client_encoding;

  • Using PGCLIENTENCODING. If the environment variable PGCLIENTENCODING is defined in the client's environment, that client encoding is automatically selected when a connection to the server is made. (This can subsequently be overridden using any of the other methods mentioned above.)

  • Using the configuration variable client_encoding. If the client_encoding variable is set, that client encoding is automatically selected when a connection to the server is made. (This can subsequently be overridden using any of the other methods mentioned above.)

If the conversion of a particular character is not possible — suppose you chose EUC_JP for the server and LATIN1 for the client, and some Japanese characters are returned that do not have a representation in LATIN1 — an error is reported.

If the client character set is defined as SQL_ASCII, encoding conversion is disabled, regardless of the server's character set. (However, if the server's character set is not SQL_ASCII, the server will still check that incoming data is valid for that encoding; so the net effect is as though the client character set were the same as the server's.) Just as for the server, use of SQL_ASCII is unwise unless you are working with all-ASCII data.

24.3.4. Available Character Set Conversions

PostgreSQL allows conversion between any two character sets for which a conversion function is listed in the pg_conversion system catalog. PostgreSQL comes with some predefined conversions, as summarized in Table 24.2 and shown in more detail in Table 24.3. You can create a new conversion using the SQL command CREATE CONVERSION. (To be used for automatic client/server conversions, a conversion must be marked as “default” for its character set pair.)

Table 24.2. Built-in Client/Server Character Set Conversions

Server Character Set | Available Client Character Sets
BIG5 | not supported as a server encoding
EUC_CN | EUC_CN, MULE_INTERNAL, UTF8
EUC_JP | EUC_JP, MULE_INTERNAL, SJIS, UTF8
EUC_JIS_2004 | EUC_JIS_2004, SHIFT_JIS_2004, UTF8
EUC_KR | EUC_KR, MULE_INTERNAL, UTF8
EUC_TW | EUC_TW, BIG5, MULE_INTERNAL, UTF8
GB18030 | not supported as a server encoding
GBK | not supported as a server encoding
ISO_8859_5 | ISO_8859_5, KOI8R, MULE_INTERNAL, UTF8, WIN866, WIN1251
ISO_8859_6 | ISO_8859_6, UTF8
ISO_8859_7 | ISO_8859_7, UTF8
ISO_8859_8 | ISO_8859_8, UTF8
JOHAB | not supported as a server encoding
KOI8R | KOI8R, ISO_8859_5, MULE_INTERNAL, UTF8, WIN866, WIN1251
KOI8U | KOI8U, UTF8
LATIN1 | LATIN1, MULE_INTERNAL, UTF8
LATIN2 | LATIN2, MULE_INTERNAL, UTF8, WIN1250
LATIN3 | LATIN3, MULE_INTERNAL, UTF8
LATIN4 | LATIN4, MULE_INTERNAL, UTF8
LATIN5 | LATIN5, UTF8
LATIN6 | LATIN6, UTF8
LATIN7 | LATIN7, UTF8
LATIN8 | LATIN8, UTF8
LATIN9 | LATIN9, UTF8
LATIN10 | LATIN10, UTF8
MULE_INTERNAL | MULE_INTERNAL, BIG5, EUC_CN, EUC_JP, EUC_KR, EUC_TW, ISO_8859_5, KOI8R, LATIN1 to LATIN4, SJIS, WIN866, WIN1250, WIN1251
SJIS | not supported as a server encoding
SHIFT_JIS_2004 | not supported as a server encoding
SQL_ASCII | any (no conversion will be performed)
UHC | not supported as a server encoding
UTF8 | all supported encodings
WIN866 | WIN866, ISO_8859_5, KOI8R, MULE_INTERNAL, UTF8, WIN1251
WIN874 | WIN874, UTF8
WIN1250 | WIN1250, LATIN2, MULE_INTERNAL, UTF8
WIN1251 | WIN1251, ISO_8859_5, KOI8R, MULE_INTERNAL, UTF8, WIN866
WIN1252 | WIN1252, UTF8
WIN1253 | WIN1253, UTF8
WIN1254 | WIN1254, UTF8
WIN1255 | WIN1255, UTF8
WIN1256 | WIN1256, UTF8
WIN1257 | WIN1257, UTF8
WIN1258 | WIN1258, UTF8

Table 24.3. All Built-in Character Set Conversions

Conversion Name [a] | Source Encoding | Destination Encoding
big5_to_euc_tw | BIG5 | EUC_TW
big5_to_mic | BIG5 | MULE_INTERNAL
big5_to_utf8 | BIG5 | UTF8
euc_cn_to_mic | EUC_CN | MULE_INTERNAL
euc_cn_to_utf8 | EUC_CN | UTF8
euc_jp_to_mic | EUC_JP | MULE_INTERNAL
euc_jp_to_sjis | EUC_JP | SJIS
euc_jp_to_utf8 | EUC_JP | UTF8
euc_kr_to_mic | EUC_KR | MULE_INTERNAL
euc_kr_to_utf8 | EUC_KR | UTF8
euc_tw_to_big5 | EUC_TW | BIG5
euc_tw_to_mic | EUC_TW | MULE_INTERNAL
euc_tw_to_utf8 | EUC_TW | UTF8
gb18030_to_utf8 | GB18030 | UTF8
gbk_to_utf8 | GBK | UTF8
iso_8859_10_to_utf8 | LATIN6 | UTF8
iso_8859_13_to_utf8 | LATIN7 | UTF8
iso_8859_14_to_utf8 | LATIN8 | UTF8
iso_8859_15_to_utf8 | LATIN9 | UTF8
iso_8859_16_to_utf8 | LATIN10 | UTF8
iso_8859_1_to_mic | LATIN1 | MULE_INTERNAL
iso_8859_1_to_utf8 | LATIN1 | UTF8
iso_8859_2_to_mic | LATIN2 | MULE_INTERNAL
iso_8859_2_to_utf8 | LATIN2 | UTF8
iso_8859_2_to_windows_1250 | LATIN2 | WIN1250
iso_8859_3_to_mic | LATIN3 | MULE_INTERNAL
iso_8859_3_to_utf8 | LATIN3 | UTF8
iso_8859_4_to_mic | LATIN4 | MULE_INTERNAL
iso_8859_4_to_utf8 | LATIN4 | UTF8
iso_8859_5_to_koi8_r | ISO_8859_5 | KOI8R
iso_8859_5_to_mic | ISO_8859_5 | MULE_INTERNAL
iso_8859_5_to_utf8 | ISO_8859_5 | UTF8
iso_8859_5_to_windows_1251 | ISO_8859_5 | WIN1251
iso_8859_5_to_windows_866 | ISO_8859_5 | WIN866
iso_8859_6_to_utf8 | ISO_8859_6 | UTF8
iso_8859_7_to_utf8 | ISO_8859_7 | UTF8
iso_8859_8_to_utf8 | ISO_8859_8 | UTF8
iso_8859_9_to_utf8 | LATIN5 | UTF8
johab_to_utf8 | JOHAB | UTF8
koi8_r_to_iso_8859_5 | KOI8R | ISO_8859_5
koi8_r_to_mic | KOI8R | MULE_INTERNAL
koi8_r_to_utf8 | KOI8R | UTF8
koi8_r_to_windows_1251 | KOI8R | WIN1251
koi8_r_to_windows_866 | KOI8R | WIN866
koi8_u_to_utf8 | KOI8U | UTF8
mic_to_big5 | MULE_INTERNAL | BIG5
mic_to_euc_cn | MULE_INTERNAL | EUC_CN
mic_to_euc_jp | MULE_INTERNAL | EUC_JP
mic_to_euc_kr | MULE_INTERNAL | EUC_KR
mic_to_euc_tw | MULE_INTERNAL | EUC_TW
mic_to_iso_8859_1 | MULE_INTERNAL | LATIN1
mic_to_iso_8859_2 | MULE_INTERNAL | LATIN2
mic_to_iso_8859_3 | MULE_INTERNAL | LATIN3
mic_to_iso_8859_4 | MULE_INTERNAL | LATIN4
mic_to_iso_8859_5 | MULE_INTERNAL | ISO_8859_5
mic_to_koi8_r | MULE_INTERNAL | KOI8R
mic_to_sjis | MULE_INTERNAL | SJIS
mic_to_windows_1250 | MULE_INTERNAL | WIN1250
mic_to_windows_1251 | MULE_INTERNAL | WIN1251
mic_to_windows_866 | MULE_INTERNAL | WIN866
sjis_to_euc_jp | SJIS | EUC_JP
sjis_to_mic | SJIS | MULE_INTERNAL
sjis_to_utf8 | SJIS | UTF8
windows_1258_to_utf8 | WIN1258 | UTF8
uhc_to_utf8 | UHC | UTF8
utf8_to_big5 | UTF8 | BIG5
utf8_to_euc_cn | UTF8 | EUC_CN
utf8_to_euc_jp | UTF8 | EUC_JP
utf8_to_euc_kr | UTF8 | EUC_KR
utf8_to_euc_tw | UTF8 | EUC_TW
utf8_to_gb18030 | UTF8 | GB18030
utf8_to_gbk | UTF8 | GBK
utf8_to_iso_8859_1 | UTF8 | LATIN1
utf8_to_iso_8859_10 | UTF8 | LATIN6
utf8_to_iso_8859_13 | UTF8 | LATIN7
utf8_to_iso_8859_14 | UTF8 | LATIN8
utf8_to_iso_8859_15 | UTF8 | LATIN9
utf8_to_iso_8859_16 | UTF8 | LATIN10
utf8_to_iso_8859_2 | UTF8 | LATIN2
utf8_to_iso_8859_3 | UTF8 | LATIN3
utf8_to_iso_8859_4 | UTF8 | LATIN4
utf8_to_iso_8859_5 | UTF8 | ISO_8859_5
utf8_to_iso_8859_6 | UTF8 | ISO_8859_6
utf8_to_iso_8859_7 | UTF8 | ISO_8859_7
utf8_to_iso_8859_8 | UTF8 | ISO_8859_8
utf8_to_iso_8859_9 | UTF8 | LATIN5
utf8_to_johab | UTF8 | JOHAB
utf8_to_koi8_r | UTF8 | KOI8R
utf8_to_koi8_u | UTF8 | KOI8U
utf8_to_sjis | UTF8 | SJIS
utf8_to_windows_1258 | UTF8 | WIN1258
utf8_to_uhc | UTF8 | UHC
utf8_to_windows_1250 | UTF8 | WIN1250
utf8_to_windows_1251 | UTF8 | WIN1251
utf8_to_windows_1252 | UTF8 | WIN1252
utf8_to_windows_1253 | UTF8 | WIN1253
utf8_to_windows_1254 | UTF8 | WIN1254
utf8_to_windows_1255 | UTF8 | WIN1255
utf8_to_windows_1256 | UTF8 | WIN1256
utf8_to_windows_1257 | UTF8 | WIN1257
utf8_to_windows_866 | UTF8 | WIN866
utf8_to_windows_874 | UTF8 | WIN874
windows_1250_to_iso_8859_2 | WIN1250 | LATIN2
windows_1250_to_mic | WIN1250 | MULE_INTERNAL
windows_1250_to_utf8 | WIN1250 | UTF8
windows_1251_to_iso_8859_5 | WIN1251 | ISO_8859_5
windows_1251_to_koi8_r | WIN1251 | KOI8R
windows_1251_to_mic | WIN1251 | MULE_INTERNAL
windows_1251_to_utf8 | WIN1251 | UTF8
windows_1251_to_windows_866 | WIN1251 | WIN866
windows_1252_to_utf8 | WIN1252 | UTF8
windows_1256_to_utf8 | WIN1256 | UTF8
windows_866_to_iso_8859_5 | WIN866 | ISO_8859_5
windows_866_to_koi8_r | WIN866 | KOI8R
windows_866_to_mic | WIN866 | MULE_INTERNAL
windows_866_to_utf8 | WIN866 | UTF8
windows_866_to_windows_1251 | WIN866 | WIN
windows_874_to_utf8 | WIN874 | UTF8
euc_jis_2004_to_utf8 | EUC_JIS_2004 | UTF8
utf8_to_euc_jis_2004 | UTF8 | EUC_JIS_2004
shift_jis_2004_to_utf8 | SHIFT_JIS_2004 | UTF8
utf8_to_shift_jis_2004 | UTF8 | SHIFT_JIS_2004
euc_jis_2004_to_shift_jis_2004 | EUC_JIS_2004 | SHIFT_JIS_2004
shift_jis_2004_to_euc_jis_2004 | SHIFT_JIS_2004 | EUC_JIS_2004
[a] The conversion names follow a standard naming scheme: The official name of the source encoding with all non-alphanumeric characters replaced by underscores, followed by _to_, followed by the similarly processed destination encoding name. Therefore, these names sometimes deviate from the customary encoding names shown in Table 24.1.

24.3.5. Further Reading

These are good sources to start learning about various kinds of encoding systems.

CJKV Information Processing: Chinese, Japanese, Korean & Vietnamese Computing

Contains detailed explanations of EUC_JP, EUC_CN, EUC_KR, EUC_TW.

https://www.unicode.org/

The web site of the Unicode Consortium.

RFC 3629

UTF-8 (8-bit UCS/Unicode Transformation Format) is defined here.

Prev | Up | Next
24.2. Collation Support | Home | Chapter 25. Routine Database Maintenance Tasks


Chapter 25. Routine Database Maintenance Tasks
Prev Up Part III. Server Administration Home Next

Chapter 25. Routine Database Maintenance Tasks

Table of Contents

25.1. Routine Vacuuming
25.1.1. Vacuuming Basics
25.1.2. Recovering Disk Space
25.1.3. Updating Planner Statistics
25.1.4. Updating the Visibility Map
25.1.5. Preventing Transaction ID Wraparound Failures
25.1.6. The Autovacuum Daemon
25.2. Routine Reindexing
25.3. Log File Maintenance

PostgreSQL, like any database software, requires that certain tasks be performed regularly to achieve optimum performance. The tasks discussed here are required, but they are repetitive in nature and can easily be automated using standard tools such as cron scripts or Windows' Task Scheduler. It is the database administrator's responsibility to set up appropriate scripts, and to check that they execute successfully.

One obvious maintenance task is the creation of backup copies of the data on a regular schedule. Without a recent backup, you have no chance of recovery after a catastrophe (disk failure, fire, mistakenly dropping a critical table, etc.). The backup and recovery mechanisms available in PostgreSQL are discussed at length in Chapter 26.

The other main category of maintenance task is periodic “vacuuming” of the database. This activity is discussed in Section 25.1. Closely related to this is updating the statistics that will be used by the query planner, as discussed in Section 25.1.3.

Another task that might need periodic attention is log file management. This is discussed in Section 25.3.

check_postgres is available for monitoring database health and reporting unusual conditions. check_postgres integrates with Nagios and MRTG, but can be run standalone too.

PostgreSQL is low-maintenance compared to some other database management systems. Nonetheless, appropriate attention to these tasks will go far towards ensuring a pleasant and productive experience with the system.

Prev | Up | Next
24.3. Character Set Support | Home | 25.1. Routine Vacuuming


25.3. Log File Maintenance
Prev Up Chapter 25. Routine Database Maintenance Tasks Home Next

25.3. Log File Maintenance

It is a good idea to save the database server's log output somewhere, rather than just discarding it via /dev/null. The log output is invaluable when diagnosing problems. However, the log output tends to be voluminous (especially at higher debug levels) so you won't want to save it indefinitely. You need to rotate the log files so that new log files are started and old ones removed after a reasonable period of time.

If you simply direct the stderr of postgres into a file, you will have log output, but the only way to truncate the log file is to stop and restart the server. This might be acceptable if you are using PostgreSQL in a development environment, but few production servers would find this behavior acceptable.

A better approach is to send the server's stderr output to some type of log rotation program. There is a built-in log rotation facility, which you can use by setting the configuration parameter logging_collector to true in postgresql.conf. The control parameters for this program are described in Section 20.8.1. You can also use this approach to capture the log data in machine readable CSV (comma-separated values) format.

Alternatively, you might prefer to use an external log rotation program if you have one that you are already using with other server software. For example, the rotatelogs tool included in the Apache distribution can be used with PostgreSQL. One way to do this is to pipe the server's stderr output to the desired program. If you start the server with pg_ctl, then stderr is already redirected to stdout, so you just need a pipe command, for example:

pg_ctl start | rotatelogs /var/log/pgsql_log 86400

You can combine these approaches by setting up logrotate to collect log files produced by PostgreSQL built-in logging collector. In this case, the logging collector defines the names and location of the log files, while logrotate periodically archives these files. When initiating log rotation, logrotate must ensure that the application sends further output to the new file. This is commonly done with a postrotate script that sends a SIGHUP signal to the application, which then reopens the log file. In PostgreSQL, you can run pg_ctl with the logrotate option instead. When the server receives this command, the server either switches to a new log file or reopens the existing file, depending on the logging configuration (see Section 20.8.1).

Note

When using static log file names, the server might fail to reopen the log file if the max open file limit is reached or a file table overflow occurs. In this case, log messages are sent to the old log file until a successful log rotation. If logrotate is configured to compress the log file and delete it, the server may lose the messages logged in this time frame. To avoid this issue, you can configure the logging collector to dynamically assign log file names and use a prerotate script to ignore open log files.

Another production-grade approach to managing log output is to send it to syslog and let syslog deal with file rotation. To do this, set the configuration parameter log_destination to syslog (to log to syslog only) in postgresql.conf. Then you can send a SIGHUP signal to the syslog daemon whenever you want to force it to start writing a new log file. If you want to automate log rotation, the logrotate program can be configured to work with log files from syslog.

On many systems, however, syslog is not very reliable, particularly with large log messages; it might truncate or drop messages just when you need them the most. Also, on Linux, syslog will flush each message to disk, yielding poor performance. (You can use a “-” at the start of the file name in the syslog configuration file to disable syncing.)

Note that all the solutions described above take care of starting new log files at configurable intervals, but they do not handle deletion of old, no-longer-useful log files. You will probably want to set up a batch job to periodically delete old log files. Another possibility is to configure the rotation program so that old log files are overwritten cyclically.

pgBadger is an external project that does sophisticated log file analysis. check_postgres provides Nagios alerts when important messages appear in the log files, as well as detection of many other extraordinary conditions.

Prev | Up | Next
25.2. Routine Reindexing | Home | Chapter 26. Backup and Restore


Chapter 26. Backup and Restore
Prev Up Part III. Server Administration Home Next

Chapter 26. Backup and Restore

Table of Contents

26.1. SQL Dump
26.1.1. Restoring the Dump
26.1.2. Using pg_dumpall
26.1.3. Handling Large Databases
26.2. File System Level Backup
26.3. Continuous Archiving and Point-in-Time Recovery (PITR)
26.3.1. Setting Up WAL Archiving
26.3.2. Making a Base Backup
26.3.3. Making a Base Backup Using the Low Level API
26.3.4. Recovering Using a Continuous Archive Backup
26.3.5. Timelines
26.3.6. Tips and Examples
26.3.7. Caveats

As with everything that contains valuable data, PostgreSQL databases should be backed up regularly. While the procedure is essentially simple, it is important to have a clear understanding of the underlying techniques and assumptions.

There are three fundamentally different approaches to backing up PostgreSQL data:

  • SQL dump

  • File system level backup

  • Continuous archiving

Each has its own strengths and weaknesses; each is discussed in turn in the following sections.

Prev | Up | Next
25.3. Log File Maintenance | Home | 26.1. SQL Dump


26.1. SQL Dump
Prev Up Chapter 26. Backup and Restore Home Next

26.1. SQL Dump

26.1.1. Restoring the Dump
26.1.2. Using pg_dumpall
26.1.3. Handling Large Databases

The idea behind this dump method is to generate a file with SQL commands that, when fed back to the server, will recreate the database in the same state as it was at the time of the dump. PostgreSQL provides the utility program pg_dump for this purpose. The basic usage of this command is:

pg_dump dbname > dumpfile

As you see, pg_dump writes its result to the standard output. We will see below how this can be useful. While the above command creates a text file, pg_dump can create files in other formats that allow for parallelism and more fine-grained control of object restoration.

pg_dump is a regular PostgreSQL client application (albeit a particularly clever one). This means that you can perform this backup procedure from any remote host that has access to the database. But remember that pg_dump does not operate with special permissions. In particular, it must have read access to all tables that you want to back up, so in order to back up the entire database you almost always have to run it as a database superuser. (If you do not have sufficient privileges to back up the entire database, you can still back up portions of the database to which you do have access using options such as -n schema or -t table.)

To specify which database server pg_dump should contact, use the command line options -h host and -p port. The default host is the local host or whatever your PGHOST environment variable specifies. Similarly, the default port is indicated by the PGPORT environment variable or, failing that, by the compiled-in default. (Conveniently, the server will normally have the same compiled-in default.)

Like any other PostgreSQL client application, pg_dump will by default connect with the database user name that is equal to the current operating system user name. To override this, either specify the -U option or set the environment variable PGUSER. Remember that pg_dump connections are subject to the normal client authentication mechanisms (which are described in Chapter 21).

An important advantage of pg_dump over the other backup methods described later is that pg_dump's output can generally be re-loaded into newer versions of PostgreSQL, whereas file-level backups and continuous archiving are both extremely server-version-specific. pg_dump is also the only method that will work when transferring a database to a different machine architecture, such as going from a 32-bit to a 64-bit server.

Dumps created by pg_dump are internally consistent, meaning, the dump represents a snapshot of the database at the time pg_dump began running. pg_dump does not block other operations on the database while it is working. (Exceptions are those operations that need to operate with an exclusive lock, such as most forms of ALTER TABLE.)

26.1.1. Restoring the Dump

Text files created by pg_dump are intended to be read in by the psql program. The general command form to restore a dump is

psql dbname < dumpfile

where dumpfile is the file output by the pg_dump command. The database dbname will not be created by this command, so you must create it yourself from template0 before executing psql (e.g., with createdb -T template0 dbname). psql supports options similar to pg_dump for specifying the database server to connect to and the user name to use. See the psql reference page for more information. Non-text file dumps are restored using the pg_restore utility.

Before restoring an SQL dump, all the users who own objects or were granted permissions on objects in the dumped database must already exist. If they do not, the restore will fail to recreate the objects with the original ownership and/or permissions. (Sometimes this is what you want, but usually it is not.)

By default, the psql script will continue to execute after an SQL error is encountered. You might wish to run psql with the ON_ERROR_STOP variable set to alter that behavior and have psql exit with an exit status of 3 if an SQL error occurs:

psql --set ON_ERROR_STOP=on dbname < dumpfile

Either way, you will only have a partially restored database. Alternatively, you can specify that the whole dump should be restored as a single transaction, so the restore is either fully completed or fully rolled back. This mode can be specified by passing the -1 or --single-transaction command-line options to psql. When using this mode, be aware that even a minor error can rollback a restore that has already run for many hours. However, that might still be preferable to manually cleaning up a complex database after a partially restored dump.

The ability of pg_dump and psql to write to or read from pipes makes it possible to dump a database directly from one server to another, for example:

pg_dump -h host1 dbname | psql -h host2 dbname

Important

The dumps produced by pg_dump are relative to template0. This means that any languages, procedures, etc. added via template1 will also be dumped by pg_dump. As a result, when restoring, if you are using a customized template1, you must create the empty database from template0, as in the example above.

After restoring a backup, it is wise to run ANALYZE on each database so the query optimizer has useful statistics; see Section 25.1.3 and Section 25.1.6 for more information. For more advice on how to load large amounts of data into PostgreSQL efficiently, refer to Section 14.4.

26.1.2. Using pg_dumpall

pg_dump dumps only a single database at a time, and it does not dump information about roles or tablespaces (because those are cluster-wide rather than per-database). To support convenient dumping of the entire contents of a database cluster, the pg_dumpall program is provided. pg_dumpall backs up each database in a given cluster, and also preserves cluster-wide data such as role and tablespace definitions. The basic usage of this command is:

pg_dumpall > dumpfile

The resulting dump can be restored with psql:

psql -f dumpfile postgres

(Actually, you can specify any existing database name to start from, but if you are loading into an empty cluster then postgres should usually be used.) It is always necessary to have database superuser access when restoring a pg_dumpall dump, as that is required to restore the role and tablespace information. If you use tablespaces, make sure that the tablespace paths in the dump are appropriate for the new installation.

pg_dumpall works by emitting commands to re-create roles, tablespaces, and empty databases, then invoking pg_dump for each database. This means that while each database will be internally consistent, the snapshots of different databases are not synchronized.

Cluster-wide data can be dumped alone using the pg_dumpall --globals-only option. This is necessary to fully backup the cluster if running the pg_dump command on individual databases.

26.1.3. Handling Large Databases

Some operating systems have maximum file size limits that cause problems when creating large pg_dump output files. Fortunately, pg_dump can write to the standard output, so you can use standard Unix tools to work around this potential problem. There are several possible methods:

Use compressed dumps. You can use your favorite compression program, for example gzip:

pg_dump dbname | gzip > filename.gz

Reload with:

gunzip -c filename.gz | psql dbname

or:

cat filename.gz | gunzip | psql dbname

Use split. The split command allows you to split the output into smaller files that are acceptable in size to the underlying file system. For example, to make 2 gigabyte chunks:

pg_dump dbname | split -b 2G - filename

Reload with:

cat filename* | psql dbname

If using GNU split, it is possible to use it and gzip together:

pg_dump dbname | split -b 2G --filter='gzip > $FILE.gz'

It can be restored using zcat.

Use pg_dump's custom dump format. If PostgreSQL was built on a system with the zlib compression library installed, the custom dump format will compress data as it writes it to the output file. This will produce dump file sizes similar to using gzip, but it has the added advantage that tables can be restored selectively. The following command dumps a database using the custom dump format:

pg_dump -Fc dbname > filename

A custom-format dump is not a script for psql, but instead must be restored with pg_restore, for example:

pg_restore -d dbname filename

See the pg_dump and pg_restore reference pages for details.

For very large databases, you might need to combine split with one of the other two approaches.

Use pg_dump's parallel dump feature. To speed up the dump of a large database, you can use pg_dump's parallel mode. This will dump multiple tables at the same time. You can control the degree of parallelism with the -j parameter. Parallel dumps are only supported for the "directory" archive format.

pg_dump -j num -F d -f out.dir dbname

You can use pg_restore -j to restore a dump in parallel. This will work for any archive of either the "custom" or the "directory" archive mode, whether or not it has been created with pg_dump -j.

Prev | Up | Next
Chapter 26. Backup and Restore | Home | 26.2. File System Level Backup


26.2. File System Level Backup
Prev Up Chapter 26. Backup and Restore Home Next

26.2. File System Level Backup

An alternative backup strategy is to directly copy the files that PostgreSQL uses to store the data in the database; Section 19.2 explains where these files are located. You can use whatever method you prefer for doing file system backups; for example:

tar -cf backup.tar /usr/local/pgsql/data

There are two restrictions, however, which make this method impractical, or at least inferior to the pg_dump method:

  1. The database server must be shut down in order to get a usable backup. Half-way measures such as disallowing all connections will not work (in part because tar and similar tools do not take an atomic snapshot of the state of the file system, but also because of internal buffering within the server). Information about stopping the server can be found in Section 19.5. Needless to say, you also need to shut down the server before restoring the data.

  2. If you have dug into the details of the file system layout of the database, you might be tempted to try to back up or restore only certain individual tables or databases from their respective files or directories. This will not work because the information contained in these files is not usable without the commit log files, pg_xact/*, which contain the commit status of all transactions. A table file is only usable with this information. Of course it is also impossible to restore only a table and the associated pg_xact data because that would render all other tables in the database cluster useless. So file system backups only work for complete backup and restoration of an entire database cluster.

An alternative file-system backup approach is to make a “consistent snapshot” of the data directory, if the file system supports that functionality (and you are willing to trust that it is implemented correctly). The typical procedure is to make a “frozen snapshot” of the volume containing the database, then copy the whole data directory (not just parts, see above) from the snapshot to a backup device, then release the frozen snapshot. This will work even while the database server is running. However, a backup created in this way saves the database files in a state as if the database server was not properly shut down; therefore, when you start the database server on the backed-up data, it will think the previous server instance crashed and will replay the WAL log. This is not a problem; just be aware of it (and be sure to include the WAL files in your backup). You can perform a CHECKPOINT before taking the snapshot to reduce recovery time.

If your database is spread across multiple file systems, there might not be any way to obtain exactly-simultaneous frozen snapshots of all the volumes. For example, if your data files and WAL log are on different disks, or if tablespaces are on different file systems, it might not be possible to use snapshot backup because the snapshots must be simultaneous. Read your file system documentation very carefully before trusting the consistent-snapshot technique in such situations.

If simultaneous snapshots are not possible, one option is to shut down the database server long enough to establish all the frozen snapshots. Another option is to perform a continuous archiving base backup (Section 26.3.2) because such backups are immune to file system changes during the backup. This requires enabling continuous archiving just during the backup process; restore is done using continuous archive recovery (Section 26.3.4).

Another option is to use rsync to perform a file system backup. This is done by first running rsync while the database server is running, then shutting down the database server long enough to do an rsync --checksum. (--checksum is necessary because rsync only has file modification-time granularity of one second.) The second rsync will be quicker than the first, because it has relatively little data to transfer, and the end result will be consistent because the server was down. This method allows a file system backup to be performed with minimal downtime.

Note that a file system backup will typically be larger than an SQL dump. (pg_dump does not need to dump the contents of indexes for example, just the commands to recreate them.) However, taking a file system backup might be faster.

Prev | Up | Next
26.1. SQL Dump | Home | 26.3. Continuous Archiving and Point-in-Time Recovery (PITR)


26.3. Continuous Archiving and Point-in-Time Recovery (PITR)
Prev Up Chapter 26. Backup and Restore Home Next

26.3. Continuous Archiving and Point-in-Time Recovery (PITR)

26.3.1. Setting Up WAL Archiving
26.3.2. Making a Base Backup
26.3.3. Making a Base Backup Using the Low Level API
26.3.4. Recovering Using a Continuous Archive Backup
26.3.5. Timelines
26.3.6. Tips and Examples
26.3.7. Caveats

At all times, PostgreSQL maintains a write ahead log (WAL) in the pg_wal/ subdirectory of the cluster's data directory. The log records every change made to the database's data files. This log exists primarily for crash-safety purposes: if the system crashes, the database can be restored to consistency by “replaying” the log entries made since the last checkpoint. However, the existence of the log makes it possible to use a third strategy for backing up databases: we can combine a file-system-level backup with backup of the WAL files. If recovery is needed, we restore the file system backup and then replay from the backed-up WAL files to bring the system to a current state. This approach is more complex to administer than either of the previous approaches, but it has some significant benefits:

  • We do not need a perfectly consistent file system backup as the starting point. Any internal inconsistency in the backup will be corrected by log replay (this is not significantly different from what happens during crash recovery). So we do not need a file system snapshot capability, just tar or a similar archiving tool.

  • Since we can combine an indefinitely long sequence of WAL files for replay, continuous backup can be achieved simply by continuing to archive the WAL files. This is particularly valuable for large databases, where it might not be convenient to take a full backup frequently.

  • It is not necessary to replay the WAL entries all the way to the end. We could stop the replay at any point and have a consistent snapshot of the database as it was at that time. Thus, this technique supports point-in-time recovery: it is possible to restore the database to its state at any time since your base backup was taken.

  • If we continuously feed the series of WAL files to another machine that has been loaded with the same base backup file, we have a warm standby system: at any point we can bring up the second machine and it will have a nearly-current copy of the database.

Note

pg_dump and pg_dumpall do not produce file-system-level backups and cannot be used as part of a continuous-archiving solution. Such dumps are logical and do not contain enough information to be used by WAL replay.

As with the plain file-system-backup technique, this method can only support restoration of an entire database cluster, not a subset. Also, it requires a lot of archival storage: the base backup might be bulky, and a busy system will generate many megabytes of WAL traffic that have to be archived. Still, it is the preferred backup technique in many situations where high reliability is needed.

To recover successfully using continuous archiving (also called “online backup” by many database vendors), you need a continuous sequence of archived WAL files that extends back at least as far as the start time of your backup. So to get started, you should set up and test your procedure for archiving WAL files before you take your first base backup. Accordingly, we first discuss the mechanics of archiving WAL files.

26.3.1. Setting Up WAL Archiving

In an abstract sense, a running PostgreSQL system produces an indefinitely long sequence of WAL records. The system physically divides this sequence into WAL segment files, which are normally 16MB apiece (although the segment size can be altered during initdb). The segment files are given numeric names that reflect their position in the abstract WAL sequence. When not using WAL archiving, the system normally creates just a few segment files and then “recycles” them by renaming no-longer-needed segment files to higher segment numbers. It's assumed that segment files whose contents precede the last checkpoint are no longer of interest and can be recycled.

When archiving WAL data, we need to capture the contents of each segment file once it is filled, and save that data somewhere before the segment file is recycled for reuse. Depending on the application and the available hardware, there could be many different ways of “saving the data somewhere”: we could copy the segment files to an NFS-mounted directory on another machine, write them onto a tape drive (ensuring that you have a way of identifying the original name of each file), or batch them together and burn them onto CDs, or something else entirely. To provide the database administrator with flexibility, PostgreSQL tries not to make any assumptions about how the archiving will be done. Instead, PostgreSQL lets the administrator specify a shell command to be executed to copy a completed segment file to wherever it needs to go. The command could be as simple as a cp, or it could invoke a complex shell script — it's all up to you.

To enable WAL archiving, set the wal_level configuration parameter to replica or higher, archive_mode to on, and specify the shell command to use in the archive_command configuration parameter. In practice these settings will always be placed in the postgresql.conf file. In archive_command, %p is replaced by the path name of the file to archive, while %f is replaced by only the file name. (The path name is relative to the current working directory, i.e., the cluster's data directory.) Use %% if you need to embed an actual % character in the command. The simplest useful command is something like:

archive_command = 'test ! -f /mnt/server/archivedir/%f && cp %p /mnt/server/archivedir/%f'  # Unix
archive_command = 'copy "%p" "C:\\server\\archivedir\\%f"'  # Windows

which will copy archivable WAL segments to the directory /mnt/server/archivedir. (This is an example, not a recommendation, and might not work on all platforms.) After the %p and %f parameters have been replaced, the actual command executed might look like this:

test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/00000001000000A900000065 /mnt/server/archivedir/00000001000000A900000065

A similar command will be generated for each new file to be archived.

The archive command will be executed under the ownership of the same user that the PostgreSQL server is running as. Since the series of WAL files being archived contains effectively everything in your database, you will want to be sure that the archived data is protected from prying eyes; for example, archive into a directory that does not have group or world read access.

It is important that the archive command return zero exit status if and only if it succeeds. Upon getting a zero result, PostgreSQL will assume that the file has been successfully archived, and will remove or recycle it. However, a nonzero status tells PostgreSQL that the file was not archived; it will try again periodically until it succeeds.

When the archive command is terminated by a signal (other than SIGTERM that is used as part of a server shutdown) or an error by the shell with an exit status greater than 125 (such as command not found), the archiver process aborts and gets restarted by the postmaster. In such cases, the failure is not reported in pg_stat_archiver.

The archive command should generally be designed to refuse to overwrite any pre-existing archive file. This is an important safety feature to preserve the integrity of your archive in case of administrator error (such as sending the output of two different servers to the same archive directory).

It is advisable to test your proposed archive command to ensure that it indeed does not overwrite an existing file, and that it returns nonzero status in this case. The example command above for Unix ensures this by including a separate test step. On some Unix platforms, cp has switches such as -i that can be used to do the same thing less verbosely, but you should not rely on these without verifying that the right exit status is returned. (In particular, GNU cp will return status zero when -i is used and the target file already exists, which is not the desired behavior.)

While designing your archiving setup, consider what will happen if the archive command fails repeatedly because some aspect requires operator intervention or the archive runs out of space. For example, this could occur if you write to tape without an autochanger; when the tape fills, nothing further can be archived until the tape is swapped. You should ensure that any error condition or request to a human operator is reported appropriately so that the situation can be resolved reasonably quickly. The pg_wal/ directory will continue to fill with WAL segment files until the situation is resolved. (If the file system containing pg_wal/ fills up, PostgreSQL will do a PANIC shutdown. No committed transactions will be lost, but the database will remain offline until you free some space.)

The speed of the archiving command is unimportant as long as it can keep up with the average rate at which your server generates WAL data. Normal operation continues even if the archiving process falls a little behind. If archiving falls significantly behind, this will increase the amount of data that would be lost in the event of a disaster. It will also mean that the pg_wal/ directory will contain large numbers of not-yet-archived segment files, which could eventually exceed available disk space. You are advised to monitor the archiving process to ensure that it is working as you intend.

In writing your archive command, you should assume that the file names to be archived can be up to 64 characters long and can contain any combination of ASCII letters, digits, and dots. It is not necessary to preserve the original relative path (%p) but it is necessary to preserve the file name (%f).

Note that although WAL archiving will allow you to restore any modifications made to the data in your PostgreSQL database, it will not restore changes made to configuration files (that is, postgresql.conf, pg_hba.conf and pg_ident.conf), since those are edited manually rather than through SQL operations. You might wish to keep the configuration files in a location that will be backed up by your regular file system backup procedures. See Section 20.2 for how to relocate the configuration files.

The archive command is only invoked on completed WAL segments. Hence, if your server generates only little WAL traffic (or has slack periods where it does so), there could be a long delay between the completion of a transaction and its safe recording in archive storage. To put a limit on how old unarchived data can be, you can set archive_timeout to force the server to switch to a new WAL segment file at least that often. Note that archived files that are archived early due to a forced switch are still the same length as completely full files. It is therefore unwise to set a very short archive_timeout — it will bloat your archive storage. archive_timeout settings of a minute or so are usually reasonable.

Also, you can force a segment switch manually with pg_switch_wal if you want to ensure that a just-finished transaction is archived as soon as possible. Other utility functions related to WAL management are listed in Table 9.87.

When wal_level is minimal some SQL commands are optimized to avoid WAL logging, as described in Section 14.4.7. If archiving or streaming replication were turned on during execution of one of these statements, WAL would not contain enough information for archive recovery. (Crash recovery is unaffected.) For this reason, wal_level can only be changed at server start. However, archive_command can be changed with a configuration file reload. If you wish to temporarily stop archiving, one way to do it is to set archive_command to the empty string (''). This will cause WAL files to accumulate in pg_wal/ until a working archive_command is re-established.

26.3.2. Making a Base Backup

The easiest way to perform a base backup is to use the pg_basebackup tool. It can create a base backup either as regular files or as a tar archive. If more flexibility than pg_basebackup can provide is required, you can also make a base backup using the low level API (see Section 26.3.3).

It is not necessary to be concerned about the amount of time it takes to make a base backup. However, if you normally run the server with full_page_writes disabled, you might notice a drop in performance while the backup runs since full_page_writes is effectively forced on during backup mode.

To make use of the backup, you will need to keep all the WAL segment files generated during and after the file system backup. To aid you in doing this, the base backup process creates a backup history file that is immediately stored into the WAL archive area. This file is named after the first WAL segment file that you need for the file system backup. For example, if the starting WAL file is 0000000100001234000055CD the backup history file will be named something like 0000000100001234000055CD.007C9330.backup. (The second part of the file name stands for an exact position within the WAL file, and can ordinarily be ignored.) Once you have safely archived the file system backup and the WAL segment files used during the backup (as specified in the backup history file), all archived WAL segments with names numerically less are no longer needed to recover the file system backup and can be deleted. However, you should consider keeping several backup sets to be absolutely certain that you can recover your data.

The backup history file is just a small text file. It contains the label string you gave to pg_basebackup, as well as the starting and ending times and WAL segments of the backup. If you used the label to identify the associated dump file, then the archived history file is enough to tell you which dump file to restore.

Since you have to keep around all the archived WAL files back to your last base backup, the interval between base backups should usually be chosen based on how much storage you want to expend on archived WAL files. You should also consider how long you are prepared to spend recovering, if recovery should be necessary — the system will have to replay all those WAL segments, and that could take awhile if it has been a long time since the last base backup.

26.3.3. Making a Base Backup Using the Low Level API

The procedure for making a base backup using the low level APIs contains a few more steps than the pg_basebackup method, but is relatively simple. It is very important that these steps are executed in sequence, and that the success of a step is verified before proceeding to the next step.

Low level base backups can be made in a non-exclusive or an exclusive way. The non-exclusive method is recommended and the exclusive one is deprecated and will eventually be removed.

26.3.3.1. Making a Non-Exclusive Low-Level Backup

A non-exclusive low level backup is one that allows other concurrent backups to be running (both those started using the same backup API and those started using pg_basebackup).

  1. Ensure that WAL archiving is enabled and working.

  2. Connect to the server (it does not matter which database) as a user with rights to run pg_start_backup (superuser, or a user who has been granted EXECUTE on the function) and issue the command:

    SELECT pg_start_backup('label', false, false);

    where label is any string you want to use to uniquely identify this backup operation. The connection calling pg_start_backup must be maintained until the end of the backup, or the backup will be automatically aborted.

    By default, pg_start_backup can take a long time to finish. This is because it performs a checkpoint, and the I/O required for the checkpoint will be spread out over a significant period of time, by default half your inter-checkpoint interval (see the configuration parameter checkpoint_completion_target). This is usually what you want, because it minimizes the impact on query processing. If you want to start the backup as soon as possible, change the second parameter to true, which will issue an immediate checkpoint using as much I/O as available.

    The third parameter being false tells pg_start_backup to initiate a non-exclusive base backup.

  3. Perform the backup, using any convenient file-system-backup tool such as tar or cpio (not pg_dump or pg_dumpall). It is neither necessary nor desirable to stop normal operation of the database while you do this. See Section 26.3.3.3 for things to consider during this backup.

  4. In the same connection as before, issue the command:

    SELECT * FROM pg_stop_backup(false, true);

    This terminates backup mode. On a primary, it also performs an automatic switch to the next WAL segment. On a standby, it is not possible to automatically switch WAL segments, so you may wish to run pg_switch_wal on the primary to perform a manual switch. The reason for the switch is to arrange for the last WAL segment file written during the backup interval to be ready to archive.

    The pg_stop_backup will return one row with three values. The second of these fields should be written to a file named backup_label in the root directory of the backup. The third field should be written to a file named tablespace_map unless the field is empty. These files are vital to the backup working and must be written byte for byte without modification, which may require opening the file in binary mode.

  5. Once the WAL segment files active during the backup are archived, you are done. The file identified by pg_stop_backup's first return value is the last segment that is required to form a complete set of backup files. On a primary, if archive_mode is enabled and the wait_for_archive parameter is true, pg_stop_backup does not return until the last segment has been archived. On a standby, archive_mode must be always in order for pg_stop_backup to wait. Archiving of these files happens automatically since you have already configured archive_command. In most cases this happens quickly, but you are advised to monitor your archive system to ensure there are no delays. If the archive process has fallen behind because of failures of the archive command, it will keep retrying until the archive succeeds and the backup is complete. If you wish to place a time limit on the execution of pg_stop_backup, set an appropriate statement_timeout value, but make note that if pg_stop_backup terminates because of this your backup may not be valid.

    If the backup process monitors and ensures that all WAL segment files required for the backup are successfully archived then the wait_for_archive parameter (which defaults to true) can be set to false to have pg_stop_backup return as soon as the stop backup record is written to the WAL. By default, pg_stop_backup will wait until all WAL has been archived, which can take some time. This option must be used with caution: if WAL archiving is not monitored correctly then the backup might not include all of the WAL files and will therefore be incomplete and not able to be restored.

26.3.3.2. Making an Exclusive Low-Level Backup

Note

The exclusive backup method is deprecated and should be avoided. Prior to PostgreSQL 9.6, this was the only low-level method available, but it is now recommended that all users upgrade their scripts to use non-exclusive backups.

The process for an exclusive backup is mostly the same as for a non-exclusive one, but it differs in a few key steps. This type of backup can only be taken on a primary and does not allow concurrent backups. Moreover, because it creates a backup label file, as described below, it can block automatic restart of the primary server after a crash. On the other hand, the erroneous removal of this file from a backup or standby is a common mistake, which can result in serious data corruption. If it is necessary to use this method, the following steps may be used.

  1. Ensure that WAL archiving is enabled and working.

  2. Connect to the server (it does not matter which database) as a user with rights to run pg_start_backup (superuser, or a user who has been granted EXECUTE on the function) and issue the command:

    SELECT pg_start_backup('label');

    where label is any string you want to use to uniquely identify this backup operation. pg_start_backup creates a backup label file, called backup_label, in the cluster directory with information about your backup, including the start time and label string. The function also creates a tablespace map file, called tablespace_map, in the cluster directory with information about tablespace symbolic links in pg_tblspc/ if one or more such link is present. Both files are critical to the integrity of the backup, should you need to restore from it.

    By default, pg_start_backup can take a long time to finish. This is because it performs a checkpoint, and the I/O required for the checkpoint will be spread out over a significant period of time, by default half your inter-checkpoint interval (see the configuration parameter checkpoint_completion_target). This is usually what you want, because it minimizes the impact on query processing. If you want to start the backup as soon as possible, use:

    SELECT pg_start_backup('label', true);

    This forces the checkpoint to be done as quickly as possible.

  3. Perform the backup, using any convenient file-system-backup tool such as tar or cpio (not pg_dump or pg_dumpall). It is neither necessary nor desirable to stop normal operation of the database while you do this. See Section 26.3.3.3 for things to consider during this backup.

    As noted above, if the server crashes during the backup it may not be possible to restart until the backup_label file has been manually deleted from the PGDATA directory. Note that it is very important to never remove the backup_label file when restoring a backup, because this will result in corruption. Confusion about when it is appropriate to remove this file is a common cause of data corruption when using this method; be very certain that you remove the file only on an existing primary and never when building a standby or restoring a backup, even if you are building a standby that will subsequently be promoted to a new primary.

  4. Again connect to the database as a user with rights to run pg_stop_backup (superuser, or a user who has been granted EXECUTE on the function), and issue the command:

    SELECT pg_stop_backup();

    This function terminates backup mode and performs an automatic switch to the next WAL segment. The reason for the switch is to arrange for the last WAL segment written during the backup interval to be ready to archive.

  5. Once the WAL segment files active during the backup are archived, you are done. The file identified by pg_stop_backup's result is the last segment that is required to form a complete set of backup files. If archive_mode is enabled, pg_stop_backup does not return until the last segment has been archived. Archiving of these files happens automatically since you have already configured archive_command. In most cases this happens quickly, but you are advised to monitor your archive system to ensure there are no delays. If the archive process has fallen behind because of failures of the archive command, it will keep retrying until the archive succeeds and the backup is complete.

    When using exclusive backup mode, it is absolutely imperative to ensure that pg_stop_backup completes successfully at the end of the backup. Even if the backup itself fails, for example due to lack of disk space, failure to call pg_stop_backup will leave the server in backup mode indefinitely, causing future backups to fail and increasing the risk of a restart failure during the time that backup_label exists.

26.3.3.3. Backing Up the Data Directory

Some file system backup tools emit warnings or errors if the files they are trying to copy change while the copy proceeds. When taking a base backup of an active database, this situation is normal and not an error. However, you need to ensure that you can distinguish complaints of this sort from real errors. For example, some versions of rsync return a separate exit code for “vanished source files”, and you can write a driver script to accept this exit code as a non-error case. Also, some versions of GNU tar return an error code indistinguishable from a fatal error if a file was truncated while tar was copying it. Fortunately, GNU tar versions 1.16 and later exit with 1 if a file was changed during the backup, and 2 for other errors. With GNU tar version 1.23 and later, you can use the warning options --warning=no-file-changed --warning=no-file-removed to hide the related warning messages.

Be certain that your backup includes all of the files under the database cluster directory (e.g., /usr/local/pgsql/data). If you are using tablespaces that do not reside underneath this directory, be careful to include them as well (and be sure that your backup archives symbolic links as links, otherwise the restore will corrupt your tablespaces).

You should, however, omit from the backup the files within the cluster's pg_wal/ subdirectory. This slight adjustment is worthwhile because it reduces the risk of mistakes when restoring. This is easy to arrange if pg_wal/ is a symbolic link pointing to someplace outside the cluster directory, which is a common setup anyway for performance reasons. You might also want to exclude postmaster.pid and postmaster.opts, which record information about the running postmaster, not about the postmaster which will eventually use this backup. (These files can confuse pg_ctl.)

It is often a good idea to also omit from the backup the files within the cluster's pg_replslot/ directory, so that replication slots that exist on the primary do not become part of the backup. Otherwise, the subsequent use of the backup to create a standby may result in indefinite retention of WAL files on the standby, and possibly bloat on the primary if hot standby feedback is enabled, because the clients that are using those replication slots will still be connecting to and updating the slots on the primary, not the standby. Even if the backup is only intended for use in creating a new primary, copying the replication slots isn't expected to be particularly useful, since the contents of those slots will likely be badly out of date by the time the new primary comes on line.

The contents of the directories pg_dynshmem/, pg_notify/, pg_serial/, pg_snapshots/, pg_stat_tmp/, and pg_subtrans/ (but not the directories themselves) can be omitted from the backup as they will be initialized on postmaster startup. If stats_temp_directory is set and is under the data directory then the contents of that directory can also be omitted.

Any file or directory beginning with pgsql_tmp can be omitted from the backup. These files are removed on postmaster start and the directories will be recreated as needed.

pg_internal.init files can be omitted from the backup whenever a file of that name is found. These files contain relation cache data that is always rebuilt when recovering.

The backup label file includes the label string you gave to pg_start_backup, as well as the time at which pg_start_backup was run, and the name of the starting WAL file. In case of confusion it is therefore possible to look inside a backup file and determine exactly which backup session the dump file came from. The tablespace map file includes the symbolic link names as they exist in the directory pg_tblspc/ and the full path of each symbolic link. These files are not merely for your information; their presence and contents are critical to the proper operation of the system's recovery process.

It is also possible to make a backup while the server is stopped. In this case, you obviously cannot use pg_start_backup or pg_stop_backup, and you will therefore be left to your own devices to keep track of which backup is which and how far back the associated WAL files go. It is generally better to follow the continuous archiving procedure above.

26.3.4. Recovering Using a Continuous Archive Backup

Okay, the worst has happened and you need to recover from your backup. Here is the procedure:

  1. Stop the server, if it's running.

  2. If you have the space to do so, copy the whole cluster data directory and any tablespaces to a temporary location in case you need them later. Note that this precaution will require that you have enough free space on your system to hold two copies of your existing database. If you do not have enough space, you should at least save the contents of the cluster's pg_wal subdirectory, as it might contain logs which were not archived before the system went down.

  3. Remove all existing files and subdirectories under the cluster data directory and under the root directories of any tablespaces you are using.

  4. Restore the database files from your file system backup. Be sure that they are restored with the right ownership (the database system user, not root!) and with the right permissions. If you are using tablespaces, you should verify that the symbolic links in pg_tblspc/ were correctly restored.

  5. Remove any files present in pg_wal/; these came from the file system backup and are therefore probably obsolete rather than current. If you didn't archive pg_wal/ at all, then recreate it with proper permissions, being careful to ensure that you re-establish it as a symbolic link if you had it set up that way before.

  6. If you have unarchived WAL segment files that you saved in step 2, copy them into pg_wal/. (It is best to copy them, not move them, so you still have the unmodified files if a problem occurs and you have to start over.)

  7. Set recovery configuration settings in postgresql.conf (see Section 20.5.4) and create a file recovery.signal in the cluster data directory. You might also want to temporarily modify pg_hba.conf to prevent ordinary users from connecting until you are sure the recovery was successful.

  8. Start the server. The server will go into recovery mode and proceed to read through the archived WAL files it needs. Should the recovery be terminated because of an external error, the server can simply be restarted and it will continue recovery. Upon completion of the recovery process, the server will remove recovery.signal (to prevent accidentally re-entering recovery mode later) and then commence normal database operations.

  9. Inspect the contents of the database to ensure you have recovered to the desired state. If not, return to step 1. If all is well, allow your users to connect by restoring pg_hba.conf to normal.

The key part of all this is to set up a recovery configuration that describes how you want to recover and how far the recovery should run. The one thing that you absolutely must specify is the restore_command, which tells PostgreSQL how to retrieve archived WAL file segments. Like the archive_command, this is a shell command string. It can contain %f, which is replaced by the name of the desired log file, and %p, which is replaced by the path name to copy the log file to. (The path name is relative to the current working directory, i.e., the cluster's data directory.) Write %% if you need to embed an actual % character in the command. The simplest useful command is something like:

restore_command = 'cp /mnt/server/archivedir/%f %p'

which will copy previously archived WAL segments from the directory /mnt/server/archivedir. Of course, you can use something much more complicated, perhaps even a shell script that requests the operator to mount an appropriate tape.

It is important that the command return nonzero exit status on failure. The command will be called requesting files that are not present in the archive; it must return nonzero when so asked. This is not an error condition. An exception is that if the command was terminated by a signal (other than SIGTERM, which is used as part of a database server shutdown) or an error by the shell (such as command not found), then recovery will abort and the server will not start up.

Not all of the requested files will be WAL segment files; you should also expect requests for files with a suffix of .history. Also be aware that the base name of the %p path will be different from %f; do not expect them to be interchangeable.

WAL segments that cannot be found in the archive will be sought in pg_wal/; this allows use of recent un-archived segments. However, segments that are available from the archive will be used in preference to files in pg_wal/.

Normally, recovery will proceed through all available WAL segments, thereby restoring the database to the current point in time (or as close as possible given the available WAL segments). Therefore, a normal recovery will end with a “file not found” message, the exact text of the error message depending upon your choice of restore_command. You may also see an error message at the start of recovery for a file named something like 00000001.history. This is also normal and does not indicate a problem in simple recovery situations; see Section 26.3.5 for discussion.

If you want to recover to some previous point in time (say, right before the junior DBA dropped your main transaction table), just specify the required stopping point. You can specify the stop point, known as the “recovery target”, either by date/time, named restore point or by completion of a specific transaction ID. As of this writing only the date/time and named restore point options are very usable, since there are no tools to help you identify with any accuracy which transaction ID to use.

Note

The stop point must be after the ending time of the base backup, i.e., the end time of pg_stop_backup. You cannot use a base backup to recover to a time when that backup was in progress. (To recover to such a time, you must go back to your previous base backup and roll forward from there.)

If recovery finds corrupted WAL data, recovery will halt at that point and the server will not start. In such a case the recovery process could be re-run from the beginning, specifying a “recovery target” before the point of corruption so that recovery can complete normally. If recovery fails for an external reason, such as a system crash or if the WAL archive has become inaccessible, then the recovery can simply be restarted and it will restart almost from where it failed. Recovery restart works much like checkpointing in normal operation: the server periodically forces all its state to disk, and then updates the pg_control file to indicate that the already-processed WAL data need not be scanned again.

26.3.5. Timelines

The ability to restore the database to a previous point in time creates some complexities that are akin to science-fiction stories about time travel and parallel universes. For example, in the original history of the database, suppose you dropped a critical table at 5:15PM on Tuesday evening, but didn't realize your mistake until Wednesday noon. Unfazed, you get out your backup, restore to the point-in-time 5:14PM Tuesday evening, and are up and running. In this history of the database universe, you never dropped the table. But suppose you later realize this wasn't such a great idea, and would like to return to sometime Wednesday morning in the original history. You won't be able to if, while your database was up-and-running, it overwrote some of the WAL segment files that led up to the time you now wish you could get back to. Thus, to avoid this, you need to distinguish the series of WAL records generated after you've done a point-in-time recovery from those that were generated in the original database history.

To deal with this problem, PostgreSQL has a notion of timelines. Whenever an archive recovery completes, a new timeline is created to identify the series of WAL records generated after that recovery. The timeline ID number is part of WAL segment file names so a new timeline does not overwrite the WAL data generated by previous timelines. It is in fact possible to archive many different timelines. While that might seem like a useless feature, it's often a lifesaver. Consider the situation where you aren't quite sure what point-in-time to recover to, and so have to do several point-in-time recoveries by trial and error until you find the best place to branch off from the old history. Without timelines this process would soon generate an unmanageable mess. With timelines, you can recover to any prior state, including states in timeline branches that you abandoned earlier.

Every time a new timeline is created, PostgreSQL creates a “timeline history” file that shows which timeline it branched off from and when. These history files are necessary to allow the system to pick the right WAL segment files when recovering from an archive that contains multiple timelines. Therefore, they are archived into the WAL archive area just like WAL segment files. The history files are just small text files, so it's cheap and appropriate to keep them around indefinitely (unlike the segment files which are large). You can, if you like, add comments to a history file to record your own notes about how and why this particular timeline was created. Such comments will be especially valuable when you have a thicket of different timelines as a result of experimentation.

The default behavior of recovery is to recover to the latest timeline found in the archive. If you wish to recover to the timeline that was current when the base backup was taken or into a specific child timeline (that is, you want to return to some state that was itself generated after a recovery attempt), you need to specify current or the target timeline ID in recovery_target_timeline. You cannot recover into timelines that branched off earlier than the base backup.

26.3.6. Tips and Examples

Some tips for configuring continuous archiving are given here.

26.3.6.1. Standalone Hot Backups

It is possible to use PostgreSQL's backup facilities to produce standalone hot backups. These are backups that cannot be used for point-in-time recovery, yet are typically much faster to backup and restore than pg_dump dumps. (They are also much larger than pg_dump dumps, so in some cases the speed advantage might be negated.)

As with base backups, the easiest way to produce a standalone hot backup is to use the pg_basebackup tool. If you include the -X parameter when calling it, all the write-ahead log required to use the backup will be included in the backup automatically, and no special action is required to restore the backup.

If more flexibility in copying the backup files is needed, a lower level process can be used for standalone hot backups as well. To prepare for low level standalone hot backups, make sure wal_level is set to replica or higher, archive_mode to on, and set up an archive_command that performs archiving only when a switch file exists. For example:

archive_command = 'test ! -f /var/lib/pgsql/backup_in_progress || (test ! -f /var/lib/pgsql/archive/%f && cp %p /var/lib/pgsql/archive/%f)'

This command will perform archiving when /var/lib/pgsql/backup_in_progress exists, and otherwise silently return zero exit status (allowing PostgreSQL to recycle the unwanted WAL file).

With this preparation, a backup can be taken using a script like the following:

touch /var/lib/pgsql/backup_in_progress
psql -c "select pg_start_backup('hot_backup');"
tar -cf /var/lib/pgsql/backup.tar /var/lib/pgsql/data/
psql -c "select pg_stop_backup();"
rm /var/lib/pgsql/backup_in_progress
tar -rf /var/lib/pgsql/backup.tar /var/lib/pgsql/archive/

The switch file /var/lib/pgsql/backup_in_progress is created first, enabling archiving of completed WAL files to occur. After the backup the switch file is removed. Archived WAL files are then added to the backup so that both base backup and all required WAL files are part of the same tar file. Please remember to add error handling to your backup scripts.

26.3.6.2. Compressed Archive Logs

If archive storage size is a concern, you can use gzip to compress the archive files:

archive_command = 'gzip < %p > /mnt/server/archivedir/%f.gz'

You will then need to use gunzip during recovery:

restore_command = 'gunzip < /mnt/server/archivedir/%f.gz > %p'

26.3.6.3. archive_command Scripts

Many people choose to use scripts to define their archive_command, so that their postgresql.conf entry looks very simple:

archive_command = 'local_backup_script.sh "%p" "%f"'

Using a separate script file is advisable any time you want to use more than a single command in the archiving process. This allows all complexity to be managed within the script, which can be written in a popular scripting language such as bash or perl.

Examples of requirements that might be solved within a script include:

  • Copying data to secure off-site data storage

  • Batching WAL files so that they are transferred every three hours, rather than one at a time

  • Interfacing with other backup and recovery software

  • Interfacing with monitoring software to report errors

Tip

When using an archive_command script, it's desirable to enable logging_collector. Any messages written to stderr from the script will then appear in the database server log, allowing complex configurations to be diagnosed easily if they fail.

26.3.7. Caveats

At this writing, there are several limitations of the continuous archiving technique. These will probably be fixed in future releases:

  • If a CREATE DATABASE command is executed while a base backup is being taken, and then the template database that the CREATE DATABASE copied is modified while the base backup is still in progress, it is possible that recovery will cause those modifications to be propagated into the created database as well. This is of course undesirable. To avoid this risk, it is best not to modify any template databases while taking a base backup.

  • CREATE TABLESPACE commands are WAL-logged with the literal absolute path, and will therefore be replayed as tablespace creations with the same absolute path. This might be undesirable if the log is being replayed on a different machine. It can be dangerous even if the log is being replayed on the same machine, but into a new data directory: the replay will still overwrite the contents of the original tablespace. To avoid potential gotchas of this sort, the best practice is to take a new base backup after creating or dropping tablespaces.

It should also be noted that the default WAL format is fairly bulky since it includes many disk page snapshots. These page snapshots are designed to support crash recovery, since we might need to fix partially-written disk pages. Depending on your system hardware and software, the risk of partial writes might be small enough to ignore, in which case you can significantly reduce the total volume of archived logs by turning off page snapshots using the full_page_writes parameter. (Read the notes and warnings in Chapter 30 before you do so.) Turning off page snapshots does not prevent use of the logs for PITR operations. An area for future development is to compress archived WAL data by removing unnecessary page copies even when full_page_writes is on. In the meantime, administrators might wish to reduce the number of page snapshots included in WAL by increasing the checkpoint interval parameters as much as feasible.

Prev | Up | Next
26.2. File System Level Backup | Home | Chapter 27. High Availability, Load Balancing, and Replication


Chapter 27. High Availability, Load Balancing, and Replication
Prev Up Part III. Server Administration Home Next

Chapter 27. High Availability, Load Balancing, and Replication

Table of Contents

27.1. Comparison of Different Solutions
27.2. Log-Shipping Standby Servers
27.2.1. Planning
27.2.2. Standby Server Operation
27.2.3. Preparing the Primary for Standby Servers
27.2.4. Setting Up a Standby Server
27.2.5. Streaming Replication
27.2.6. Replication Slots
27.2.7. Cascading Replication
27.2.8. Synchronous Replication
27.2.9. Continuous Archiving in Standby
27.3. Failover
27.4. Hot Standby
27.4.1. User's Overview
27.4.2. Handling Query Conflicts
27.4.3. Administrator's Overview
27.4.4. Hot Standby Parameter Reference
27.4.5. Caveats

Database servers can work together to allow a second server to take over quickly if the primary server fails (high availability), or to allow several computers to serve the same data (load balancing). Ideally, database servers could work together seamlessly. Web servers serving static web pages can be combined quite easily by merely load-balancing web requests to multiple machines. In fact, read-only database servers can be combined relatively easily too. Unfortunately, most database servers have a read/write mix of requests, and read/write servers are much harder to combine. This is because though read-only data needs to be placed on each server only once, a write to any server has to be propagated to all servers so that future read requests to those servers return consistent results.

This synchronization problem is the fundamental difficulty for servers working together. Because there is no single solution that eliminates the impact of the sync problem for all use cases, there are multiple solutions. Each solution addresses this problem in a different way, and minimizes its impact for a specific workload.

Some solutions deal with synchronization by allowing only one server to modify the data. Servers that can modify data are called read/write, master or primary servers. Servers that track changes in the primary are called standby or secondary servers. A standby server that cannot be connected to until it is promoted to a primary server is called a warm standby server, and one that can accept connections and serves read-only queries is called a hot standby server.

Some solutions are synchronous, meaning that a data-modifying transaction is not considered committed until all servers have committed the transaction. This guarantees that a failover will not lose any data and that all load-balanced servers will return consistent results no matter which server is queried. In contrast, asynchronous solutions allow some delay between the time of a commit and its propagation to the other servers, opening the possibility that some transactions might be lost in the switch to a backup server, and that load balanced servers might return slightly stale results. Asynchronous communication is used when synchronous would be too slow.

Solutions can also be categorized by their granularity. Some solutions can deal only with an entire database server, while others allow control at the per-table or per-database level.

Performance must be considered in any choice. There is usually a trade-off between functionality and performance. For example, a fully synchronous solution over a slow network might cut performance by more than half, while an asynchronous one might have a minimal performance impact.

The remainder of this section outlines various failover, replication, and load balancing solutions.

Prev | Up | Next
26.3. Continuous Archiving and Point-in-Time Recovery (PITR) | Home | 27.1. Comparison of Different Solutions


27.1. Comparison of Different Solutions
Prev Up Chapter 27. High Availability, Load Balancing, and Replication Home Next

27.1. Comparison of Different Solutions

Shared Disk Failover

Shared disk failover avoids synchronization overhead by having only one copy of the database. It uses a single disk array that is shared by multiple servers. If the main database server fails, the standby server is able to mount and start the database as though it were recovering from a database crash. This allows rapid failover with no data loss.

Shared hardware functionality is common in network storage devices. Using a network file system is also possible, though care must be taken that the file system has full POSIX behavior (see Section 19.2.2.1). One significant limitation of this method is that if the shared disk array fails or becomes corrupt, the primary and standby servers are both nonfunctional. Another issue is that the standby server should never access the shared storage while the primary server is running.

File System (Block Device) Replication

A modified version of shared hardware functionality is file system replication, where all changes to a file system are mirrored to a file system residing on another computer. The only restriction is that the mirroring must be done in a way that ensures the standby server has a consistent copy of the file system — specifically, writes to the standby must be done in the same order as those on the primary. DRBD is a popular file system replication solution for Linux.

Write-Ahead Log Shipping

Warm and hot standby servers can be kept current by reading a stream of write-ahead log (WAL) records. If the main server fails, the standby contains almost all of the data of the main server, and can be quickly made the new primary database server. This can be synchronous or asynchronous and can only be done for the entire database server.

A standby server can be implemented using file-based log shipping (Section 27.2) or streaming replication (see Section 27.2.5), or a combination of both. For information on hot standby, see Section 27.4.

Logical Replication

Logical replication allows a database server to send a stream of data modifications to another server. PostgreSQL logical replication constructs a stream of logical data modifications from the WAL. Logical replication allows replication of data changes on a per-table basis. In addition, a server that is publishing its own changes can also subscribe to changes from another server, allowing data to flow in multiple directions. For more information on logical replication, see Chapter 31. Through the logical decoding interface (Chapter 49), third-party extensions can also provide similar functionality.

Trigger-Based Primary-Standby Replication

A trigger-based replication setup typically funnels data modification queries to a designated primary server. Operating on a per-table basis, the primary server sends data changes (typically) asynchronously to the standby servers. Standby servers can answer queries while the primary is running, and may allow some local data changes or write activity. This form of replication is often used for offloading large analytical or data warehouse queries.

Slony-I is an example of this type of replication, with per-table granularity, and support for multiple standby servers. Because it updates the standby server asynchronously (in batches), there is possible data loss during fail over.

SQL-Based Replication Middleware

With SQL-based replication middleware, a program intercepts every SQL query and sends it to one or all servers. Each server operates independently. Read-write queries must be sent to all servers, so that every server receives any changes. But read-only queries can be sent to just one server, allowing the read workload to be distributed among them.

If queries are simply broadcast unmodified, functions like random(), CURRENT_TIMESTAMP, and sequences can have different values on different servers. This is because each server operates independently, and because SQL queries are broadcast rather than actual data changes. If this is unacceptable, either the middleware or the application must determine such values from a single source and then use those values in write queries. Care must also be taken that all transactions either commit or abort on all servers, perhaps using two-phase commit (PREPARE TRANSACTION and COMMIT PREPARED). Pgpool-II and Continuent Tungsten are examples of this type of replication.

Asynchronous Multimaster Replication

For servers that are not regularly connected or have slow communication links, like laptops or remote servers, keeping data consistent among servers is a challenge. Using asynchronous multimaster replication, each server works independently, and periodically communicates with the other servers to identify conflicting transactions. The conflicts can be resolved by users or conflict resolution rules. Bucardo is an example of this type of replication.

Synchronous Multimaster Replication

In synchronous multimaster replication, each server can accept write requests, and modified data is transmitted from the original server to every other server before each transaction commits. Heavy write activity can cause excessive locking and commit delays, leading to poor performance. Read requests can be sent to any server. Some implementations use shared disk to reduce the communication overhead. Synchronous multimaster replication is best for mostly read workloads, though its big advantage is that any server can accept write requests — there is no need to partition workloads between primary and standby servers, and because the data changes are sent from one server to another, there is no problem with non-deterministic functions like random().

PostgreSQL does not offer this type of replication, though PostgreSQL two-phase commit (PREPARE TRANSACTION and COMMIT PREPARED) can be used to implement this in application code or middleware.

Table 27.1 summarizes the capabilities of the various solutions listed above.

Table 27.1. High Availability, Load Balancing, and Replication Feature Matrix

Feature | Shared Disk | File System Repl. | Write-Ahead Log Shipping | Logical Repl. | Trigger-​Based Repl. | SQL Repl. Middle-ware | Async. MM Repl. | Sync. MM Repl.
Popular examples | NAS | DRBD | built-in streaming repl. | built-in logical repl., pglogical | Londiste, Slony | pgpool-II | Bucardo |
Comm. method | shared disk | disk blocks | WAL | logical decoding | table rows | SQL | table rows | table rows and row locks
No special hardware required |  | • | • | • | • | • | • | •
Allows multiple primary servers |  |  |  | • |  | • | • | •
No overhead on primary | • |  | • | • |  | • |  |
No waiting for multiple servers | • |  | with sync off | with sync off | • |  | • |
Primary failure will never lose data | • | • | with sync on | with sync on |  | • |  | •
Replicas accept read-only queries |  |  | with hot standby | • | • | • | • | •
Per-table granularity |  |  |  | • | • |  | • | •
No conflict resolution necessary | • | • | • |  | • | • |  | •

There are a few solutions that do not fit into the above categories:

Data Partitioning

Data partitioning splits tables into data sets. Each set can be modified by only one server. For example, data can be partitioned by offices, e.g., London and Paris, with a server in each office. If queries combining London and Paris data are necessary, an application can query both servers, or primary/standby replication can be used to keep a read-only copy of the other office's data on each server.

Multiple-Server Parallel Query Execution

Many of the above solutions allow multiple servers to handle multiple queries, but none allow a single query to use multiple servers to complete faster. This solution allows multiple servers to work concurrently on a single query. It is usually accomplished by splitting the data among servers and having each server execute its part of the query and return results to a central server where they are combined and returned to the user. This can be implemented using the PL/Proxy tool set.

It should also be noted that because PostgreSQL is open source and easily extended, a number of companies have taken PostgreSQL and created commercial closed-source solutions with unique failover, replication, and load balancing capabilities. These are not discussed here.

Prev | Up | Next
Chapter 27. High Availability, Load Balancing, and Replication | Home | 27.2. Log-Shipping Standby Servers


27.4. Hot Standby
Prev Up Chapter 27. High Availability, Load Balancing, and Replication Home Next

27.4. Hot Standby

27.4.1. User's Overview
27.4.2. Handling Query Conflicts
27.4.3. Administrator's Overview
27.4.4. Hot Standby Parameter Reference
27.4.5. Caveats

Hot Standby is the term used to describe the ability to connect to the server and run read-only queries while the server is in archive recovery or standby mode. This is useful both for replication purposes and for restoring a backup to a desired state with great precision. The term Hot Standby also refers to the ability of the server to move from recovery through to normal operation while users continue running queries and/or keep their connections open.

Running queries in hot standby mode is similar to normal query operation, though there are several usage and administrative differences explained below.

27.4.1. User's Overview

When the hot_standby parameter is set to true on a standby server, it will begin accepting connections once the recovery has brought the system to a consistent state. All such connections are strictly read-only; not even temporary tables may be written.

The data on the standby takes some time to arrive from the primary server so there will be a measurable delay between primary and standby. Running the same query nearly simultaneously on both primary and standby might therefore return differing results. We say that data on the standby is eventually consistent with the primary. Once the commit record for a transaction is replayed on the standby, the changes made by that transaction will be visible to any new snapshots taken on the standby. Snapshots may be taken at the start of each query or at the start of each transaction, depending on the current transaction isolation level. For more details, see Section 13.2.

Transactions started during hot standby may issue the following commands:

  • Query access: SELECT, COPY TO

  • Cursor commands: DECLARE, FETCH, CLOSE

  • Settings: SHOW, SET, RESET

  • Transaction management commands:

    • BEGIN, END, ABORT, START TRANSACTION

    • SAVEPOINT, RELEASE, ROLLBACK TO SAVEPOINT

    • EXCEPTION blocks and other internal subtransactions

  • LOCK TABLE, though only when explicitly in one of these modes: ACCESS SHARE, ROW SHARE or ROW EXCLUSIVE.

  • Plans and resources: PREPARE, EXECUTE, DEALLOCATE, DISCARD

  • Plugins and extensions: LOAD

  • UNLISTEN

Transactions started during hot standby will never be assigned a transaction ID and cannot write to the system write-ahead log. Therefore, the following actions will produce error messages:

  • Data Manipulation Language (DML): INSERT, UPDATE, DELETE, COPY FROM, TRUNCATE. Note that there are no allowed actions that result in a trigger being executed during recovery. This restriction applies even to temporary tables, because table rows cannot be read or written without assigning a transaction ID, which is currently not possible in a Hot Standby environment.

  • Data Definition Language (DDL): CREATE, DROP, ALTER, COMMENT. This restriction applies even to temporary tables, because carrying out these operations would require updating the system catalog tables.

  • SELECT ... FOR SHARE | UPDATE, because row locks cannot be taken without updating the underlying data files.

  • Rules on SELECT statements that generate DML commands.

  • LOCK that explicitly requests a mode higher than ROW EXCLUSIVE MODE.

  • LOCK in short default form, since it requests ACCESS EXCLUSIVE MODE.

  • Transaction management commands that explicitly set non-read-only state:

    • BEGIN READ WRITE, START TRANSACTION READ WRITE

    • SET TRANSACTION READ WRITE, SET SESSION CHARACTERISTICS AS TRANSACTION READ WRITE

    • SET transaction_read_only = off

  • Two-phase commit commands: PREPARE TRANSACTION, COMMIT PREPARED, ROLLBACK PREPARED because even read-only transactions need to write WAL in the prepare phase (the first phase of two phase commit).

  • Sequence updates: nextval(), setval()

  • LISTEN, NOTIFY

In normal operation, “read-only” transactions are allowed to use LISTEN and NOTIFY, so Hot Standby sessions operate under slightly tighter restrictions than ordinary read-only sessions. It is possible that some of these restrictions might be loosened in a future release.

During hot standby, the parameter transaction_read_only is always true and may not be changed. But as long as no attempt is made to modify the database, connections during hot standby will act much like any other database connection. If failover or switchover occurs, the database will switch to normal processing mode. Sessions will remain connected while the server changes mode. Once hot standby finishes, it will be possible to initiate read-write transactions (even from a session begun during hot standby).

Users can determine whether hot standby is currently active for their session by issuing SHOW in_hot_standby. (In server versions before 14, the in_hot_standby parameter did not exist; a workable substitute method for older servers is SHOW transaction_read_only.) In addition, a set of functions (Table 9.88) allow users to access information about the standby server. These allow you to write programs that are aware of the current state of the database. These can be used to monitor the progress of recovery, or to allow you to write complex programs that restore the database to particular states.

27.4.2. Handling Query Conflicts

The primary and standby servers are in many ways loosely connected. Actions on the primary will have an effect on the standby. As a result, there is potential for negative interactions or conflicts between them. The easiest conflict to understand is performance: if a huge data load is taking place on the primary then this will generate a similar stream of WAL records on the standby, so standby queries may contend for system resources, such as I/O.

There are also additional types of conflict that can occur with Hot Standby. These conflicts are hard conflicts in the sense that queries might need to be canceled and, in some cases, sessions disconnected to resolve them. The user is provided with several ways to handle these conflicts. Conflict cases include:

  • Access Exclusive locks taken on the primary server, including both explicit LOCK commands and various DDL actions, conflict with table accesses in standby queries.

  • Dropping a tablespace on the primary conflicts with standby queries using that tablespace for temporary work files.

  • Dropping a database on the primary conflicts with sessions connected to that database on the standby.

  • Application of a vacuum cleanup record from WAL conflicts with standby transactions whose snapshots can still “see” any of the rows to be removed.

  • Application of a vacuum cleanup record from WAL conflicts with queries accessing the target page on the standby, whether or not the data to be removed is visible.

On the primary server, these cases simply result in waiting; and the user might choose to cancel either of the conflicting actions. However, on the standby there is no choice: the WAL-logged action already occurred on the primary so the standby must not fail to apply it. Furthermore, allowing WAL application to wait indefinitely may be very undesirable, because the standby's state will become increasingly far behind the primary's. Therefore, a mechanism is provided to forcibly cancel standby queries that conflict with to-be-applied WAL records.

An example of the problem situation is an administrator on the primary server running DROP TABLE on a table that is currently being queried on the standby server. Clearly the standby query cannot continue if the DROP TABLE is applied on the standby. If this situation occurred on the primary, the DROP TABLE would wait until the other query had finished. But when DROP TABLE is run on the primary, the primary doesn't have information about what queries are running on the standby, so it will not wait for any such standby queries. The WAL change records come through to the standby while the standby query is still running, causing a conflict. The standby server must either delay application of the WAL records (and everything after them, too) or else cancel the conflicting query so that the DROP TABLE can be applied.

When a conflicting query is short, it's typically desirable to allow it to complete by delaying WAL application for a little bit; but a long delay in WAL application is usually not desirable. So the cancel mechanism has parameters, max_standby_archive_delay and max_standby_streaming_delay, that define the maximum allowed delay in WAL application. Conflicting queries will be canceled once it has taken longer than the relevant delay setting to apply any newly-received WAL data. There are two parameters so that different delay values can be specified for the case of reading WAL data from an archive (i.e., initial recovery from a base backup or “catching up” a standby server that has fallen far behind) versus reading WAL data via streaming replication.

In a standby server that exists primarily for high availability, it's best to set the delay parameters relatively short, so that the server cannot fall far behind the primary due to delays caused by standby queries. However, if the standby server is meant for executing long-running queries, then a high or even infinite delay value may be preferable. Keep in mind however that a long-running query could cause other sessions on the standby server to not see recent changes on the primary, if it delays application of WAL records.

Once the delay specified by max_standby_archive_delay or max_standby_streaming_delay has been exceeded, conflicting queries will be canceled. This usually results just in a cancellation error, although in the case of replaying a DROP DATABASE the entire conflicting session will be terminated. Also, if the conflict is over a lock held by an idle transaction, the conflicting session is terminated (this behavior might change in the future).

Canceled queries may be retried immediately (after beginning a new transaction, of course). Since query cancellation depends on the nature of the WAL records being replayed, a query that was canceled may well succeed if it is executed again.

Keep in mind that the delay parameters are compared to the elapsed time since the WAL data was received by the standby server. Thus, the grace period allowed to any one query on the standby is never more than the delay parameter, and could be considerably less if the standby has already fallen behind as a result of waiting for previous queries to complete, or as a result of being unable to keep up with a heavy update load.

The most common reason for conflict between standby queries and WAL replay is “early cleanup”. Normally, PostgreSQL allows cleanup of old row versions when there are no transactions that need to see them to ensure correct visibility of data according to MVCC rules. However, this rule can only be applied for transactions executing on the primary. So it is possible that cleanup on the primary will remove row versions that are still visible to a transaction on the standby.

Experienced users should note that both row version cleanup and row version freezing will potentially conflict with standby queries. Running a manual VACUUM FREEZE is likely to cause conflicts even on tables with no updated or deleted rows.

Users should be clear that tables that are regularly and heavily updated on the primary server will quickly cause cancellation of longer running queries on the standby. In such cases the setting of a finite value for max_standby_archive_delay or max_standby_streaming_delay can be considered similar to setting statement_timeout.

Remedial possibilities exist if the number of standby-query cancellations is found to be unacceptable. The first option is to set the parameter hot_standby_feedback, which prevents VACUUM from removing recently-dead rows and so cleanup conflicts do not occur. If you do this, you should note that this will delay cleanup of dead rows on the primary, which may result in undesirable table bloat. However, the cleanup situation will be no worse than if the standby queries were running directly on the primary server, and you are still getting the benefit of off-loading execution onto the standby. If standby servers connect and disconnect frequently, you might want to make adjustments to handle the period when hot_standby_feedback feedback is not being provided. For example, consider increasing max_standby_archive_delay so that queries are not rapidly canceled by conflicts in WAL archive files during disconnected periods. You should also consider increasing max_standby_streaming_delay to avoid rapid cancellations by newly-arrived streaming WAL entries after reconnection.

Another option is to increase vacuum_defer_cleanup_age on the primary server, so that dead rows will not be cleaned up as quickly as they normally would be. This will allow more time for queries to execute before they are canceled on the standby, without having to set a high max_standby_streaming_delay. However it is difficult to guarantee any specific execution-time window with this approach, since vacuum_defer_cleanup_age is measured in transactions executed on the primary server.

The number of query cancels and the reason for them can be viewed using the pg_stat_database_conflicts system view on the standby server. The pg_stat_database system view also contains summary information.

Users can control whether a log message is produced when WAL replay is waiting longer than deadlock_timeout for conflicts. This is controlled by the log_recovery_conflict_waits parameter.

27.4.3. Administrator's Overview

If hot_standby is on in postgresql.conf (the default value) and there is a standby.signal file present, the server will run in Hot Standby mode. However, it may take some time for Hot Standby connections to be allowed, because the server will not accept connections until it has completed sufficient recovery to provide a consistent state against which queries can run. During this period, clients that attempt to connect will be refused with an error message. To confirm the server has come up, either loop trying to connect from the application, or look for these messages in the server logs:

LOG:  entering standby mode

... then some time later ...

LOG:  consistent recovery state reached
LOG:  database system is ready to accept read-only connections

Consistency information is recorded once per checkpoint on the primary. It is not possible to enable hot standby when reading WAL written during a period when wal_level was not set to replica or logical on the primary. Reaching a consistent state can also be delayed in the presence of both of these conditions:

  • A write transaction has more than 64 subtransactions

  • Very long-lived write transactions

If you are running file-based log shipping ("warm standby"), you might need to wait until the next WAL file arrives, which could be as long as the archive_timeout setting on the primary.

The settings of some parameters determine the size of shared memory for tracking transaction IDs, locks, and prepared transactions. These shared memory structures must be no smaller on a standby than on the primary in order to ensure that the standby does not run out of shared memory during recovery. For example, if the primary had used a prepared transaction but the standby had not allocated any shared memory for tracking prepared transactions, then recovery could not continue until the standby's configuration is changed. The parameters affected are:

  • max_connections

  • max_prepared_transactions

  • max_locks_per_transaction

  • max_wal_senders

  • max_worker_processes

The easiest way to ensure this does not become a problem is to have these parameters set on the standbys to values equal to or greater than on the primary. Therefore, if you want to increase these values, you should do so on all standby servers first, before applying the changes to the primary server. Conversely, if you want to decrease these values, you should do so on the primary server first, before applying the changes to all standby servers. Keep in mind that when a standby is promoted, it becomes the new reference for the required parameter settings for the standbys that follow it. Therefore, to avoid this becoming a problem during a switchover or failover, it is recommended to keep these settings the same on all standby servers.

The WAL tracks changes to these parameters on the primary. If a hot standby processes WAL that indicates that the current value on the primary is higher than its own value, it will log a warning and pause recovery, for example:

WARNING:  hot standby is not possible because of insufficient parameter settings
DETAIL:  max_connections = 80 is a lower setting than on the primary server, where its value was 100.
LOG:  recovery has paused
DETAIL:  If recovery is unpaused, the server will shut down.
HINT:  You can then restart the server after making the necessary configuration changes.

At that point, the settings on the standby need to be updated and the instance restarted before recovery can continue. If the standby is not a hot standby, then when it encounters the incompatible parameter change, it will shut down immediately without pausing, since there is then no value in keeping it up.

It is important that the administrator select appropriate settings for max_standby_archive_delay and max_standby_streaming_delay. The best choices vary depending on business priorities. For example if the server is primarily tasked as a High Availability server, then you will want low delay settings, perhaps even zero, though that is a very aggressive setting. If the standby server is tasked as an additional server for decision support queries then it might be acceptable to set the maximum delay values to many hours, or even -1 which means wait forever for queries to complete.

Transaction status "hint bits" written on the primary are not WAL-logged, so data on the standby will likely re-write the hints again on the standby. Thus, the standby server will still perform disk writes even though all users are read-only; no changes occur to the data values themselves. Users will still write large sort temporary files and re-generate relcache info files, so no part of the database is truly read-only during hot standby mode. Note also that writes to remote databases using dblink module, and other operations outside the database using PL functions will still be possible, even though the transaction is read-only locally.

The following types of administration commands are not accepted during recovery mode:

  • Data Definition Language (DDL): e.g., CREATE INDEX

  • Privilege and Ownership: GRANT, REVOKE, REASSIGN

  • Maintenance commands: ANALYZE, VACUUM, CLUSTER, REINDEX

Again, note that some of these commands are actually allowed during "read only" mode transactions on the primary.

As a result, you cannot create additional indexes that exist solely on the standby, nor statistics that exist solely on the standby. If these administration commands are needed, they should be executed on the primary, and eventually those changes will propagate to the standby.

pg_cancel_backend() and pg_terminate_backend() will work on user backends, but not the Startup process, which performs recovery. pg_stat_activity does not show recovering transactions as active. As a result, pg_prepared_xacts is always empty during recovery. If you wish to resolve in-doubt prepared transactions, view pg_prepared_xacts on the primary and issue commands to resolve transactions there or resolve them after the end of recovery.

pg_locks will show locks held by backends, as normal. pg_locks also shows a virtual transaction managed by the Startup process that owns all AccessExclusiveLocks held by transactions being replayed by recovery. Note that the Startup process does not acquire locks to make database changes, and thus locks other than AccessExclusiveLocks do not show in pg_locks for the Startup process; they are just presumed to exist.

The Nagios plugin check_pgsql will work, because the simple information it checks for exists. The check_postgres monitoring script will also work, though some reported values could give different or confusing results. For example, last vacuum time will not be maintained, since no vacuum occurs on the standby. Vacuums running on the primary do still send their changes to the standby.

WAL file control commands will not work during recovery, e.g., pg_start_backup, pg_switch_wal etc.

Dynamically loadable modules work, including pg_stat_statements.

Advisory locks work normally in recovery, including deadlock detection. Note that advisory locks are never WAL logged, so it is impossible for an advisory lock on either the primary or the standby to conflict with WAL replay. Nor is it possible to acquire an advisory lock on the primary and have it initiate a similar advisory lock on the standby. Advisory locks relate only to the server on which they are acquired.

Trigger-based replication systems such as Slony, Londiste and Bucardo won't run on the standby at all, though they will run happily on the primary server as long as the changes are not sent to standby servers to be applied. WAL replay is not trigger-based so you cannot relay from the standby to any system that requires additional database writes or relies on the use of triggers.

New OIDs cannot be assigned, though some UUID generators may still work as long as they do not rely on writing new status to the database.

Currently, temporary table creation is not allowed during read-only transactions, so in some cases existing scripts will not run correctly. This restriction might be relaxed in a later release. This is both an SQL Standard compliance issue and a technical issue.

DROP TABLESPACE can only succeed if the tablespace is empty. Some standby users may be actively using the tablespace via their temp_tablespaces parameter. If there are temporary files in the tablespace, all active queries are canceled to ensure that temporary files are removed, so the tablespace can be removed and WAL replay can continue.

Running DROP DATABASE or ALTER DATABASE ... SET TABLESPACE on the primary will generate a WAL entry that will cause all users connected to that database on the standby to be forcibly disconnected. This action occurs immediately, whatever the setting of max_standby_streaming_delay. Note that ALTER DATABASE ... RENAME does not disconnect users, which in most cases will go unnoticed, though might in some cases cause a program confusion if it depends in some way upon database name.

In normal (non-recovery) mode, if you issue DROP USER or DROP ROLE for a role with login capability while that user is still connected then nothing happens to the connected user — they remain connected. The user cannot reconnect however. This behavior applies in recovery also, so a DROP USER on the primary does not disconnect that user on the standby.

The statistics collector is active during recovery. All scans, reads, blocks, index usage, etc., will be recorded normally on the standby. Replayed actions will not duplicate their effects on primary, so replaying an insert will not increment the Inserts column of pg_stat_user_tables. The stats file is deleted at the start of recovery, so stats from primary and standby will differ; this is considered a feature, not a bug.

Autovacuum is not active during recovery. It will start normally at the end of recovery.

The checkpointer process and the background writer process are active during recovery. The checkpointer process will perform restartpoints (similar to checkpoints on the primary) and the background writer process will perform normal block cleaning activities. This can include updates of the hint bit information stored on the standby server. The CHECKPOINT command is accepted during recovery, though it performs a restartpoint rather than a new checkpoint.

27.4.4. Hot Standby Parameter Reference

Various parameters have been mentioned above in Section 27.4.2 and Section 27.4.3.

On the primary, parameters wal_level and vacuum_defer_cleanup_age can be used. max_standby_archive_delay and max_standby_streaming_delay have no effect if set on the primary.

On the standby, parameters hot_standby, max_standby_archive_delay and max_standby_streaming_delay can be used. vacuum_defer_cleanup_age has no effect as long as the server remains in standby mode, though it will become relevant if the standby becomes primary.

27.4.5. Caveats

There are several limitations of Hot Standby. These can and probably will be fixed in future releases:

  • Full knowledge of running transactions is required before snapshots can be taken. Transactions that use large numbers of subtransactions (currently greater than 64) will delay the start of read-only connections until the completion of the longest running write transaction. If this situation occurs, explanatory messages will be sent to the server log.

  • Valid starting points for standby queries are generated at each checkpoint on the primary. If the standby is shut down while the primary is in a shutdown state, it might not be possible to re-enter Hot Standby until the primary is started up, so that it generates further starting points in the WAL logs. This situation isn't a problem in the most common situations where it might happen. Generally, if the primary is shut down and not available anymore, that's likely due to a serious failure that requires the standby being converted to operate as the new primary anyway. And in situations where the primary is being intentionally taken down, coordinating to make sure the standby becomes the new primary smoothly is also standard procedure.

  • At the end of recovery, AccessExclusiveLocks held by prepared transactions will require twice the normal number of lock table entries. If you plan on running either a large number of concurrent prepared transactions that normally take AccessExclusiveLocks, or you plan on having one large transaction that takes many AccessExclusiveLocks, you are advised to select a larger value of max_locks_per_transaction, perhaps as much as twice the value of the parameter on the primary server. You need not consider this at all if your setting of max_prepared_transactions is 0.

  • The Serializable transaction isolation level is not yet available in hot standby. (See Section 13.2.3 and Section 13.4.1 for details.) An attempt to set a transaction to the serializable isolation level in hot standby mode will generate an error.

Prev | Up | Next
27.3. Failover | Home | Chapter 28. Monitoring Database Activity


Chapter 28. Monitoring Database Activity
Prev Up Part III. Server Administration Home Next

Chapter 28. Monitoring Database Activity

Table of Contents

28.1. Standard Unix Tools
28.2. The Statistics Collector
28.2.1. Statistics Collection Configuration
28.2.2. Viewing Statistics
28.2.3. pg_stat_activity
28.2.4. pg_stat_replication
28.2.5. pg_stat_replication_slots
28.2.6. pg_stat_wal_receiver
28.2.7. pg_stat_subscription
28.2.8. pg_stat_ssl
28.2.9. pg_stat_gssapi
28.2.10. pg_stat_archiver
28.2.11. pg_stat_bgwriter
28.2.12. pg_stat_wal
28.2.13. pg_stat_database
28.2.14. pg_stat_database_conflicts
28.2.15. pg_stat_all_tables
28.2.16. pg_stat_all_indexes
28.2.17. pg_statio_all_tables
28.2.18. pg_statio_all_indexes
28.2.19. pg_statio_all_sequences
28.2.20. pg_stat_user_functions
28.2.21. pg_stat_slru
28.2.22. Statistics Functions
28.3. Viewing Locks
28.4. Progress Reporting
28.4.1. ANALYZE Progress Reporting
28.4.2. CREATE INDEX Progress Reporting
28.4.3. VACUUM Progress Reporting
28.4.4. CLUSTER Progress Reporting
28.4.5. Base Backup Progress Reporting
28.4.6. COPY Progress Reporting
28.5. Dynamic Tracing
28.5.1. Compiling for Dynamic Tracing
28.5.2. Built-in Probes
28.5.3. Using Probes
28.5.4. Defining New Probes

A database administrator frequently wonders, “What is the system doing right now?” This chapter discusses how to find that out.

Several tools are available for monitoring database activity and analyzing performance. Most of this chapter is devoted to describing PostgreSQL's statistics collector, but one should not neglect regular Unix monitoring programs such as ps, top, iostat, and vmstat. Also, once one has identified a poorly-performing query, further investigation might be needed using PostgreSQL's EXPLAIN command. Section 14.1 discusses EXPLAIN and other methods for understanding the behavior of an individual query.

Prev | Up | Next
27.4. Hot Standby | Home | 28.1. Standard Unix Tools


28.1. Standard Unix Tools
Prev Up Chapter 28. Monitoring Database Activity Home Next

28.1. Standard Unix Tools

On most Unix platforms, PostgreSQL modifies its command title as reported by ps, so that individual server processes can readily be identified. A sample display is

$ ps auxww | grep ^postgres
postgres  15551  0.0  0.1  57536  7132 pts/0    S    18:02   0:00 postgres -i
postgres  15554  0.0  0.0  57536  1184 ?        Ss   18:02   0:00 postgres: background writer
postgres  15555  0.0  0.0  57536   916 ?        Ss   18:02   0:00 postgres: checkpointer
postgres  15556  0.0  0.0  57536   916 ?        Ss   18:02   0:00 postgres: walwriter
postgres  15557  0.0  0.0  58504  2244 ?        Ss   18:02   0:00 postgres: autovacuum launcher
postgres  15558  0.0  0.0  17512  1068 ?        Ss   18:02   0:00 postgres: stats collector
postgres  15582  0.0  0.0  58772  3080 ?        Ss   18:04   0:00 postgres: joe runbug 127.0.0.1 idle
postgres  15606  0.0  0.0  58772  3052 ?        Ss   18:07   0:00 postgres: tgl regression [local] SELECT waiting
postgres  15610  0.0  0.0  58772  3056 ?        Ss   18:07   0:00 postgres: tgl regression [local] idle in transaction

(The appropriate invocation of ps varies across different platforms, as do the details of what is shown. This example is from a recent Linux system.) The first process listed here is the primary server process. The command arguments shown for it are the same ones used when it was launched. The next five processes are background worker processes automatically launched by the primary process. (The “stats collector” process will not be present if you have set the system not to start the statistics collector; likewise the “autovacuum launcher” process can be disabled.) Each of the remaining processes is a server process handling one client connection. Each such process sets its command line display in the form

postgres: user database host activity

The user, database, and (client) host items remain the same for the life of the client connection, but the activity indicator changes. The activity can be idle (i.e., waiting for a client command), idle in transaction (waiting for client inside a BEGIN block), or a command type name such as SELECT. Also, waiting is appended if the server process is presently waiting on a lock held by another session. In the above example we can infer that process 15606 is waiting for process 15610 to complete its transaction and thereby release some lock. (Process 15610 must be the blocker, because there is no other active session. In more complicated cases it would be necessary to look into the pg_locks system view to determine who is blocking whom.)

If cluster_name has been configured the cluster name will also be shown in ps output:

$ psql -c 'SHOW cluster_name'
 cluster_name
--------------
 server1
(1 row)

$ ps aux|grep server1
postgres   27093  0.0  0.0  30096  2752 ?        Ss   11:34   0:00 postgres: server1: background writer
...

If you have turned off update_process_title then the activity indicator is not updated; the process title is set only once when a new process is launched. On some platforms this saves a measurable amount of per-command overhead; on others it's insignificant.

Tip

Solaris requires special handling. You must use /usr/ucb/ps, rather than /bin/ps. You also must use two w flags, not just one. In addition, your original invocation of the postgres command must have a shorter ps status display than that provided by each server process. If you fail to do all three things, the ps output for each server process will be the original postgres command line.

Prev | Up | Next
Chapter 28. Monitoring Database Activity | Home | 28.2. The Statistics Collector


28.2. The Statistics Collector
Prev Up Chapter 28. Monitoring Database Activity Home Next

28.2. The Statistics Collector

28.2.1. Statistics Collection Configuration
28.2.2. Viewing Statistics
28.2.3. pg_stat_activity
28.2.4. pg_stat_replication
28.2.5. pg_stat_replication_slots
28.2.6. pg_stat_wal_receiver
28.2.7. pg_stat_subscription
28.2.8. pg_stat_ssl
28.2.9. pg_stat_gssapi
28.2.10. pg_stat_archiver
28.2.11. pg_stat_bgwriter
28.2.12. pg_stat_wal
28.2.13. pg_stat_database
28.2.14. pg_stat_database_conflicts
28.2.15. pg_stat_all_tables
28.2.16. pg_stat_all_indexes
28.2.17. pg_statio_all_tables
28.2.18. pg_statio_all_indexes
28.2.19. pg_statio_all_sequences
28.2.20. pg_stat_user_functions
28.2.21. pg_stat_slru
28.2.22. Statistics Functions

PostgreSQL's statistics collector is a subsystem that supports collection and reporting of information about server activity. Presently, the collector can count accesses to tables and indexes in both disk-block and individual-row terms. It also tracks the total number of rows in each table, and information about vacuum and analyze actions for each table. It can also count calls to user-defined functions and the total time spent in each one.

PostgreSQL also supports reporting dynamic information about exactly what is going on in the system right now, such as the exact command currently being executed by other server processes, and which other connections exist in the system. This facility is independent of the collector process.

28.2.1. Statistics Collection Configuration

Since collection of statistics adds some overhead to query execution, the system can be configured to collect or not collect information. This is controlled by configuration parameters that are normally set in postgresql.conf. (See Chapter 20 for details about setting configuration parameters.)

The parameter track_activities enables monitoring of the current command being executed by any server process.

The parameter track_counts controls whether statistics are collected about table and index accesses.

The parameter track_functions enables tracking of usage of user-defined functions.

The parameter track_io_timing enables monitoring of block read and write times.

The parameter track_wal_io_timing enables monitoring of WAL write times.

Normally these parameters are set in postgresql.conf so that they apply to all server processes, but it is possible to turn them on or off in individual sessions using the SET command. (To prevent ordinary users from hiding their activity from the administrator, only superusers are allowed to change these parameters with SET.)

The statistics collector transmits the collected information to other PostgreSQL processes through temporary files. These files are stored in the directory named by the stats_temp_directory parameter, pg_stat_tmp by default. For better performance, stats_temp_directory can be pointed at a RAM-based file system, decreasing physical I/O requirements. When the server shuts down cleanly, a permanent copy of the statistics data is stored in the pg_stat subdirectory, so that statistics can be retained across server restarts. When recovery is performed at server start (e.g., after immediate shutdown, server crash, and point-in-time recovery), all statistics counters are reset.

28.2.2. Viewing Statistics

Several predefined views, listed in Table 28.1, are available to show the current state of the system. There are also several other views, listed in Table 28.2, available to show the results of statistics collection. Alternatively, one can build custom views using the underlying statistics functions, as discussed in Section 28.2.22.

When using the statistics to monitor collected data, it is important to realize that the information does not update instantaneously. Each individual server process transmits new statistical counts to the collector just before going idle; so a query or transaction still in progress does not affect the displayed totals. Also, the collector itself emits a new report at most once per PGSTAT_STAT_INTERVAL milliseconds (500 ms unless altered while building the server). So the displayed information lags behind actual activity. However, current-query information collected by track_activities is always up-to-date.

Another important point is that when a server process is asked to display any of these statistics, it first fetches the most recent report emitted by the collector process and then continues to use this snapshot for all statistical views and functions until the end of its current transaction. So the statistics will show static information as long as you continue the current transaction. Similarly, information about the current queries of all sessions is collected when any such information is first requested within a transaction, and the same information will be displayed throughout the transaction. This is a feature, not a bug, because it allows you to perform several queries on the statistics and correlate the results without worrying that the numbers are changing underneath you. But if you want to see new results with each query, be sure to do the queries outside any transaction block. Alternatively, you can invoke pg_stat_clear_snapshot(), which will discard the current transaction's statistics snapshot (if any). The next use of statistical information will cause a new snapshot to be fetched.

A transaction can also see its own statistics (as yet untransmitted to the collector) in the views pg_stat_xact_all_tables, pg_stat_xact_sys_tables, pg_stat_xact_user_tables, and pg_stat_xact_user_functions. These numbers do not act as stated above; instead they update continuously throughout the transaction.

Some of the information in the dynamic statistics views shown in Table 28.1 is security restricted. Ordinary users can only see all the information about their own sessions (sessions belonging to a role that they are a member of). In rows about other sessions, many columns will be null. Note, however, that the existence of a session and its general properties such as its sessions user and database are visible to all users. Superusers and members of the built-in role pg_read_all_stats (see also Section 22.5) can see all the information about all sessions.

Table 28.1. Dynamic Statistics Views

View Name | Description
pg_stat_activity | One row per server process, showing information related to the current activity of that process, such as state and current query. See pg_stat_activity for details.
pg_stat_replication | One row per WAL sender process, showing statistics about replication to that sender's connected standby server. See pg_stat_replication for details.
pg_stat_wal_receiver | Only one row, showing statistics about the WAL receiver from that receiver's connected server. See pg_stat_wal_receiver for details.
pg_stat_subscription | At least one row per subscription, showing information about the subscription workers. See pg_stat_subscription for details.
pg_stat_ssl | One row per connection (regular and replication), showing information about SSL used on this connection. See pg_stat_ssl for details.
pg_stat_gssapi | One row per connection (regular and replication), showing information about GSSAPI authentication and encryption used on this connection. See pg_stat_gssapi for details.
pg_stat_progress_analyze | One row for each backend (including autovacuum worker processes) running ANALYZE, showing current progress. See Section 28.4.1.
pg_stat_progress_create_index | One row for each backend running CREATE INDEX or REINDEX, showing current progress. See Section 28.4.2.
pg_stat_progress_vacuum | One row for each backend (including autovacuum worker processes) running VACUUM, showing current progress. See Section 28.4.3.
pg_stat_progress_cluster | One row for each backend running CLUSTER or VACUUM FULL, showing current progress. See Section 28.4.4.
pg_stat_progress_basebackup | One row for each WAL sender process streaming a base backup, showing current progress. See Section 28.4.5.
pg_stat_progress_copy | One row for each backend running COPY, showing current progress. See Section 28.4.6.

Table 28.2. Collected Statistics Views

View Name | Description
pg_stat_archiver | One row only, showing statistics about the WAL archiver process's activity. See pg_stat_archiver for details.
pg_stat_bgwriter | One row only, showing statistics about the background writer process's activity. See pg_stat_bgwriter for details.
pg_stat_wal | One row only, showing statistics about WAL activity. See pg_stat_wal for details.
pg_stat_database | One row per database, showing database-wide statistics. See pg_stat_database for details.
pg_stat_database_conflicts | One row per database, showing database-wide statistics about query cancels due to conflict with recovery on standby servers. See pg_stat_database_conflicts for details.
pg_stat_all_tables | One row for each table in the current database, showing statistics about accesses to that specific table. See pg_stat_all_tables for details.
pg_stat_sys_tables | Same as pg_stat_all_tables, except that only system tables are shown.
pg_stat_user_tables | Same as pg_stat_all_tables, except that only user tables are shown.
pg_stat_xact_all_tables | Similar to pg_stat_all_tables, but counts actions taken so far within the current transaction (which are not yet included in pg_stat_all_tables and related views). The columns for numbers of live and dead rows and vacuum and analyze actions are not present in this view.
pg_stat_xact_sys_tables | Same as pg_stat_xact_all_tables, except that only system tables are shown.
pg_stat_xact_user_tables | Same as pg_stat_xact_all_tables, except that only user tables are shown.
pg_stat_all_indexes | One row for each index in the current database, showing statistics about accesses to that specific index. See pg_stat_all_indexes for details.
pg_stat_sys_indexes | Same as pg_stat_all_indexes, except that only indexes on system tables are shown.
pg_stat_user_indexes | Same as pg_stat_all_indexes, except that only indexes on user tables are shown.
pg_statio_all_tables | One row for each table in the current database, showing statistics about I/O on that specific table. See pg_statio_all_tables for details.
pg_statio_sys_tables | Same as pg_statio_all_tables, except that only system tables are shown.
pg_statio_user_tables | Same as pg_statio_all_tables, except that only user tables are shown.
pg_statio_all_indexes | One row for each index in the current database, showing statistics about I/O on that specific index. See pg_statio_all_indexes for details.
pg_statio_sys_indexes | Same as pg_statio_all_indexes, except that only indexes on system tables are shown.
pg_statio_user_indexes | Same as pg_statio_all_indexes, except that only indexes on user tables are shown.
pg_statio_all_sequences | One row for each sequence in the current database, showing statistics about I/O on that specific sequence. See pg_statio_all_sequences for details.
pg_statio_sys_sequences | Same as pg_statio_all_sequences, except that only system sequences are shown. (Presently, no system sequences are defined, so this view is always empty.)
pg_statio_user_sequences | Same as pg_statio_all_sequences, except that only user sequences are shown.
pg_stat_user_functions | One row for each tracked function, showing statistics about executions of that function. See pg_stat_user_functions for details.
pg_stat_xact_user_functions | Similar to pg_stat_user_functions, but counts only calls during the current transaction (which are not yet included in pg_stat_user_functions).
pg_stat_slru | One row per SLRU, showing statistics of operations. See pg_stat_slru for details.
pg_stat_replication_slots | One row per replication slot, showing statistics about the replication slot's usage. See pg_stat_replication_slots for details.

The per-index statistics are particularly useful to determine which indexes are being used and how effective they are.

The pg_statio_ views are primarily useful to determine the effectiveness of the buffer cache. When the number of actual disk reads is much smaller than the number of buffer hits, then the cache is satisfying most read requests without invoking a kernel call. However, these statistics do not give the entire story: due to the way in which PostgreSQL handles disk I/O, data that is not in the PostgreSQL buffer cache might still reside in the kernel's I/O cache, and might therefore still be fetched without requiring a physical read. Users interested in obtaining more detailed information on PostgreSQL I/O behavior are advised to use the PostgreSQL statistics collector in combination with operating system utilities that allow insight into the kernel's handling of I/O.

28.2.3. pg_stat_activity

The pg_stat_activity view will have one row per server process, showing information related to the current activity of that process.

Table 28.3. pg_stat_activity View

Column Type

Description

datid oid

OID of the database this backend is connected to

datname name

Name of the database this backend is connected to

pid integer

Process ID of this backend

leader_pid integer

Process ID of the parallel group leader, if this process is a parallel query worker. NULL if this process is a parallel group leader or does not participate in parallel query.

usesysid oid

OID of the user logged into this backend

usename name

Name of the user logged into this backend

application_name text

Name of the application that is connected to this backend

client_addr inet

IP address of the client connected to this backend. If this field is null, it indicates either that the client is connected via a Unix socket on the server machine or that this is an internal process such as autovacuum.

client_hostname text

Host name of the connected client, as reported by a reverse DNS lookup of client_addr. This field will only be non-null for IP connections, and only when log_hostname is enabled.

client_port integer

TCP port number that the client is using for communication with this backend, or -1 if a Unix socket is used. If this field is null, it indicates that this is an internal server process.

backend_start timestamp with time zone

Time when this process was started. For client backends, this is the time the client connected to the server.

xact_start timestamp with time zone

Time when this process' current transaction was started, or null if no transaction is active. If the current query is the first of its transaction, this column is equal to the query_start column.

query_start timestamp with time zone

Time when the currently active query was started, or if state is not active, when the last query was started

state_change timestamp with time zone

Time when the state was last changed

wait_event_type text

The type of event for which the backend is waiting, if any; otherwise NULL. See Table 28.4.

wait_event text

Wait event name if backend is currently waiting, otherwise NULL. See Table 28.5 through Table 28.13.

state text

Current overall state of this backend. Possible values are:

  • active: The backend is executing a query.

  • idle: The backend is waiting for a new client command.

  • idle in transaction: The backend is in a transaction, but is not currently executing a query.

  • idle in transaction (aborted): This state is similar to idle in transaction, except one of the statements in the transaction caused an error.

  • fastpath function call: The backend is executing a fast-path function.

  • disabled: This state is reported if track_activities is disabled in this backend.

backend_xid xid

Top-level transaction identifier of this backend, if any.

backend_xmin xid

The current backend's xmin horizon.

query_id bigint

Identifier of this backend's most recent query. If state is active this field shows the identifier of the currently executing query. In all other states, it shows the identifier of last query that was executed. Query identifiers are not computed by default so this field will be null unless compute_query_id parameter is enabled or a third-party module that computes query identifiers is configured.

query text

Text of this backend's most recent query. If state is active this field shows the currently executing query. In all other states, it shows the last query that was executed. By default the query text is truncated at 1024 bytes; this value can be changed via the parameter track_activity_query_size.

backend_type text

Type of current backend. Possible types are autovacuum launcher, autovacuum worker, logical replication launcher, logical replication worker, parallel worker, background writer, client backend, checkpointer, archiver, startup, walreceiver, walsender and walwriter. In addition, background workers registered by extensions may have additional types.

Note

The wait_event and state columns are independent. If a backend is in the active state, it may or may not be waiting on some event. If the state is active and wait_event is non-null, it means that a query is being executed, but is being blocked somewhere in the system.

Table 28.4. Wait Event Types

Wait Event Type | Description
Activity | The server process is idle. This event type indicates a process waiting for activity in its main processing loop. wait_event will identify the specific wait point; see Table 28.5.
BufferPin | The server process is waiting for exclusive access to a data buffer. Buffer pin waits can be protracted if another process holds an open cursor that last read data from the buffer in question. See Table 28.6.
Client | The server process is waiting for activity on a socket connected to a user application. Thus, the server expects something to happen that is independent of its internal processes. wait_event will identify the specific wait point; see Table 28.7.
Extension | The server process is waiting for some condition defined by an extension module. See Table 28.8.
IO | The server process is waiting for an I/O operation to complete. wait_event will identify the specific wait point; see Table 28.9.
IPC | The server process is waiting for some interaction with another server process. wait_event will identify the specific wait point; see Table 28.10.
Lock | The server process is waiting for a heavyweight lock. Heavyweight locks, also known as lock manager locks or simply locks, primarily protect SQL-visible objects such as tables. However, they are also used to ensure mutual exclusion for certain internal operations such as relation extension. wait_event will identify the type of lock awaited; see Table 28.11.
LWLock | The server process is waiting for a lightweight lock. Most such locks protect a particular data structure in shared memory. wait_event will contain a name identifying the purpose of the lightweight lock. (Some locks have specific names; others are part of a group of locks each with a similar purpose.) See Table 28.12.
Timeout | The server process is waiting for a timeout to expire. wait_event will identify the specific wait point; see Table 28.13.

Table 28.5. Wait Events of Type Activity

Activity Wait Event | Description
ArchiverMain | Waiting in main loop of archiver process.
AutoVacuumMain | Waiting in main loop of autovacuum launcher process.
BgWriterHibernate | Waiting in background writer process, hibernating.
BgWriterMain | Waiting in main loop of background writer process.
CheckpointerMain | Waiting in main loop of checkpointer process.
LogicalApplyMain | Waiting in main loop of logical replication apply process.
LogicalLauncherMain | Waiting in main loop of logical replication launcher process.
PgStatMain | Waiting in main loop of statistics collector process.
RecoveryWalStream | Waiting in main loop of startup process for WAL to arrive, during streaming recovery.
SysLoggerMain | Waiting in main loop of syslogger process.
WalReceiverMain | Waiting in main loop of WAL receiver process.
WalSenderMain | Waiting in main loop of WAL sender process.
WalWriterMain | Waiting in main loop of WAL writer process.

Table 28.6. Wait Events of Type BufferPin

BufferPin Wait Event | Description
BufferPin | Waiting to acquire an exclusive pin on a buffer.

Table 28.7. Wait Events of Type Client

Client Wait Event | Description
ClientRead | Waiting to read data from the client.
ClientWrite | Waiting to write data to the client.
GSSOpenServer | Waiting to read data from the client while establishing a GSSAPI session.
LibPQWalReceiverConnect | Waiting in WAL receiver to establish connection to remote server.
LibPQWalReceiverReceive | Waiting in WAL receiver to receive data from remote server.
SSLOpenServer | Waiting for SSL while attempting connection.
WalSenderWaitForWAL | Waiting for WAL to be flushed in WAL sender process.
WalSenderWriteData | Waiting for any activity when processing replies from WAL receiver in WAL sender process.

Table 28.8. Wait Events of Type Extension

Extension Wait Event | Description
Extension | Waiting in an extension.

Table 28.9. Wait Events of Type IO

IO Wait Event | Description
BaseBackupRead | Waiting for base backup to read from a file.
BufFileRead | Waiting for a read from a buffered file.
BufFileWrite | Waiting for a write to a buffered file.
BufFileTruncate | Waiting for a buffered file to be truncated.
ControlFileRead | Waiting for a read from the pg_control file.
ControlFileSync | Waiting for the pg_control file to reach durable storage.
ControlFileSyncUpdate | Waiting for an update to the pg_control file to reach durable storage.
ControlFileWrite | Waiting for a write to the pg_control file.
ControlFileWriteUpdate | Waiting for a write to update the pg_control file.
CopyFileRead | Waiting for a read during a file copy operation.
CopyFileWrite | Waiting for a write during a file copy operation.
DSMFillZeroWrite | Waiting to fill a dynamic shared memory backing file with zeroes.
DataFileExtend | Waiting for a relation data file to be extended.
DataFileFlush | Waiting for a relation data file to reach durable storage.
DataFileImmediateSync | Waiting for an immediate synchronization of a relation data file to durable storage.
DataFilePrefetch | Waiting for an asynchronous prefetch from a relation data file.
DataFileRead | Waiting for a read from a relation data file.
DataFileSync | Waiting for changes to a relation data file to reach durable storage.
DataFileTruncate | Waiting for a relation data file to be truncated.
DataFileWrite | Waiting for a write to a relation data file.
LockFileAddToDataDirRead | Waiting for a read while adding a line to the data directory lock file.
LockFileAddToDataDirSync | Waiting for data to reach durable storage while adding a line to the data directory lock file.
LockFileAddToDataDirWrite | Waiting for a write while adding a line to the data directory lock file.
LockFileCreateRead | Waiting to read while creating the data directory lock file.
LockFileCreateSync | Waiting for data to reach durable storage while creating the data directory lock file.
LockFileCreateWrite | Waiting for a write while creating the data directory lock file.
LockFileReCheckDataDirRead | Waiting for a read during recheck of the data directory lock file.
LogicalRewriteCheckpointSync | Waiting for logical rewrite mappings to reach durable storage during a checkpoint.
LogicalRewriteMappingSync | Waiting for mapping data to reach durable storage during a logical rewrite.
LogicalRewriteMappingWrite | Waiting for a write of mapping data during a logical rewrite.
LogicalRewriteSync | Waiting for logical rewrite mappings to reach durable storage.
LogicalRewriteTruncate | Waiting for truncate of mapping data during a logical rewrite.
LogicalRewriteWrite | Waiting for a write of logical rewrite mappings.
RelationMapRead | Waiting for a read of the relation map file.
RelationMapSync | Waiting for the relation map file to reach durable storage.
RelationMapWrite | Waiting for a write to the relation map file.
ReorderBufferRead | Waiting for a read during reorder buffer management.
ReorderBufferWrite | Waiting for a write during reorder buffer management.
ReorderLogicalMappingRead | Waiting for a read of a logical mapping during reorder buffer management.
ReplicationSlotRead | Waiting for a read from a replication slot control file.
ReplicationSlotRestoreSync | Waiting for a replication slot control file to reach durable storage while restoring it to memory.
ReplicationSlotSync | Waiting for a replication slot control file to reach durable storage.
ReplicationSlotWrite | Waiting for a write to a replication slot control file.
SLRUFlushSync | Waiting for SLRU data to reach durable storage during a checkpoint or database shutdown.
SLRURead | Waiting for a read of an SLRU page.
SLRUSync | Waiting for SLRU data to reach durable storage following a page write.
SLRUWrite | Waiting for a write of an SLRU page.
SnapbuildRead | Waiting for a read of a serialized historical catalog snapshot.
SnapbuildSync | Waiting for a serialized historical catalog snapshot to reach durable storage.
SnapbuildWrite | Waiting for a write of a serialized historical catalog snapshot.
TimelineHistoryFileSync | Waiting for a timeline history file received via streaming replication to reach durable storage.
TimelineHistoryFileWrite | Waiting for a write of a timeline history file received via streaming replication.
TimelineHistoryRead | Waiting for a read of a timeline history file.
TimelineHistorySync | Waiting for a newly created timeline history file to reach durable storage.
TimelineHistoryWrite | Waiting for a write of a newly created timeline history file.
TwophaseFileRead | Waiting for a read of a two phase state file.
TwophaseFileSync | Waiting for a two phase state file to reach durable storage.
TwophaseFileWrite | Waiting for a write of a two phase state file.
WALBootstrapSync | Waiting for WAL to reach durable storage during bootstrapping.
WALBootstrapWrite | Waiting for a write of a WAL page during bootstrapping.
WALCopyRead | Waiting for a read when creating a new WAL segment by copying an existing one.
WALCopySync | Waiting for a new WAL segment created by copying an existing one to reach durable storage.
WALCopyWrite | Waiting for a write when creating a new WAL segment by copying an existing one.
WALInitSync | Waiting for a newly initialized WAL file to reach durable storage.
WALInitWrite | Waiting for a write while initializing a new WAL file.
WALRead | Waiting for a read from a WAL file.
WALSenderTimelineHistoryRead | Waiting for a read from a timeline history file during a walsender timeline command.
WALSync | Waiting for a WAL file to reach durable storage.
WALSyncMethodAssign | Waiting for data to reach durable storage while assigning a new WAL sync method.
WALWrite | Waiting for a write to a WAL file.
LogicalChangesRead | Waiting for a read from a logical changes file.
LogicalChangesWrite | Waiting for a write to a logical changes file.
LogicalSubxactRead | Waiting for a read from a logical subxact file.
LogicalSubxactWrite | Waiting for a write to a logical subxact file.

Table 28.10. Wait Events of Type IPC

IPC Wait Event | Description
AppendReady | Waiting for subplan nodes of an Append plan node to be ready.
BackendTermination | Waiting for the termination of another backend.
BackupWaitWalArchive | Waiting for WAL files required for a backup to be successfully archived.
BgWorkerShutdown | Waiting for background worker to shut down.
BgWorkerStartup | Waiting for background worker to start up.
BtreePage | Waiting for the page number needed to continue a parallel B-tree scan to become available.
BufferIO | Waiting for buffer I/O to complete.
CheckpointDone | Waiting for a checkpoint to complete.
CheckpointStart | Waiting for a checkpoint to start.
ExecuteGather | Waiting for activity from a child process while executing a Gather plan node.
HashBatchAllocate | Waiting for an elected Parallel Hash participant to allocate a hash table.
HashBatchElect | Waiting to elect a Parallel Hash participant to allocate a hash table.
HashBatchLoad | Waiting for other Parallel Hash participants to finish loading a hash table.
HashBuildAllocate | Waiting for an elected Parallel Hash participant to allocate the initial hash table.
HashBuildElect | Waiting to elect a Parallel Hash participant to allocate the initial hash table.
HashBuildHashInner | Waiting for other Parallel Hash participants to finish hashing the inner relation.
HashBuildHashOuter | Waiting for other Parallel Hash participants to finish partitioning the outer relation.
HashGrowBatchesAllocate | Waiting for an elected Parallel Hash participant to allocate more batches.
HashGrowBatchesDecide | Waiting to elect a Parallel Hash participant to decide on future batch growth.
HashGrowBatchesElect | Waiting to elect a Parallel Hash participant to allocate more batches.
HashGrowBatchesFinish | Waiting for an elected Parallel Hash participant to decide on future batch growth.
HashGrowBatchesRepartition | Waiting for other Parallel Hash participants to finish repartitioning.
HashGrowBucketsAllocate | Waiting for an elected Parallel Hash participant to finish allocating more buckets.
HashGrowBucketsElect | Waiting to elect a Parallel Hash participant to allocate more buckets.
HashGrowBucketsReinsert | Waiting for other Parallel Hash participants to finish inserting tuples into new buckets.
LogicalSyncData | Waiting for a logical replication remote server to send data for initial table synchronization.
LogicalSyncStateChange | Waiting for a logical replication remote server to change state.
MessageQueueInternal | Waiting for another process to be attached to a shared message queue.
MessageQueuePutMessage | Waiting to write a protocol message to a shared message queue.
MessageQueueReceive | Waiting to receive bytes from a shared message queue.
MessageQueueSend | Waiting to send bytes to a shared message queue.
ParallelBitmapScan | Waiting for parallel bitmap scan to become initialized.
ParallelCreateIndexScan | Waiting for parallel CREATE INDEX workers to finish heap scan.
ParallelFinish | Waiting for parallel workers to finish computing.
ProcArrayGroupUpdate | Waiting for the group leader to clear the transaction ID at end of a parallel operation.
ProcSignalBarrier | Waiting for a barrier event to be processed by all backends.
Promote | Waiting for standby promotion.
RecoveryConflictSnapshot | Waiting for recovery conflict resolution for a vacuum cleanup.
RecoveryConflictTablespace | Waiting for recovery conflict resolution for dropping a tablespace.
RecoveryPause | Waiting for recovery to be resumed.
ReplicationOriginDrop | Waiting for a replication origin to become inactive so it can be dropped.
ReplicationSlotDrop | Waiting for a replication slot to become inactive so it can be dropped.
SafeSnapshot | Waiting to obtain a valid snapshot for a READ ONLY DEFERRABLE transaction.
SyncRep | Waiting for confirmation from a remote server during synchronous replication.
WalReceiverExit | Waiting for the WAL receiver to exit.
WalReceiverWaitStart | Waiting for startup process to send initial data for streaming replication.
XactGroupUpdate | Waiting for the group leader to update transaction status at end of a parallel operation.

Table 28.11. Wait Events of Type Lock

Lock Wait Event | Description
advisory | Waiting to acquire an advisory user lock.
extend | Waiting to extend a relation.
frozenid | Waiting to update pg_database.datfrozenxid and pg_database.datminmxid.
object | Waiting to acquire a lock on a non-relation database object.
page | Waiting to acquire a lock on a page of a relation.
relation | Waiting to acquire a lock on a relation.
spectoken | Waiting to acquire a speculative insertion lock.
transactionid | Waiting for a transaction to finish.
tuple | Waiting to acquire a lock on a tuple.
userlock | Waiting to acquire a user lock.
virtualxid | Waiting to acquire a virtual transaction ID lock.

Table 28.12. Wait Events of Type LWLock

LWLock Wait Event | Description
AddinShmemInit | Waiting to manage an extension's space allocation in shared memory.
AutoFile | Waiting to update the postgresql.auto.conf file.
Autovacuum | Waiting to read or update the current state of autovacuum workers.
AutovacuumSchedule | Waiting to ensure that a table selected for autovacuum still needs vacuuming.
BackgroundWorker | Waiting to read or update background worker state.
BtreeVacuum | Waiting to read or update vacuum-related information for a B-tree index.
BufferContent | Waiting to access a data page in memory.
BufferMapping | Waiting to associate a data block with a buffer in the buffer pool.
CheckpointerComm | Waiting to manage fsync requests.
CommitTs | Waiting to read or update the last value set for a transaction commit timestamp.
CommitTsBuffer | Waiting for I/O on a commit timestamp SLRU buffer.
CommitTsSLRU | Waiting to access the commit timestamp SLRU cache.
ControlFile | Waiting to read or update the pg_control file or create a new WAL file.
DynamicSharedMemoryControl | Waiting to read or update dynamic shared memory allocation information.
LockFastPath | Waiting to read or update a process' fast-path lock information.
LockManager | Waiting to read or update information about “heavyweight” locks.
LogicalRepWorker | Waiting to read or update the state of logical replication workers.
MultiXactGen | Waiting to read or update shared multixact state.
MultiXactMemberBuffer | Waiting for I/O on a multixact member SLRU buffer.
MultiXactMemberSLRU | Waiting to access the multixact member SLRU cache.
MultiXactOffsetBuffer | Waiting for I/O on a multixact offset SLRU buffer.
MultiXactOffsetSLRU | Waiting to access the multixact offset SLRU cache.
MultiXactTruncation | Waiting to read or truncate multixact information.
NotifyBuffer | Waiting for I/O on a NOTIFY message SLRU buffer.
NotifyQueue | Waiting to read or update NOTIFY messages.
NotifyQueueTail | Waiting to update limit on NOTIFY message storage.
NotifySLRU | Waiting to access the NOTIFY message SLRU cache.
OidGen | Waiting to allocate a new OID.
OldSnapshotTimeMap | Waiting to read or update old snapshot control information.
ParallelAppend | Waiting to choose the next subplan during Parallel Append plan execution.
ParallelHashJoin | Waiting to synchronize workers during Parallel Hash Join plan execution.
ParallelQueryDSA | Waiting for parallel query dynamic shared memory allocation.
PerSessionDSA | Waiting for parallel query dynamic shared memory allocation.
PerSessionRecordType | Waiting to access a parallel query's information about composite types.
PerSessionRecordTypmod | Waiting to access a parallel query's information about type modifiers that identify anonymous record types.
PerXactPredicateList | Waiting to access the list of predicate locks held by the current serializable transaction during a parallel query.
PredicateLockManager | Waiting to access predicate lock information used by serializable transactions.
ProcArray | Waiting to access the shared per-process data structures (typically, to get a snapshot or report a session's transaction ID).
RelationMapping | Waiting to read or update a pg_filenode.map file (used to track the filenode assignments of certain system catalogs).
RelCacheInit | Waiting to read or update a pg_internal.init relation cache initialization file.
ReplicationOrigin | Waiting to create, drop or use a replication origin.
ReplicationOriginState | Waiting to read or update the progress of one replication origin.
ReplicationSlotAllocation | Waiting to allocate or free a replication slot.
ReplicationSlotControl | Waiting to read or update replication slot state.
ReplicationSlotIO | Waiting for I/O on a replication slot.
SerialBuffer | Waiting for I/O on a serializable transaction conflict SLRU buffer.
SerializableFinishedList | Waiting to access the list of finished serializable transactions.
SerializablePredicateList | Waiting to access the list of predicate locks held by serializable transactions.
SerializableXactHash | Waiting to read or update information about serializable transactions.
SerialSLRU | Waiting to access the serializable transaction conflict SLRU cache.
SharedTidBitmap | Waiting to access a shared TID bitmap during a parallel bitmap index scan.
SharedTupleStore | Waiting to access a shared tuple store during parallel query.
ShmemIndex | Waiting to find or allocate space in shared memory.
SInvalRead | Waiting to retrieve messages from the shared catalog invalidation queue.
SInvalWrite | Waiting to add a message to the shared catalog invalidation queue.
SubtransBuffer | Waiting for I/O on a sub-transaction SLRU buffer.
SubtransSLRU | Waiting to access the sub-transaction SLRU cache.
SyncRep | Waiting to read or update information about the state of synchronous replication.
SyncScan | Waiting to select the starting location of a synchronized table scan.
TablespaceCreate | Waiting to create or drop a tablespace.
TwoPhaseState | Waiting to read or update the state of prepared transactions.
WALBufMapping | Waiting to replace a page in WAL buffers.
WALInsert | Waiting to insert WAL data into a memory buffer.
WALWrite | Waiting for WAL buffers to be written to disk.
WrapLimitsVacuum | Waiting to update limits on transaction id and multixact consumption.
XactBuffer | Waiting for I/O on a transaction status SLRU buffer.
XactSLRU | Waiting to access the transaction status SLRU cache.
XactTruncation | Waiting to execute pg_xact_status or update the oldest transaction ID available to it.
XidGen | Waiting to allocate a new transaction ID.

Note

Extensions can add LWLock types to the list shown in Table 28.12. In some cases, the name assigned by an extension will not be available in all server processes; so an LWLock wait event might be reported as just “extension” rather than the extension-assigned name.

Table 28.13. Wait Events of Type Timeout

Timeout Wait Event | Description
BaseBackupThrottle | Waiting during base backup when throttling activity.
CheckpointWriteDelay | Waiting between writes while performing a checkpoint.
PgSleep | Waiting due to a call to pg_sleep or a sibling function.
RecoveryApplyDelay | Waiting to apply WAL during recovery because of a delay setting.
RecoveryRetrieveRetryInterval | Waiting during recovery when WAL data is not available from any source (pg_wal, archive or stream).
RegisterSyncRequest | Waiting while sending synchronization requests to the checkpointer, because the request queue is full.
VacuumDelay | Waiting in a cost-based vacuum delay point.

Here is an example of how wait events can be viewed:

SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event is NOT NULL;
 pid  | wait_event_type | wait_event
------+-----------------+------------
 2540 | Lock            | relation
 6644 | LWLock          | ProcArray
(2 rows)

28.2.4. pg_stat_replication

The pg_stat_replication view will contain one row per WAL sender process, showing statistics about replication to that sender's connected standby server. Only directly connected standbys are listed; no information is available about downstream standby servers.

Table 28.14. pg_stat_replication View

Column Type

Description

pid integer

Process ID of a WAL sender process

usesysid oid

OID of the user logged into this WAL sender process

usename name

Name of the user logged into this WAL sender process

application_name text

Name of the application that is connected to this WAL sender

client_addr inet

IP address of the client connected to this WAL sender. If this field is null, it indicates that the client is connected via a Unix socket on the server machine.

client_hostname text

Host name of the connected client, as reported by a reverse DNS lookup of client_addr. This field will only be non-null for IP connections, and only when log_hostname is enabled.

client_port integer

TCP port number that the client is using for communication with this WAL sender, or -1 if a Unix socket is used

backend_start timestamp with time zone

Time when this process was started, i.e., when the client connected to this WAL sender

backend_xmin xid

This standby's xmin horizon reported by hot_standby_feedback.

state text

Current WAL sender state. Possible values are:

  • startup: This WAL sender is starting up.

  • catchup: This WAL sender's connected standby is catching up with the primary.

  • streaming: This WAL sender is streaming changes after its connected standby server has caught up with the primary.

  • backup: This WAL sender is sending a backup.

  • stopping: This WAL sender is stopping.

sent_lsn pg_lsn

Last write-ahead log location sent on this connection

write_lsn pg_lsn

Last write-ahead log location written to disk by this standby server

flush_lsn pg_lsn

Last write-ahead log location flushed to disk by this standby server

replay_lsn pg_lsn

Last write-ahead log location replayed into the database on this standby server

write_lag interval

Time elapsed between flushing recent WAL locally and receiving notification that this standby server has written it (but not yet flushed it or applied it). This can be used to gauge the delay that synchronous_commit level remote_write incurred while committing if this server was configured as a synchronous standby.

flush_lag interval

Time elapsed between flushing recent WAL locally and receiving notification that this standby server has written and flushed it (but not yet applied it). This can be used to gauge the delay that synchronous_commit level on incurred while committing if this server was configured as a synchronous standby.

replay_lag interval

Time elapsed between flushing recent WAL locally and receiving notification that this standby server has written, flushed and applied it. This can be used to gauge the delay that synchronous_commit level remote_apply incurred while committing if this server was configured as a synchronous standby.

sync_priority integer

Priority of this standby server for being chosen as the synchronous standby in a priority-based synchronous replication. This has no effect in a quorum-based synchronous replication.

sync_state text

Synchronous state of this standby server. Possible values are:

  • async: This standby server is asynchronous.

  • potential: This standby server is now asynchronous, but can potentially become synchronous if one of current synchronous ones fails.

  • sync: This standby server is synchronous.

  • quorum: This standby server is considered as a candidate for quorum standbys.

reply_time timestamp with time zone

Send time of last reply message received from standby server

The lag times reported in the pg_stat_replication view are measurements of the time taken for recent WAL to be written, flushed and replayed and for the sender to know about it. These times represent the commit delay that was (or would have been) introduced by each synchronous commit level, if the remote server was configured as a synchronous standby. For an asynchronous standby, the replay_lag column approximates the delay before recent transactions became visible to queries. If the standby server has entirely caught up with the sending server and there is no more WAL activity, the most recently measured lag times will continue to be displayed for a short time and then show NULL.

Lag times work automatically for physical replication. Logical decoding plugins may optionally emit tracking messages; if they do not, the tracking mechanism will simply display NULL lag.

Note

The reported lag times are not predictions of how long it will take for the standby to catch up with the sending server assuming the current rate of replay. Such a system would show similar times while new WAL is being generated, but would differ when the sender becomes idle. In particular, when the standby has caught up completely, pg_stat_replication shows the time taken to write, flush and replay the most recent reported WAL location rather than zero as some users might expect. This is consistent with the goal of measuring synchronous commit and transaction visibility delays for recent write transactions. To reduce confusion for users expecting a different model of lag, the lag columns revert to NULL after a short time on a fully replayed idle system. Monitoring systems should choose whether to represent this as missing data, zero or continue to display the last known value.

28.2.5. pg_stat_replication_slots

The pg_stat_replication_slots view will contain one row per logical replication slot, showing statistics about its usage.

Table 28.15. pg_stat_replication_slots View

Column Type

Description

slot_name text

A unique, cluster-wide identifier for the replication slot

spill_txns bigint

Number of transactions spilled to disk once the memory used by logical decoding to decode changes from WAL has exceeded logical_decoding_work_mem. The counter gets incremented for both top-level transactions and subtransactions.

spill_count bigint

Number of times transactions were spilled to disk while decoding changes from WAL for this slot. This counter is incremented each time a transaction is spilled, and the same transaction may be spilled multiple times.

spill_bytes bigint

Amount of decoded transaction data spilled to disk while performing decoding of changes from WAL for this slot. This and other spill counters can be used to gauge the I/O which occurred during logical decoding and allow tuning logical_decoding_work_mem.

stream_txns bigint

Number of in-progress transactions streamed to the decoding output plugin after the memory used by logical decoding to decode changes from WAL for this slot has exceeded logical_decoding_work_mem. Streaming only works with top-level transactions (subtransactions can't be streamed independently), so the counter is not incremented for subtransactions.

stream_countbigint

Number of times in-progress transactions were streamed to the decoding output plugin while decoding changes from WAL for this slot. This counter is incremented each time a transaction is streamed, and the same transaction may be streamed multiple times.

stream_bytesbigint

Amount of transaction data decoded for streaming in-progress transactions to the decoding output plugin while decoding changes from WAL for this slot. This and other streaming counters for this slot can be used to tune logical_decoding_work_mem.

total_txns bigint

Number of decoded transactions sent to the decoding output plugin for this slot. This counts top-level transactions only, and is not incremented for subtransactions. Note that this includes the transactions that are streamed and/or spilled.

total_bytesbigint

Amount of transaction data decoded for sending transactions to the decoding output plugin while decoding changes from WAL for this slot. Note that this includes data that is streamed and/or spilled.

stats_reset timestamp with time zone

Time at which these statistics were last reset

28.2.6. pg_stat_wal_receiver

The pg_stat_wal_receiver view will contain only one row, showing statistics about the WAL receiver from that receiver's connected server.

Table 28.16. pg_stat_wal_receiver View

Column Type

Description

pid integer

Process ID of the WAL receiver process

status text

Activity status of the WAL receiver process

receive_start_lsn pg_lsn

First write-ahead log location used when WAL receiver is started

receive_start_tli integer

First timeline number used when WAL receiver is started

written_lsn pg_lsn

Last write-ahead log location already received and written to disk, but not flushed. This should not be used for data integrity checks.

flushed_lsn pg_lsn

Last write-ahead log location already received and flushed to disk, the initial value of this field being the first log location used when WAL receiver is started

received_tli integer

Timeline number of last write-ahead log location received and flushed to disk, the initial value of this field being the timeline number of the first log location used when WAL receiver is started

last_msg_send_time timestamp with time zone

Send time of last message received from origin WAL sender

last_msg_receipt_time timestamp with time zone

Receipt time of last message received from origin WAL sender

latest_end_lsn pg_lsn

Last write-ahead log location reported to origin WAL sender

latest_end_time timestamp with time zone

Time of last write-ahead log location reported to origin WAL sender

slot_name text

Replication slot name used by this WAL receiver

sender_host text

Host of the PostgreSQL instance this WAL receiver is connected to. This can be a host name, an IP address, or a directory path if the connection is via Unix socket. (The path case can be distinguished because it will always be an absolute path, beginning with /.)

sender_port integer

Port number of the PostgreSQL instance this WAL receiver is connected to.

conninfo text

Connection string used by this WAL receiver, with security-sensitive fields obfuscated.

28.2.7. pg_stat_subscription

The pg_stat_subscription view will contain one row per subscription for main worker (with null PID if the worker is not running), and additional rows for workers handling the initial data copy of the subscribed tables.

Table 28.17. pg_stat_subscription View

Column Type

Description

subid oid

OID of the subscription

subname name

Name of the subscription

pid integer

Process ID of the subscription worker process

relid oid

OID of the relation that the worker is synchronizing; null for the main apply worker

received_lsn pg_lsn

Last write-ahead log location received, the initial value of this field being 0

last_msg_send_time timestamp with time zone

Send time of last message received from origin WAL sender

last_msg_receipt_time timestamp with time zone

Receipt time of last message received from origin WAL sender

latest_end_lsn pg_lsn

Last write-ahead log location reported to origin WAL sender

latest_end_time timestamp with time zone

Time of last write-ahead log location reported to origin WAL sender

28.2.8. pg_stat_ssl

The pg_stat_ssl view will contain one row per backend or WAL sender process, showing statistics about SSL usage on this connection. It can be joined to pg_stat_activity or pg_stat_replication on the pid column to get more details about the connection.

Table 28.18. pg_stat_ssl View

Column Type

Description

pid integer

Process ID of a backend or WAL sender process

ssl boolean

True if SSL is used on this connection

version text

Version of SSL in use, or NULL if SSL is not in use on this connection

cipher text

Name of SSL cipher in use, or NULL if SSL is not in use on this connection

bits integer

Number of bits in the encryption algorithm used, or NULL if SSL is not used on this connection

client_dn text

Distinguished Name (DN) field from the client certificate used, or NULL if no client certificate was supplied or if SSL is not in use on this connection. This field is truncated if the DN field is longer than NAMEDATALEN (64 characters in a standard build).

client_serial numeric

Serial number of the client certificate, or NULL if no client certificate was supplied or if SSL is not in use on this connection. The combination of certificate serial number and certificate issuer uniquely identifies a certificate (unless the issuer erroneously reuses serial numbers).

issuer_dn text

DN of the issuer of the client certificate, or NULL if no client certificate was supplied or if SSL is not in use on this connection. This field is truncated like client_dn.

28.2.9. pg_stat_gssapi

The pg_stat_gssapi view will contain one row per backend, showing information about GSSAPI usage on this connection. It can be joined to pg_stat_activity or pg_stat_replication on the pid column to get more details about the connection.

Table 28.19. pg_stat_gssapi View

Column Type

Description

pid integer

Process ID of a backend

gss_authenticated boolean

True if GSSAPI authentication was used for this connection

principal text

Principal used to authenticate this connection, or NULL if GSSAPI was not used to authenticate this connection. This field is truncated if the principal is longer than NAMEDATALEN (64 characters in a standard build).

encrypted boolean

True if GSSAPI encryption is in use on this connection

28.2.10. pg_stat_archiver

The pg_stat_archiver view will always have a single row, containing data about the archiver process of the cluster.

Table 28.20. pg_stat_archiver View

Column Type

Description

archived_count bigint

Number of WAL files that have been successfully archived

last_archived_wal text

Name of the last WAL file successfully archived

last_archived_time timestamp with time zone

Time of the last successful archive operation

failed_count bigint

Number of failed attempts for archiving WAL files

last_failed_wal text

Name of the WAL file of the last failed archival operation

last_failed_time timestamp with time zone

Time of the last failed archival operation

stats_reset timestamp with time zone

Time at which these statistics were last reset

28.2.11. pg_stat_bgwriter

The pg_stat_bgwriter view will always have a single row, containing global data for the cluster.

Table 28.21. pg_stat_bgwriter View

Column Type

Description

checkpoints_timed bigint

Number of scheduled checkpoints that have been performed

checkpoints_req bigint

Number of requested checkpoints that have been performed

checkpoint_write_time double precision

Total amount of time that has been spent in the portion of checkpoint processing where files are written to disk, in milliseconds

checkpoint_sync_time double precision

Total amount of time that has been spent in the portion of checkpoint processing where files are synchronized to disk, in milliseconds

buffers_checkpoint bigint

Number of buffers written during checkpoints

buffers_clean bigint

Number of buffers written by the background writer

maxwritten_clean bigint

Number of times the background writer stopped a cleaning scan because it had written too many buffers

buffers_backend bigint

Number of buffers written directly by a backend

buffers_backend_fsync bigint

Number of times a backend had to execute its own fsync call (normally the background writer handles those even when the backend does its own write)

buffers_alloc bigint

Number of buffers allocated

stats_reset timestamp with time zone

Time at which these statistics were last reset

28.2.12. pg_stat_wal

The pg_stat_wal view will always have a single row, containing data about WAL activity of the cluster.

Table 28.22. pg_stat_wal View

Column Type

Description

wal_records bigint

Total number of WAL records generated

wal_fpi bigint

Total number of WAL full page images generated

wal_bytes numeric

Total amount of WAL generated in bytes

wal_buffers_full bigint

Number of times WAL data was written to disk because WAL buffers became full

wal_write bigint

Number of times WAL buffers were written out to disk via XLogWrite request. See Section 30.5 for more information about the internal WAL function XLogWrite.

wal_sync bigint

Number of times WAL files were synced to disk via issue_xlog_fsync request (if fsync is on and wal_sync_method is either fdatasync, fsync or fsync_writethrough, otherwise zero). See Section 30.5 for more information about the internal WAL function issue_xlog_fsync.

wal_write_time double precision

Total amount of time spent writing WAL buffers to disk via XLogWrite request, in milliseconds (if track_wal_io_timing is enabled, otherwise zero). This includes the sync time when wal_sync_method is either open_datasync or open_sync.

wal_sync_time double precision

Total amount of time spent syncing WAL files to disk via issue_xlog_fsync request, in milliseconds (if track_wal_io_timing is enabled, fsync is on, and wal_sync_method is either fdatasync, fsync or fsync_writethrough, otherwise zero).

stats_reset timestamp with time zone

Time at which these statistics were last reset

28.2.13. pg_stat_database

The pg_stat_database view will contain one row for each database in the cluster, plus one for shared objects, showing database-wide statistics.

Table 28.23. pg_stat_database View

Column Type

Description

datid oid

OID of this database, or 0 for objects belonging to a shared relation

datname name

Name of this database, or NULL for shared objects.

numbackends integer

Number of backends currently connected to this database, or NULL for shared objects. This is the only column in this view that returns a value reflecting current state; all other columns return the accumulated values since the last reset.

xact_commit bigint

Number of transactions in this database that have been committed

xact_rollback bigint

Number of transactions in this database that have been rolled back

blks_read bigint

Number of disk blocks read in this database

blks_hit bigint

Number of times disk blocks were found already in the buffer cache, so that a read was not necessary (this only includes hits in the PostgreSQL buffer cache, not the operating system's file system cache)

tup_returned bigint

Number of rows returned by queries in this database

tup_fetched bigint

Number of rows fetched by queries in this database

tup_inserted bigint

Number of rows inserted by queries in this database

tup_updated bigint

Number of rows updated by queries in this database

tup_deleted bigint

Number of rows deleted by queries in this database

conflicts bigint

Number of queries canceled due to conflicts with recovery in this database. (Conflicts occur only on standby servers; see pg_stat_database_conflicts for details.)

temp_files bigint

Number of temporary files created by queries in this database. All temporary files are counted, regardless of why the temporary file was created (e.g., sorting or hashing), and regardless of the log_temp_files setting.

temp_bytes bigint

Total amount of data written to temporary files by queries in this database. All temporary files are counted, regardless of why the temporary file was created, and regardless of the log_temp_files setting.

deadlocks bigint

Number of deadlocks detected in this database

checksum_failures bigint

Number of data page checksum failures detected in this database (or on a shared object), or NULL if data checksums are not enabled.

checksum_last_failure timestamp with time zone

Time at which the last data page checksum failure was detected in this database (or on a shared object), or NULL if data checksums are not enabled.

blk_read_time double precision

Time spent reading data file blocks by backends in this database, in milliseconds (if track_io_timing is enabled, otherwise zero)

blk_write_time double precision

Time spent writing data file blocks by backends in this database, in milliseconds (if track_io_timing is enabled, otherwise zero)

session_time double precision

Time spent by database sessions in this database, in milliseconds (note that statistics are only updated when the state of a session changes, so if sessions have been idle for a long time, this idle time won't be included)

active_time double precision

Time spent executing SQL statements in this database, in milliseconds (this corresponds to the states active and fastpath function call in pg_stat_activity)

idle_in_transaction_time double precision

Time spent idling while in a transaction in this database, in milliseconds (this corresponds to the states idle in transaction and idle in transaction (aborted) in pg_stat_activity)

sessions bigint

Total number of sessions established to this database

sessions_abandoned bigint

Number of database sessions to this database that were terminated because connection to the client was lost

sessions_fatal bigint

Number of database sessions to this database that were terminated by fatal errors

sessions_killed bigint

Number of database sessions to this database that were terminated by operator intervention

stats_reset timestamp with time zone

Time at which these statistics were last reset

28.2.14. pg_stat_database_conflicts

The pg_stat_database_conflicts view will contain one row per database, showing database-wide statistics about query cancels occurring due to conflicts with recovery on standby servers. This view will only contain information on standby servers, since conflicts do not occur on primary servers.

Table 28.24. pg_stat_database_conflicts View

Column Type

Description

datid oid

OID of a database

datname name

Name of this database

confl_tablespace bigint

Number of queries in this database that have been canceled due to dropped tablespaces

confl_lock bigint

Number of queries in this database that have been canceled due to lock timeouts

confl_snapshot bigint

Number of queries in this database that have been canceled due to old snapshots

confl_bufferpin bigint

Number of queries in this database that have been canceled due to pinned buffers

confl_deadlock bigint

Number of queries in this database that have been canceled due to deadlocks

28.2.15. pg_stat_all_tables

The pg_stat_all_tables view will contain one row for each table in the current database (including TOAST tables), showing statistics about accesses to that specific table. The pg_stat_user_tables and pg_stat_sys_tables views contain the same information, but filtered to only show user and system tables respectively.

Table 28.25. pg_stat_all_tables View

Column Type

Description

relid oid

OID of a table

schemaname name

Name of the schema that this table is in

relname name

Name of this table

seq_scan bigint

Number of sequential scans initiated on this table

seq_tup_read bigint

Number of live rows fetched by sequential scans

idx_scan bigint

Number of index scans initiated on this table

idx_tup_fetch bigint

Number of live rows fetched by index scans

n_tup_ins bigint

Number of rows inserted

n_tup_upd bigint

Number of rows updated (includes HOT updated rows)

n_tup_del bigint

Number of rows deleted

n_tup_hot_upd bigint

Number of rows HOT updated (i.e., with no separate index update required)

n_live_tup bigint

Estimated number of live rows

n_dead_tup bigint

Estimated number of dead rows

n_mod_since_analyze bigint

Estimated number of rows modified since this table was last analyzed

n_ins_since_vacuum bigint

Estimated number of rows inserted since this table was last vacuumed

last_vacuum timestamp with time zone

Last time at which this table was manually vacuumed (not counting VACUUM FULL)

last_autovacuum timestamp with time zone

Last time at which this table was vacuumed by the autovacuum daemon

last_analyze timestamp with time zone

Last time at which this table was manually analyzed

last_autoanalyze timestamp with time zone

Last time at which this table was analyzed by the autovacuum daemon

vacuum_count bigint

Number of times this table has been manually vacuumed (not counting VACUUM FULL)

autovacuum_count bigint

Number of times this table has been vacuumed by the autovacuum daemon

analyze_count bigint

Number of times this table has been manually analyzed

autoanalyze_count bigint

Number of times this table has been analyzed by the autovacuum daemon

28.2.16. pg_stat_all_indexes

The pg_stat_all_indexes view will contain one row for each index in the current database, showing statistics about accesses to that specific index. The pg_stat_user_indexes and pg_stat_sys_indexes views contain the same information, but filtered to only show user and system indexes respectively.

Table 28.26. pg_stat_all_indexes View

Column Type

Description

relid oid

OID of the table for this index

indexrelid oid

OID of this index

schemaname name

Name of the schema this index is in

relname name

Name of the table for this index

indexrelname name

Name of this index

idx_scan bigint

Number of index scans initiated on this index

idx_tup_read bigint

Number of index entries returned by scans on this index

idx_tup_fetch bigint

Number of live table rows fetched by simple index scans using this index

Indexes can be used by simple index scans, “bitmap” index scans, and the optimizer. In a bitmap scan the output of several indexes can be combined via AND or OR rules, so it is difficult to associate individual heap row fetches with specific indexes when a bitmap scan is used. Therefore, a bitmap scan increments the pg_stat_all_indexes.idx_tup_read count(s) for the index(es) it uses, and it increments the pg_stat_all_tables.idx_tup_fetch count for the table, but it does not affect pg_stat_all_indexes.idx_tup_fetch. The optimizer also accesses indexes to check for supplied constants whose values are outside the recorded range of the optimizer statistics because the optimizer statistics might be stale.

Note

The idx_tup_read and idx_tup_fetch counts can be different even without any use of bitmap scans, because idx_tup_read counts index entries retrieved from the index while idx_tup_fetch counts live rows fetched from the table. The latter will be less if any dead or not-yet-committed rows are fetched using the index, or if any heap fetches are avoided by means of an index-only scan.

28.2.17. pg_statio_all_tables

The pg_statio_all_tables view will contain one row for each table in the current database (including TOAST tables), showing statistics about I/O on that specific table. The pg_statio_user_tables and pg_statio_sys_tables views contain the same information, but filtered to only show user and system tables respectively.

Table 28.27. pg_statio_all_tables View

Column Type

Description

relid oid

OID of a table

schemaname name

Name of the schema that this table is in

relname name

Name of this table

heap_blks_read bigint

Number of disk blocks read from this table

heap_blks_hit bigint

Number of buffer hits in this table

idx_blks_read bigint

Number of disk blocks read from all indexes on this table

idx_blks_hit bigint

Number of buffer hits in all indexes on this table

toast_blks_read bigint

Number of disk blocks read from this table's TOAST table (if any)

toast_blks_hit bigint

Number of buffer hits in this table's TOAST table (if any)

tidx_blks_read bigint

Number of disk blocks read from this table's TOAST table indexes (if any)

tidx_blks_hit bigint

Number of buffer hits in this table's TOAST table indexes (if any)

28.2.18. pg_statio_all_indexes

The pg_statio_all_indexes view will contain one row for each index in the current database, showing statistics about I/O on that specific index. The pg_statio_user_indexes and pg_statio_sys_indexes views contain the same information, but filtered to only show user and system indexes respectively.

Table 28.28. pg_statio_all_indexes View

Column Type

Description

relid oid

OID of the table for this index

indexrelid oid

OID of this index

schemaname name

Name of the schema this index is in

relname name

Name of the table for this index

indexrelname name

Name of this index

idx_blks_read bigint

Number of disk blocks read from this index

idx_blks_hit bigint

Number of buffer hits in this index

28.2.19. pg_statio_all_sequences

The pg_statio_all_sequences view will contain one row for each sequence in the current database, showing statistics about I/O on that specific sequence.

Table 28.29. pg_statio_all_sequences View

Column Type

Description

relid oid

OID of a sequence

schemaname name

Name of the schema this sequence is in

relname name

Name of this sequence

blks_read bigint

Number of disk blocks read from this sequence

blks_hit bigint

Number of buffer hits in this sequence

28.2.20. pg_stat_user_functions

The pg_stat_user_functions view will contain one row for each tracked function, showing statistics about executions of that function. The track_functions parameter controls exactly which functions are tracked.

Table 28.30. pg_stat_user_functions View

Column Type

Description

funcid oid

OID of a function

schemaname name

Name of the schema this function is in

funcname name

Name of this function

calls bigint

Number of times this function has been called

total_time double precision

Total time spent in this function and all other functions called by it, in milliseconds

self_time double precision

Total time spent in this function itself, not including other functions called by it, in milliseconds

28.2.21. pg_stat_slru

PostgreSQL accesses certain on-disk information via SLRU (simple least-recently-used) caches. The pg_stat_slru view will contain one row for each tracked SLRU cache, showing statistics about access to cached pages.

Table 28.31. pg_stat_slru View

Column Type

Description

name text

Name of the SLRU

blks_zeroed bigint

Number of blocks zeroed during initializations

blks_hit bigint

Number of times disk blocks were found already in the SLRU, so that a read was not necessary (this only includes hits in the SLRU, not the operating system's file system cache)

blks_read bigint

Number of disk blocks read for this SLRU

blks_written bigint

Number of disk blocks written for this SLRU

blks_exists bigint

Number of blocks checked for existence for this SLRU

flushes bigint

Number of flushes of dirty data for this SLRU

truncates bigint

Number of truncates for this SLRU

stats_reset timestamp with time zone

Time at which these statistics were last reset

28.2.22. Statistics Functions

Other ways of looking at the statistics can be set up by writing queries that use the same underlying statistics access functions used by the standard views shown above. For details such as the functions' names, consult the definitions of the standard views. (For example, in psql you could issue \d+ pg_stat_activity.) The access functions for per-database statistics take a database OID as an argument to identify which database to report on. The per-table and per-index functions take a table or index OID. The functions for per-function statistics take a function OID. Note that only tables, indexes, and functions in the current database can be seen with these functions.

Additional functions related to statistics collection are listed in Table 28.32.

Table 28.32. Additional Statistics Functions

Function

Description

pg_backend_pid () → integer

Returns the process ID of the server process attached to the current session.

pg_stat_get_activity ( integer ) → setof record

Returns a record of information about the backend with the specified process ID, or one record for each active backend in the system if NULL is specified. The fields returned are a subset of those in the pg_stat_activity view.

pg_stat_get_snapshot_timestamp () → timestamp with time zone

Returns the timestamp of the current statistics snapshot.

pg_stat_clear_snapshot () → void

Discards the current statistics snapshot.

pg_stat_reset () → void

Resets all statistics counters for the current database to zero.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

pg_stat_reset_shared ( text ) → void

Resets some cluster-wide statistics counters to zero, depending on the argument. The argument can be bgwriter to reset all the counters shown in the pg_stat_bgwriter view, archiver to reset all the counters shown in the pg_stat_archiver view or wal to reset all the counters shown in the pg_stat_wal view.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

pg_stat_reset_single_table_counters ( oid ) → void

Resets statistics for a single table or index in the current database to zero.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

pg_stat_reset_single_function_counters ( oid ) → void

Resets statistics for a single function in the current database to zero.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

pg_stat_reset_slru ( text ) → void

Resets statistics to zero for a single SLRU cache, or for all SLRUs in the cluster. If the argument is NULL, all counters shown in the pg_stat_slru view for all SLRU caches are reset. The argument can be one of CommitTs, MultiXactMember, MultiXactOffset, Notify, Serial, Subtrans, or Xact to reset the counters for only that entry. If the argument is other (or indeed, any unrecognized name), then the counters for all other SLRU caches, such as extension-defined caches, are reset.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

pg_stat_reset_replication_slot ( text ) → void

Resets statistics of the replication slot defined by the argument. If the argument is NULL, resets statistics for all the replication slots.

This function is restricted to superusers by default, but other users can be granted EXECUTE to run the function.

pg_stat_get_activity, the underlying function of the pg_stat_activity view, returns a set of records containing all the available information about each backend process. Sometimes it may be more convenient to obtain just a subset of this information. In such cases, an older set of per-backend statistics access functions can be used; these are shown in Table 28.33. These access functions use a backend ID number, which ranges from one to the number of currently active backends. The function pg_stat_get_backend_idset provides a convenient way to generate one row for each active backend for invoking these functions. For example, to show the PIDs and current queries of all backends:

SELECT pg_stat_get_backend_pid(s.backendid) AS pid,
       pg_stat_get_backend_activity(s.backendid) AS query
    FROM (SELECT pg_stat_get_backend_idset() AS backendid) AS s;

Table 28.33. Per-Backend Statistics Functions

Function

Description

pg_stat_get_backend_idset () → setof integer

Returns the set of currently active backend ID numbers (from 1 to the number of active backends).

pg_stat_get_backend_activity ( integer ) → text

Returns the text of this backend's most recent query.

pg_stat_get_backend_activity_start ( integer ) → timestamp with time zone

Returns the time when the backend's most recent query was started.

pg_stat_get_backend_client_addr ( integer ) → inet

Returns the IP address of the client connected to this backend.

pg_stat_get_backend_client_port ( integer ) → integer

Returns the TCP port number that the client is using for communication.

pg_stat_get_backend_dbid ( integer ) → oid

Returns the OID of the database this backend is connected to.

pg_stat_get_backend_pid ( integer ) → integer

Returns the process ID of this backend.

pg_stat_get_backend_start ( integer ) → timestamp with time zone

Returns the time when this process was started.

pg_stat_get_backend_userid ( integer ) → oid

Returns the OID of the user logged into this backend.

pg_stat_get_backend_wait_event_type ( integer ) → text

Returns the wait event type name if this backend is currently waiting, otherwise NULL. See Table 28.4 for details.

pg_stat_get_backend_wait_event ( integer ) → text

Returns the wait event name if this backend is currently waiting, otherwise NULL. See Table 28.5 through Table 28.13.

pg_stat_get_backend_xact_start ( integer ) → timestamp with time zone

Returns the time when the backend's current transaction was started.

Prev | Up | Next
28.1. Standard Unix Tools | Home | 28.3. Viewing Locks


28.3. Viewing Locks
Prev Up Chapter 28. Monitoring Database Activity Home Next

28.3. Viewing Locks

Another useful tool for monitoring database activity is the pg_locks system table. It allows the database administrator to view information about the outstanding locks in the lock manager. For example, this capability can be used to:

  • View all the locks currently outstanding, all the locks on relations in a particular database, all the locks on a particular relation, or all the locks held by a particular PostgreSQL session.

  • Determine the relation in the current database with the most ungranted locks (which might be a source of contention among database clients).

  • Determine the effect of lock contention on overall database performance, as well as the extent to which contention varies with overall database traffic.

Details of the pg_locks view appear in Section 52.74. For more information on locking and managing concurrency with PostgreSQL, refer to Chapter 13.

Prev | Up | Next
28.2. The Statistics Collector | Home | 28.4. Progress Reporting


28.4. Progress Reporting
Prev Up Chapter 28. Monitoring Database Activity Home Next

28.4. Progress Reporting

28.4.1. ANALYZE Progress Reporting
28.4.2. CREATE INDEX Progress Reporting
28.4.3. VACUUM Progress Reporting
28.4.4. CLUSTER Progress Reporting
28.4.5. Base Backup Progress Reporting
28.4.6. COPY Progress Reporting

PostgreSQL has the ability to report the progress of certain commands during command execution. Currently, the only commands which support progress reporting are ANALYZE, CLUSTER, CREATE INDEX, VACUUM, COPY, and BASE_BACKUP (i.e., replication command that pg_basebackup issues to take a base backup). This may be expanded in the future.

28.4.1. ANALYZE Progress Reporting

Whenever ANALYZE is running, the pg_stat_progress_analyze view will contain a row for each backend that is currently running that command. The tables below describe the information that will be reported and provide information about how to interpret it.

Table 28.34. pg_stat_progress_analyze View

Column Type

Description

pid integer

Process ID of backend.

datid oid

OID of the database to which this backend is connected.

datname name

Name of the database to which this backend is connected.

relid oid

OID of the table being analyzed.

phase text

Current processing phase. See Table 28.35.

sample_blks_total bigint

Total number of heap blocks that will be sampled.

sample_blks_scanned bigint

Number of heap blocks scanned.

ext_stats_total bigint

Number of extended statistics.

ext_stats_computed bigint

Number of extended statistics computed. This counter only advances when the phase is computing extended statistics.

child_tables_total bigint

Number of child tables.

child_tables_done bigint

Number of child tables scanned. This counter only advances when the phase is acquiring inherited sample rows.

current_child_table_relid oid

OID of the child table currently being scanned. This field is only valid when the phase is acquiring inherited sample rows.

Table 28.35. ANALYZE phases

Phase | Description
initializing | The command is preparing to begin scanning the heap. This phase is expected to be very brief.
acquiring sample rows | The command is currently scanning the table given by relid to obtain sample rows.
acquiring inherited sample rows | The command is currently scanning child tables to obtain sample rows. Columns child_tables_total, child_tables_done, and current_child_table_relid contain the progress information for this phase.
computing statistics | The command is computing statistics from the sample rows obtained during the table scan.
computing extended statistics | The command is computing extended statistics from the sample rows obtained during the table scan.
finalizing analyze | The command is updating pg_class. When this phase is completed, ANALYZE will end.

Note

Note that when ANALYZE is run on a partitioned table, all of its partitions are also recursively analyzed. In that case, ANALYZE progress is reported first for the parent table, whereby its inheritance statistics are collected, followed by that for each partition.

28.4.2. CREATE INDEX Progress Reporting

Whenever CREATE INDEX or REINDEX is running, the pg_stat_progress_create_index view will contain one row for each backend that is currently creating indexes. The tables below describe the information that will be reported and provide information about how to interpret it.

Table 28.36. pg_stat_progress_create_index View

Column Type

Description

pid integer

Process ID of backend.

datid oid

OID of the database to which this backend is connected.

datname name

Name of the database to which this backend is connected.

relid oid

OID of the table on which the index is being created.

index_relid oid

OID of the index being created or reindexed. During a non-concurrent CREATE INDEX, this is 0.

command text

The command that is running: CREATE INDEX, CREATE INDEX CONCURRENTLY, REINDEX, or REINDEX CONCURRENTLY.

phase text

Current processing phase of index creation. See Table 28.37.

lockers_total bigint

Total number of lockers to wait for, when applicable.

lockers_done bigint

Number of lockers already waited for.

current_locker_pid bigint

Process ID of the locker currently being waited for.

blocks_total bigint

Total number of blocks to be processed in the current phase.

blocks_done bigint

Number of blocks already processed in the current phase.

tuples_total bigint

Total number of tuples to be processed in the current phase.

tuples_done bigint

Number of tuples already processed in the current phase.

partitions_total bigint

When creating an index on a partitioned table, this column is set to the total number of partitions on which the index is to be created. This field is 0 during a REINDEX.

partitions_done bigint

When creating an index on a partitioned table, this column is set to the number of partitions on which the index has been created. This field is 0 during a REINDEX.

Table 28.37. CREATE INDEX Phases

Phase | Description
initializing | CREATE INDEX or REINDEX is preparing to create the index. This phase is expected to be very brief.
waiting for writers before build | CREATE INDEX CONCURRENTLY or REINDEX CONCURRENTLY is waiting for transactions with write locks that can potentially see the table to finish. This phase is skipped when not in concurrent mode. Columns lockers_total, lockers_done and current_locker_pid contain the progress information for this phase.
building index | The index is being built by the access method-specific code. In this phase, access methods that support progress reporting fill in their own progress data, and the subphase is indicated in this column. Typically, blocks_total and blocks_done will contain progress data, as well as potentially tuples_total and tuples_done.
waiting for writers before validation | CREATE INDEX CONCURRENTLY or REINDEX CONCURRENTLY is waiting for transactions with write locks that can potentially write into the table to finish. This phase is skipped when not in concurrent mode. Columns lockers_total, lockers_done and current_locker_pid contain the progress information for this phase.
index validation: scanning index | CREATE INDEX CONCURRENTLY is scanning the index searching for tuples that need to be validated. This phase is skipped when not in concurrent mode. Columns blocks_total (set to the total size of the index) and blocks_done contain the progress information for this phase.
index validation: sorting tuples | CREATE INDEX CONCURRENTLY is sorting the output of the index scanning phase.
index validation: scanning table | CREATE INDEX CONCURRENTLY is scanning the table to validate the index tuples collected in the previous two phases. This phase is skipped when not in concurrent mode. Columns blocks_total (set to the total size of the table) and blocks_done contain the progress information for this phase.
waiting for old snapshots | CREATE INDEX CONCURRENTLY or REINDEX CONCURRENTLY is waiting for transactions that can potentially see the table to release their snapshots. This phase is skipped when not in concurrent mode. Columns lockers_total, lockers_done and current_locker_pid contain the progress information for this phase.
waiting for readers before marking dead | REINDEX CONCURRENTLY is waiting for transactions with read locks on the table to finish, before marking the old index dead. This phase is skipped when not in concurrent mode. Columns lockers_total, lockers_done and current_locker_pid contain the progress information for this phase.
waiting for readers before dropping | REINDEX CONCURRENTLY is waiting for transactions with read locks on the table to finish, before dropping the old index. This phase is skipped when not in concurrent mode. Columns lockers_total, lockers_done and current_locker_pid contain the progress information for this phase.

28.4.3. VACUUM Progress Reporting

Whenever VACUUM is running, the pg_stat_progress_vacuum view will contain one row for each backend (including autovacuum worker processes) that is currently vacuuming. The tables below describe the information that will be reported and provide information about how to interpret it. Progress for VACUUM FULL commands is reported via pg_stat_progress_cluster because both VACUUM FULL and CLUSTER rewrite the table, while regular VACUUM only modifies it in place. See Section 28.4.4.

Table 28.38. pg_stat_progress_vacuum View

Column Type

Description

pid integer

Process ID of backend.

datid oid

OID of the database to which this backend is connected.

datname name

Name of the database to which this backend is connected.

relid oid

OID of the table being vacuumed.

phase text

Current processing phase of vacuum. See Table 28.39.

heap_blks_total bigint

Total number of heap blocks in the table. This number is reported as of the beginning of the scan; blocks added later will not be (and need not be) visited by this VACUUM.

heap_blks_scanned bigint

Number of heap blocks scanned. Because the visibility map is used to optimize scans, some blocks will be skipped without inspection; skipped blocks are included in this total, so that this number will eventually become equal to heap_blks_total when the vacuum is complete. This counter only advances when the phase is scanning heap.

heap_blks_vacuumed bigint

Number of heap blocks vacuumed. Unless the table has no indexes, this counter only advances when the phase is vacuuming heap. Blocks that contain no dead tuples are skipped, so the counter may sometimes skip forward in large increments.

index_vacuum_count bigint

Number of completed index vacuum cycles.

max_dead_tuples bigint

Number of dead tuples that we can store before needing to perform an index vacuum cycle, based on maintenance_work_mem.

num_dead_tuples bigint

Number of dead tuples collected since the last index vacuum cycle.

Table 28.39. VACUUM Phases

Phase | Description
initializing | VACUUM is preparing to begin scanning the heap. This phase is expected to be very brief.
scanning heap | VACUUM is currently scanning the heap. It will prune and defragment each page if required, and possibly perform freezing activity. The heap_blks_scanned column can be used to monitor the progress of the scan.
vacuuming indexes | VACUUM is currently vacuuming the indexes. If a table has any indexes, this will happen at least once per vacuum, after the heap has been completely scanned. It may happen multiple times per vacuum if maintenance_work_mem (or, in the case of autovacuum, autovacuum_work_mem if set) is insufficient to store the number of dead tuples found.
vacuuming heap | VACUUM is currently vacuuming the heap. Vacuuming the heap is distinct from scanning the heap, and occurs after each instance of vacuuming indexes. If heap_blks_scanned is less than heap_blks_total, the system will return to scanning the heap after this phase is completed; otherwise, it will begin cleaning up indexes after this phase is completed.
cleaning up indexes | VACUUM is currently cleaning up indexes. This occurs after the heap has been completely scanned and all vacuuming of the indexes and the heap has been completed.
truncating heap | VACUUM is currently truncating the heap so as to return empty pages at the end of the relation to the operating system. This occurs after cleaning up indexes.
performing final cleanup | VACUUM is performing final cleanup. During this phase, VACUUM will vacuum the free space map, update statistics in pg_class, and report statistics to the statistics collector. When this phase is completed, VACUUM will end.

28.4.4. CLUSTER Progress Reporting

Whenever CLUSTER or VACUUM FULL is running, the pg_stat_progress_cluster view will contain a row for each backend that is currently running either command. The tables below describe the information that will be reported and provide information about how to interpret it.

Table 28.40. pg_stat_progress_cluster View

Column Type

Description

pid integer

Process ID of backend.

datid oid

OID of the database to which this backend is connected.

datname name

Name of the database to which this backend is connected.

relid oid

OID of the table being clustered.

command text

The command that is running. Either CLUSTER or VACUUM FULL.

phase text

Current processing phase. See Table 28.41.

cluster_index_relid oid

If the table is being scanned using an index, this is the OID of the index being used; otherwise, it is zero.

heap_tuples_scanned bigint

Number of heap tuples scanned. This counter only advances when the phase is seq scanning heap, index scanning heap or writing new heap.

heap_tuples_written bigint

Number of heap tuples written. This counter only advances when the phase is seq scanning heap, index scanning heap or writing new heap.

heap_blks_total bigint

Total number of heap blocks in the table. This number is reported as of the beginning of seq scanning heap.

heap_blks_scanned bigint

Number of heap blocks scanned. This counter only advances when the phase is seq scanning heap.

index_rebuild_count bigint

Number of indexes rebuilt. This counter only advances when the phase is rebuilding index.

Table 28.41. CLUSTER and VACUUM FULL Phases

Phase | Description
initializing | The command is preparing to begin scanning the heap. This phase is expected to be very brief.
seq scanning heap | The command is currently scanning the table using a sequential scan.
index scanning heap | CLUSTER is currently scanning the table using an index scan.
sorting tuples | CLUSTER is currently sorting tuples.
writing new heap | CLUSTER is currently writing the new heap.
swapping relation files | The command is currently swapping newly-built files into place.
rebuilding index | The command is currently rebuilding an index.
performing final cleanup | The command is performing final cleanup. When this phase is completed, CLUSTER or VACUUM FULL will end.

28.4.5. Base Backup Progress Reporting

Whenever an application like pg_basebackup is taking a base backup, the pg_stat_progress_basebackup view will contain a row for each WAL sender process that is currently running the BASE_BACKUP replication command and streaming the backup. The tables below describe the information that will be reported and provide information about how to interpret it.

Table 28.42. pg_stat_progress_basebackup View

Column Type

Description

pid integer

Process ID of a WAL sender process.

phase text

Current processing phase. See Table 28.43.

backup_total bigint

Total amount of data that will be streamed. This is estimated and reported as of the beginning of streaming database files phase. Note that this is only an approximation since the database may change during streaming database files phase and WAL log may be included in the backup later. This is always the same value as backup_streamed once the amount of data streamed exceeds the estimated total size. If the estimation is disabled in pg_basebackup (i.e., --no-estimate-size option is specified), this is NULL.

backup_streamed bigint

Amount of data streamed. This counter only advances when the phase is streaming database files or transferring wal files.

tablespaces_total bigint

Total number of tablespaces that will be streamed.

tablespaces_streamed bigint

Number of tablespaces streamed. This counter only advances when the phase is streaming database files.

Table 28.43. Base backup phases

Phase | Description
initializing | The WAL sender process is preparing to begin the backup. This phase is expected to be very brief.
waiting for checkpoint to finish | The WAL sender process is currently performing pg_start_backup to prepare to take a base backup, and waiting for the start-of-backup checkpoint to finish.
estimating backup size | The WAL sender process is currently estimating the total amount of database files that will be streamed as a base backup.
streaming database files | The WAL sender process is currently streaming database files as a base backup.
waiting for wal archiving to finish | The WAL sender process is currently performing pg_stop_backup to finish the backup, and waiting for all the WAL files required for the base backup to be successfully archived. If either --wal-method=none or --wal-method=stream is specified in pg_basebackup, the backup will end when this phase is completed.
transferring wal files | The WAL sender process is currently transferring all WAL logs generated during the backup. This phase occurs after waiting for wal archiving to finish phase if --wal-method=fetch is specified in pg_basebackup. The backup will end when this phase is completed.

28.4.6. COPY Progress Reporting

Whenever COPY is running, the pg_stat_progress_copy view will contain one row for each backend that is currently running a COPY command. The table below describes the information that will be reported and provides information about how to interpret it.

Table 28.44. pg_stat_progress_copy View

Column Type

Description

pid integer

Process ID of backend.

datid oid

OID of the database to which this backend is connected.

datname name

Name of the database to which this backend is connected.

relid oid

OID of the table on which the COPY command is executed. It is set to 0 if copying from a SELECT query.

command text

The command that is running: COPY FROM, or COPY TO.

type text

The io type that the data is read from or written to: FILE, PROGRAM, PIPE (for COPY FROM STDIN and COPY TO STDOUT), or CALLBACK (used for example during the initial table synchronization in logical replication).

bytes_processed bigint

Number of bytes already processed by COPY command.

bytes_total bigint

Size of source file for COPY FROM command in bytes. It is set to 0 if not available.

tuples_processed bigint

Number of tuples already processed by COPY command.

tuples_excluded bigint

Number of tuples not processed because they were excluded by the WHERE clause of the COPY command.

Prev | Up | Next
28.3. Viewing Locks | Home | 28.5. Dynamic Tracing


28.5. Dynamic Tracing
Prev Up Chapter 28. Monitoring Database Activity Home Next

28.5. Dynamic Tracing

28.5.1. Compiling for Dynamic Tracing
28.5.2. Built-in Probes
28.5.3. Using Probes
28.5.4. Defining New Probes

PostgreSQL provides facilities to support dynamic tracing of the database server. This allows an external utility to be called at specific points in the code and thereby trace execution.

A number of probes or trace points are already inserted into the source code. These probes are intended to be used by database developers and administrators. By default the probes are not compiled into PostgreSQL; the user needs to explicitly tell the configure script to make the probes available.

Currently, the DTrace utility is supported, which, at the time of this writing, is available on Solaris, macOS, FreeBSD, NetBSD, and Oracle Linux. The SystemTap project for Linux provides a DTrace equivalent and can also be used. Supporting other dynamic tracing utilities is theoretically possible by changing the definitions for the macros in src/include/utils/probes.h.

28.5.1. Compiling for Dynamic Tracing

By default, probes are not available, so you will need to explicitly tell the configure script to make the probes available in PostgreSQL. To include DTrace support specify --enable-dtrace to configure. See Section 17.4 for further information.

28.5.2. Built-in Probes

A number of standard probes are provided in the source code, as shown in Table 28.45; Table 28.46 shows the types used in the probes. More probes can certainly be added to enhance PostgreSQL's observability.

Table 28.45. Built-in DTrace Probes

Name | Parameters | Description
transaction-start | (LocalTransactionId) | Probe that fires at the start of a new transaction. arg0 is the transaction ID.
transaction-commit | (LocalTransactionId) | Probe that fires when a transaction completes successfully. arg0 is the transaction ID.
transaction-abort | (LocalTransactionId) | Probe that fires when a transaction completes unsuccessfully. arg0 is the transaction ID.
query-start | (const char *) | Probe that fires when the processing of a query is started. arg0 is the query string.
query-done | (const char *) | Probe that fires when the processing of a query is complete. arg0 is the query string.
query-parse-start | (const char *) | Probe that fires when the parsing of a query is started. arg0 is the query string.
query-parse-done | (const char *) | Probe that fires when the parsing of a query is complete. arg0 is the query string.
query-rewrite-start | (const char *) | Probe that fires when the rewriting of a query is started. arg0 is the query string.
query-rewrite-done | (const char *) | Probe that fires when the rewriting of a query is complete. arg0 is the query string.
query-plan-start | () | Probe that fires when the planning of a query is started.
query-plan-done | () | Probe that fires when the planning of a query is complete.
query-execute-start | () | Probe that fires when the execution of a query is started.
query-execute-done | () | Probe that fires when the execution of a query is complete.
statement-status | (const char *) | Probe that fires anytime the server process updates its pg_stat_activity.status. arg0 is the new status string.
checkpoint-start | (int) | Probe that fires when a checkpoint is started. arg0 holds the bitwise flags used to distinguish different checkpoint types, such as shutdown, immediate or force.
checkpoint-done | (int, int, int, int, int) | Probe that fires when a checkpoint is complete. (The probes listed next fire in sequence during checkpoint processing.) arg0 is the number of buffers written. arg1 is the total number of buffers. arg2, arg3 and arg4 contain the number of WAL files added, removed and recycled respectively.
clog-checkpoint-start | (bool) | Probe that fires when the CLOG portion of a checkpoint is started. arg0 is true for normal checkpoint, false for shutdown checkpoint.
clog-checkpoint-done | (bool) | Probe that fires when the CLOG portion of a checkpoint is complete. arg0 has the same meaning as for clog-checkpoint-start.
subtrans-checkpoint-start | (bool) | Probe that fires when the SUBTRANS portion of a checkpoint is started. arg0 is true for normal checkpoint, false for shutdown checkpoint.
subtrans-checkpoint-done | (bool) | Probe that fires when the SUBTRANS portion of a checkpoint is complete. arg0 has the same meaning as for subtrans-checkpoint-start.
multixact-checkpoint-start | (bool) | Probe that fires when the MultiXact portion of a checkpoint is started. arg0 is true for normal checkpoint, false for shutdown checkpoint.
multixact-checkpoint-done | (bool) | Probe that fires when the MultiXact portion of a checkpoint is complete. arg0 has the same meaning as for multixact-checkpoint-start.
buffer-checkpoint-start | (int) | Probe that fires when the buffer-writing portion of a checkpoint is started. arg0 holds the bitwise flags used to distinguish different checkpoint types, such as shutdown, immediate or force.
buffer-sync-start | (int, int) | Probe that fires when we begin to write dirty buffers during checkpoint (after identifying which buffers must be written). arg0 is the total number of buffers. arg1 is the number that are currently dirty and need to be written.
buffer-sync-written | (int) | Probe that fires after each buffer is written during checkpoint. arg0 is the ID number of the buffer.
buffer-sync-done | (int, int, int) | Probe that fires when all dirty buffers have been written. arg0 is the total number of buffers. arg1 is the number of buffers actually written by the checkpoint process. arg2 is the number that were expected to be written (arg1 of buffer-sync-start); any difference reflects other processes flushing buffers during the checkpoint.
buffer-checkpoint-sync-start | () | Probe that fires after dirty buffers have been written to the kernel, and before starting to issue fsync requests.
buffer-checkpoint-done | () | Probe that fires when syncing of buffers to disk is complete.
twophase-checkpoint-start | () | Probe that fires when the two-phase portion of a checkpoint is started.
twophase-checkpoint-done | () | Probe that fires when the two-phase portion of a checkpoint is complete.
buffer-read-start | (ForkNumber, BlockNumber, Oid, Oid, Oid, int, bool) | Probe that fires when a buffer read is started. arg0 and arg1 contain the fork and block numbers of the page (but arg1 will be -1 if this is a relation extension request). arg2, arg3, and arg4 contain the tablespace, database, and relation OIDs identifying the relation. arg5 is the ID of the backend which created the temporary relation for a local buffer, or InvalidBackendId (-1) for a shared buffer. arg6 is true for a relation extension request, false for normal read.
buffer-read-done | (ForkNumber, BlockNumber, Oid, Oid, Oid, int, bool, bool) | Probe that fires when a buffer read is complete. arg0 and arg1 contain the fork and block numbers of the page (if this is a relation extension request, arg1 now contains the block number of the newly added block). arg2, arg3, and arg4 contain the tablespace, database, and relation OIDs identifying the relation. arg5 is the ID of the backend which created the temporary relation for a local buffer, or InvalidBackendId (-1) for a shared buffer. arg6 is true for a relation extension request, false for normal read. arg7 is true if the buffer was found in the pool, false if not.
buffer-flush-start | (ForkNumber, BlockNumber, Oid, Oid, Oid) | Probe that fires before issuing any write request for a shared buffer. arg0 and arg1 contain the fork and block numbers of the page. arg2, arg3, and arg4 contain the tablespace, database, and relation OIDs identifying the relation.
buffer-flush-done | (ForkNumber, BlockNumber, Oid, Oid, Oid) | Probe that fires when a write request is complete. (Note that this just reflects the time to pass the data to the kernel; it's typically not actually been written to disk yet.) The arguments are the same as for buffer-flush-start.
buffer-write-dirty-start | (ForkNumber, BlockNumber, Oid, Oid, Oid) | Probe that fires when a server process begins to write a dirty buffer. (If this happens often, it implies that shared_buffers is too small or the background writer control parameters need adjustment.) arg0 and arg1 contain the fork and block numbers of the page. arg2, arg3, and arg4 contain the tablespace, database, and relation OIDs identifying the relation.
buffer-write-dirty-done | (ForkNumber, BlockNumber, Oid, Oid, Oid) | Probe that fires when a dirty-buffer write is complete. The arguments are the same as for buffer-write-dirty-start.
wal-buffer-write-dirty-start | () | Probe that fires when a server process begins to write a dirty WAL buffer because no more WAL buffer space is available. (If this happens often, it implies that wal_buffers is too small.)
wal-buffer-write-dirty-done | () | Probe that fires when a dirty WAL buffer write is complete.
wal-insert | (unsigned char, unsigned char) | Probe that fires when a WAL record is inserted. arg0 is the resource manager (rmid) for the record. arg1 contains the info flags.
wal-switch | () | Probe that fires when a WAL segment switch is requested.
smgr-md-read-start | (ForkNumber, BlockNumber, Oid, Oid, Oid, int) | Probe that fires when beginning to read a block from a relation. arg0 and arg1 contain the fork and block numbers of the page. arg2, arg3, and arg4 contain the tablespace, database, and relation OIDs identifying the relation. arg5 is the ID of the backend which created the temporary relation for a local buffer, or InvalidBackendId (-1) for a shared buffer.
smgr-md-read-done | (ForkNumber, BlockNumber, Oid, Oid, Oid, int, int, int) | Probe that fires when a block read is complete. arg0 and arg1 contain the fork and block numbers of the page. arg2, arg3, and arg4 contain the tablespace, database, and relation OIDs identifying the relation. arg5 is the ID of the backend which created the temporary relation for a local buffer, or InvalidBackendId (-1) for a shared buffer. arg6 is the number of bytes actually read, while arg7 is the number requested (if these are different it indicates trouble).
smgr-md-write-start | (ForkNumber, BlockNumber, Oid, Oid, Oid, int) | Probe that fires when beginning to write a block to a relation. arg0 and arg1 contain the fork and block numbers of the page. arg2, arg3, and arg4 contain the tablespace, database, and relation OIDs identifying the relation. arg5 is the ID of the backend which created the temporary relation for a local buffer, or InvalidBackendId (-1) for a shared buffer.
smgr-md-write-done | (ForkNumber, BlockNumber, Oid, Oid, Oid, int, int, int) | Probe that fires when a block write is complete. arg0 and arg1 contain the fork and block numbers of the page. arg2, arg3, and arg4 contain the tablespace, database, and relation OIDs identifying the relation. arg5 is the ID of the backend which created the temporary relation for a local buffer, or InvalidBackendId (-1) for a shared buffer. arg6 is the number of bytes actually written, while arg7 is the number requested (if these are different it indicates trouble).
sort-start | (int, bool, int, int, bool, int) | Probe that fires when a sort operation is started. arg0 indicates heap, index or datum sort. arg1 is true for unique-value enforcement. arg2 is the number of key columns. arg3 is the number of kilobytes of work memory allowed. arg4 is true if random access to the sort result is required. arg5 indicates serial when 0, parallel worker when 1, or parallel leader when 2.
sort-done | (bool, long) | Probe that fires when a sort is complete. arg0 is true for external sort, false for internal sort. arg1 is the number of disk blocks used for an external sort, or kilobytes of memory used for an internal sort.
lwlock-acquire | (char *, LWLockMode) | Probe that fires when an LWLock has been acquired. arg0 is the LWLock's tranche. arg1 is the requested lock mode, either exclusive or shared.
lwlock-release | (char *) | Probe that fires when an LWLock has been released (but note that any released waiters have not yet been awakened). arg0 is the LWLock's tranche.
lwlock-wait-start | (char *, LWLockMode) | Probe that fires when an LWLock was not immediately available and a server process has begun to wait for the lock to become available. arg0 is the LWLock's tranche. arg1 is the requested lock mode, either exclusive or shared.
lwlock-wait-done | (char *, LWLockMode) | Probe that fires when a server process has been released from its wait for an LWLock (it does not actually have the lock yet). arg0 is the LWLock's tranche. arg1 is the requested lock mode, either exclusive or shared.
lwlock-condacquire | (char *, LWLockMode) | Probe that fires when an LWLock was successfully acquired when the caller specified no waiting. arg0 is the LWLock's tranche. arg1 is the requested lock mode, either exclusive or shared.
lwlock-condacquire-fail | (char *, LWLockMode) | Probe that fires when an LWLock was not successfully acquired when the caller specified no waiting. arg0 is the LWLock's tranche. arg1 is the requested lock mode, either exclusive or shared.
lock-wait-start | (unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, LOCKMODE) | Probe that fires when a request for a heavyweight lock (lmgr lock) has begun to wait because the lock is not available. arg0 through arg3 are the tag fields identifying the object being locked. arg4 indicates the type of object being locked. arg5 indicates the lock type being requested.
lock-wait-done | (unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, LOCKMODE) | Probe that fires when a request for a heavyweight lock (lmgr lock) has finished waiting (i.e., has acquired the lock). The arguments are the same as for lock-wait-start.
deadlock-found | () | Probe that fires when a deadlock is found by the deadlock detector.

Table 28.46. Defined Types Used in Probe Parameters

Type | Definition
LocalTransactionId | unsigned int
LWLockMode | int
LOCKMODE | int
BlockNumber | unsigned int
Oid | unsigned int
ForkNumber | int
bool | unsigned char

28.5.3. Using Probes

The example below shows a DTrace script for analyzing transaction counts in the system, as an alternative to snapshotting pg_stat_database before and after a performance test:

#!/usr/sbin/dtrace -qs

postgresql$1:::transaction-start
{
      @start["Start"] = count();
      self->ts  = timestamp;
}

postgresql$1:::transaction-abort
{
      @abort["Abort"] = count();
}

postgresql$1:::transaction-commit
/self->ts/
{
      @commit["Commit"] = count();
      @time["Total time (ns)"] = sum(timestamp - self->ts);
      self->ts=0;
}

When executed, the example D script gives output such as:

# ./txn_count.d `pgrep -n postgres` or ./txn_count.d <PID>
^C

Start                                          71
Commit                                         70
Total time (ns)                        2312105013

Note

SystemTap uses a different notation for trace scripts than DTrace does, even though the underlying trace points are compatible. One point worth noting is that at this writing, SystemTap scripts must reference probe names using double underscores in place of hyphens. This is expected to be fixed in future SystemTap releases.

You should remember that DTrace scripts need to be carefully written and debugged, otherwise the trace information collected might be meaningless. In most cases where problems are found it is the instrumentation that is at fault, not the underlying system. When discussing information found using dynamic tracing, be sure to enclose the script used to allow that too to be checked and discussed.

28.5.4. Defining New Probes

New probes can be defined within the code wherever the developer desires, though this will require a recompilation. Below are the steps for inserting new probes:

  1. Decide on probe names and data to be made available through the probes

  2. Add the probe definitions to src/backend/utils/probes.d

  3. Include pg_trace.h if it is not already present in the module(s) containing the probe points, and insert TRACE_POSTGRESQL probe macros at the desired locations in the source code

  4. Recompile and verify that the new probes are available

Example: Here is an example of how you would add a probe to trace all new transactions by transaction ID.

  1. Decide that the probe will be named transaction-start and requires a parameter of type LocalTransactionId

  2. Add the probe definition to src/backend/utils/probes.d:

    probe transaction__start(LocalTransactionId);

    Note the use of the double underline in the probe name. In a DTrace script using the probe, the double underline needs to be replaced with a hyphen, so transaction-start is the name to document for users.

  3. At compile time, transaction__start is converted to a macro called TRACE_POSTGRESQL_TRANSACTION_START (notice the underscores are single here), which is available by including pg_trace.h. Add the macro call to the appropriate location in the source code. In this case, it looks like the following:

    TRACE_POSTGRESQL_TRANSACTION_START(vxid.localTransactionId);

  4. After recompiling and running the new binary, check that your newly added probe is available by executing the following DTrace command. You should see similar output:

    # dtrace -ln transaction-start
   ID    PROVIDER          MODULE           FUNCTION NAME
18705 postgresql49878     postgres     StartTransactionCommand transaction-start
18755 postgresql49877     postgres     StartTransactionCommand transaction-start
18805 postgresql49876     postgres     StartTransactionCommand transaction-start
18855 postgresql49875     postgres     StartTransactionCommand transaction-start
18986 postgresql49873     postgres     StartTransactionCommand transaction-start

There are a few things to be careful about when adding trace macros to the C code:

  • You should take care that the data types specified for a probe's parameters match the data types of the variables used in the macro. Otherwise, you will get compilation errors.

  • On most platforms, if PostgreSQL is built with --enable-dtrace, the arguments to a trace macro will be evaluated whenever control passes through the macro, even if no tracing is being done. This is usually not worth worrying about if you are just reporting the values of a few local variables. But beware of putting expensive function calls into the arguments. If you need to do that, consider protecting the macro with a check to see if the trace is actually enabled:

    if (TRACE_POSTGRESQL_TRANSACTION_START_ENABLED())
    TRACE_POSTGRESQL_TRANSACTION_START(some_function(...));

    Each trace macro has a corresponding ENABLED macro.

Prev | Up | Next
28.4. Progress Reporting | Home | Chapter 29. Monitoring Disk Usage


Chapter 29. Monitoring Disk Usage
Prev Up Part III. Server Administration Home Next

Chapter 29. Monitoring Disk Usage

Table of Contents

29.1. Determining Disk Usage
29.2. Disk Full Failure

This chapter discusses how to monitor the disk usage of a PostgreSQL database system.

Prev | Up | Next
28.5. Dynamic Tracing | Home | 29.1. Determining Disk Usage


29.1. Determining Disk Usage
Prev Up Chapter 29. Monitoring Disk Usage Home Next

29.1. Determining Disk Usage

Each table has a primary heap disk file where most of the data is stored. If the table has any columns with potentially-wide values, there also might be a TOAST file associated with the table, which is used to store values too wide to fit comfortably in the main table (see Section 70.2). There will be one valid index on the TOAST table, if present. There also might be indexes associated with the base table. Each table and index is stored in a separate disk file — possibly more than one file, if the file would exceed one gigabyte. Naming conventions for these files are described in Section 70.1.

You can monitor disk space in three ways: using the SQL functions listed in Table 9.92, using the oid2name module, or using manual inspection of the system catalogs. The SQL functions are the easiest to use and are generally recommended. The remainder of this section shows how to do it by inspection of the system catalogs.

Using psql on a recently vacuumed or analyzed database, you can issue queries to see the disk usage of any table:

SELECT pg_relation_filepath(oid), relpages FROM pg_class WHERE relname = 'customer';

 pg_relation_filepath | relpages
----------------------+----------
 base/16384/16806     |       60
(1 row)

Each page is typically 8 kilobytes. (Remember, relpages is only updated by VACUUM, ANALYZE, and a few DDL commands such as CREATE INDEX.) The file path name is of interest if you want to examine the table's disk file directly.

To show the space used by TOAST tables, use a query like the following:

SELECT relname, relpages
FROM pg_class,
     (SELECT reltoastrelid
      FROM pg_class
      WHERE relname = 'customer') AS ss
WHERE oid = ss.reltoastrelid OR
      oid = (SELECT indexrelid
             FROM pg_index
             WHERE indrelid = ss.reltoastrelid)
ORDER BY relname;

       relname        | relpages
----------------------+----------
 pg_toast_16806       |        0
 pg_toast_16806_index |        1

You can easily display index sizes, too:

SELECT c2.relname, c2.relpages
FROM pg_class c, pg_class c2, pg_index i
WHERE c.relname = 'customer' AND
      c.oid = i.indrelid AND
      c2.oid = i.indexrelid
ORDER BY c2.relname;

      relname      | relpages
-------------------+----------
 customer_id_index |       26

It is easy to find your largest tables and indexes using this information:

SELECT relname, relpages
FROM pg_class
ORDER BY relpages DESC;

       relname        | relpages
----------------------+----------
 bigtable             |     3290
 customer             |     3144

Prev | Up | Next
Chapter 29. Monitoring Disk Usage | Home | 29.2. Disk Full Failure


29.2. Disk Full Failure
Prev Up Chapter 29. Monitoring Disk Usage Home Next

29.2. Disk Full Failure

The most important disk monitoring task of a database administrator is to make sure the disk doesn't become full. A filled data disk will not result in data corruption, but it might prevent useful activity from occurring. If the disk holding the WAL files grows full, database server panic and consequent shutdown might occur.

If you cannot free up additional space on the disk by deleting other things, you can move some of the database files to other file systems by making use of tablespaces. See Section 23.6 for more information about that.

Tip

Some file systems perform badly when they are almost full, so do not wait until the disk is completely full to take action.

If your system supports per-user disk quotas, then the database will naturally be subject to whatever quota is placed on the user the server runs as. Exceeding the quota will have the same bad effects as running out of disk space entirely.

Prev | Up | Next
29.1. Determining Disk Usage | Home | Chapter 30. Reliability and the Write-Ahead Log


30.2. Data Checksums
Prev Up Chapter 30. Reliability and the Write-Ahead Log Home Next

30.2. Data Checksums

30.2.1. Off-line Enabling of Checksums

By default, data pages are not protected by checksums, but this can optionally be enabled for a cluster. When enabled, each data page includes a checksum that is updated when the page is written and verified each time the page is read. Only data pages are protected by checksums; internal data structures and temporary files are not.

Checksums are normally enabled when the cluster is initialized using initdb. They can also be enabled or disabled at a later time as an offline operation. Data checksums are enabled or disabled at the full cluster level, and cannot be specified individually for databases or tables.

The current state of checksums in the cluster can be verified by viewing the value of the read-only configuration variable data_checksums by issuing the command SHOW data_checksums.

When attempting to recover from page corruptions, it may be necessary to bypass the checksum protection. To do this, temporarily set the configuration parameter ignore_checksum_failure.

30.2.1. Off-line Enabling of Checksums

The pg_checksums application can be used to enable or disable data checksums, as well as verify checksums, on an offline cluster.

Prev | Up | Next
30.1. Reliability | Home | 30.3. Write-Ahead Logging (WAL)


Chapter 31. Logical Replication
Prev Up Part III. Server Administration Home Next

Chapter 31. Logical Replication

Table of Contents

31.1. Publication
31.2. Subscription
31.2.1. Replication Slot Management
31.3. Conflicts
31.4. Restrictions
31.5. Architecture
31.5.1. Initial Snapshot
31.6. Monitoring
31.7. Security
31.8. Configuration Settings
31.9. Quick Setup

Logical replication is a method of replicating data objects and their changes, based upon their replication identity (usually a primary key). We use the term logical in contrast to physical replication, which uses exact block addresses and byte-by-byte replication. PostgreSQL supports both mechanisms concurrently, see Chapter 27. Logical replication allows fine-grained control over both data replication and security.

Logical replication uses a publish and subscribe model with one or more subscribers subscribing to one or more publications on a publisher node. Subscribers pull data from the publications they subscribe to and may subsequently re-publish data to allow cascading replication or more complex configurations.

Logical replication of a table typically starts with taking a snapshot of the data on the publisher database and copying that to the subscriber. Once that is done, the changes on the publisher are sent to the subscriber as they occur in real-time. The subscriber applies the data in the same order as the publisher so that transactional consistency is guaranteed for publications within a single subscription. This method of data replication is sometimes referred to as transactional replication.

The typical use-cases for logical replication are:

  • Sending incremental changes in a single database or a subset of a database to subscribers as they occur.

  • Firing triggers for individual changes as they arrive on the subscriber.

  • Consolidating multiple databases into a single one (for example for analytical purposes).

  • Replicating between different major versions of PostgreSQL.

  • Replicating between PostgreSQL instances on different platforms (for example Linux to Windows)

  • Giving access to replicated data to different groups of users.

  • Sharing a subset of the database between multiple databases.

The subscriber database behaves in the same way as any other PostgreSQL instance and can be used as a publisher for other databases by defining its own publications. When the subscriber is treated as read-only by application, there will be no conflicts from a single subscription. On the other hand, if there are other writes done either by an application or by other subscribers to the same set of tables, conflicts can arise.

Prev | Up | Next
30.6. WAL Internals | Home | 31.1. Publication


31.1. Publication
Prev Up Chapter 31. Logical Replication Home Next

31.1. Publication

A publication can be defined on any physical replication primary. The node where a publication is defined is referred to as publisher. A publication is a set of changes generated from a table or a group of tables, and might also be described as a change set or replication set. Each publication exists in only one database.

Publications are different from schemas and do not affect how the table is accessed. Each table can be added to multiple publications if needed. Publications may currently only contain tables. Objects must be added explicitly, except when a publication is created for ALL TABLES.

Publications can choose to limit the changes they produce to any combination of INSERT, UPDATE, DELETE, and TRUNCATE, similar to how triggers are fired by particular event types. By default, all operation types are replicated.

A published table must have a “replica identity” configured in order to be able to replicate UPDATE and DELETE operations, so that appropriate rows to update or delete can be identified on the subscriber side. By default, this is the primary key, if there is one. Another unique index (with certain additional requirements) can also be set to be the replica identity. If the table does not have any suitable key, then it can be set to replica identity “full”, which means the entire row becomes the key. This, however, is very inefficient and should only be used as a fallback if no other solution is possible. If a replica identity other than “full” is set on the publisher side, a replica identity comprising the same or fewer columns must also be set on the subscriber side. See REPLICA IDENTITY for details on how to set the replica identity. If a table without a replica identity is added to a publication that replicates UPDATE or DELETE operations then subsequent UPDATE or DELETE operations will cause an error on the publisher. INSERT operations can proceed regardless of any replica identity.

Every publication can have multiple subscribers.

A publication is created using the CREATE PUBLICATION command and may later be altered or dropped using corresponding commands.

The individual tables can be added and removed dynamically using ALTER PUBLICATION. Both the ADD TABLE and DROP TABLE operations are transactional; so the table will start or stop replicating at the correct snapshot once the transaction has committed.

Prev | Up | Next
Chapter 31. Logical Replication | Home | 31.2. Subscription


31.2. Subscription
Prev Up Chapter 31. Logical Replication Home Next

31.2. Subscription

31.2.1. Replication Slot Management

A subscription is the downstream side of logical replication. The node where a subscription is defined is referred to as the subscriber. A subscription defines the connection to another database and set of publications (one or more) to which it wants to subscribe.

The subscriber database behaves in the same way as any other PostgreSQL instance and can be used as a publisher for other databases by defining its own publications.

A subscriber node may have multiple subscriptions if desired. It is possible to define multiple subscriptions between a single publisher-subscriber pair, in which case care must be taken to ensure that the subscribed publication objects don't overlap.

Each subscription will receive changes via one replication slot (see Section 27.2.6). Additional replication slots may be required for the initial data synchronization of pre-existing table data and those will be dropped at the end of data synchronization.

A logical replication subscription can be a standby for synchronous replication (see Section 27.2.8). The standby name is by default the subscription name. An alternative name can be specified as application_name in the connection information of the subscription.

Subscriptions are dumped by pg_dump if the current user is a superuser. Otherwise a warning is written and subscriptions are skipped, because non-superusers cannot read all subscription information from the pg_subscription catalog.

The subscription is added using CREATE SUBSCRIPTION and can be stopped/resumed at any time using the ALTER SUBSCRIPTION command and removed using DROP SUBSCRIPTION.

When a subscription is dropped and recreated, the synchronization information is lost. This means that the data has to be resynchronized afterwards.

The schema definitions are not replicated, and the published tables must exist on the subscriber. Only regular tables may be the target of replication. For example, you can't replicate to a view.

The tables are matched between the publisher and the subscriber using the fully qualified table name. Replication to differently-named tables on the subscriber is not supported.

Columns of a table are also matched by name. The order of columns in the subscriber table does not need to match that of the publisher. The data types of the columns do not need to match, as long as the text representation of the data can be converted to the target type. For example, you can replicate from a column of type integer to a column of type bigint. The target table can also have additional columns not provided by the published table. Any such columns will be filled with the default value as specified in the definition of the target table.

31.2.1. Replication Slot Management

As mentioned earlier, each (active) subscription receives changes from a replication slot on the remote (publishing) side.

Additional table synchronization slots are normally transient, created internally to perform initial table synchronization and dropped automatically when they are no longer needed. These table synchronization slots have generated names: “pg_%u_sync_%u_%llu” (parameters: Subscription oid, Table relid, system identifier sysid)

Normally, the remote replication slot is created automatically when the subscription is created using CREATE SUBSCRIPTION and it is dropped automatically when the subscription is dropped using DROP SUBSCRIPTION. In some situations, however, it can be useful or necessary to manipulate the subscription and the underlying replication slot separately. Here are some scenarios:

  • When creating a subscription, the replication slot already exists. In that case, the subscription can be created using the create_slot = false option to associate with the existing slot.

  • When creating a subscription, the remote host is not reachable or in an unclear state. In that case, the subscription can be created using the connect = false option. The remote host will then not be contacted at all. This is what pg_dump uses. The remote replication slot will then have to be created manually before the subscription can be activated.

  • When dropping a subscription, the replication slot should be kept. This could be useful when the subscriber database is being moved to a different host and will be activated from there. In that case, disassociate the slot from the subscription using ALTER SUBSCRIPTION before attempting to drop the subscription.

  • When dropping a subscription, the remote host is not reachable. In that case, disassociate the slot from the subscription using ALTER SUBSCRIPTION before attempting to drop the subscription. If the remote database instance no longer exists, no further action is then necessary. If, however, the remote database instance is just unreachable, the replication slot (and any still remaining table synchronization slots) should then be dropped manually; otherwise it/they would continue to reserve WAL and might eventually cause the disk to fill up. Such cases should be carefully investigated.

Prev | Up | Next
31.1. Publication | Home | 31.3. Conflicts


31.3. Conflicts
Prev Up Chapter 31. Logical Replication Home Next

31.3. Conflicts

Logical replication behaves similarly to normal DML operations in that the data will be updated even if it was changed locally on the subscriber node. If incoming data violates any constraints the replication will stop. This is referred to as a conflict. When replicating UPDATE or DELETE operations, missing data will not produce a conflict and such operations will simply be skipped.

A conflict will produce an error and will stop the replication; it must be resolved manually by the user. Details about the conflict can be found in the subscriber's server log.

The resolution can be done either by changing data on the subscriber so that it does not conflict with the incoming change or by skipping the transaction that conflicts with the existing data. The transaction can be skipped by calling the pg_replication_origin_advance() function with a node_name corresponding to the subscription name, and a position. The current position of origins can be seen in the pg_replication_origin_status system view.

Prev | Up | Next
31.2. Subscription | Home | 31.4. Restrictions


31.4. Restrictions
Prev Up Chapter 31. Logical Replication Home Next

31.4. Restrictions

Logical replication currently has the following restrictions or missing functionality. These might be addressed in future releases.

  • The database schema and DDL commands are not replicated. The initial schema can be copied by hand using pg_dump --schema-only. Subsequent schema changes would need to be kept in sync manually. (Note, however, that there is no need for the schemas to be absolutely the same on both sides.) Logical replication is robust when schema definitions change in a live database: When the schema is changed on the publisher and replicated data starts arriving at the subscriber but does not fit into the table schema, replication will error until the schema is updated. In many cases, intermittent errors can be avoided by applying additive schema changes to the subscriber first.

  • Sequence data is not replicated. The data in serial or identity columns backed by sequences will of course be replicated as part of the table, but the sequence itself would still show the start value on the subscriber. If the subscriber is used as a read-only database, then this should typically not be a problem. If, however, some kind of switchover or failover to the subscriber database is intended, then the sequences would need to be updated to the latest values, either by copying the current data from the publisher (perhaps using pg_dump) or by determining a sufficiently high value from the tables themselves.

  • Replication of TRUNCATE commands is supported, but some care must be taken when truncating groups of tables connected by foreign keys. When replicating a truncate action, the subscriber will truncate the same group of tables that was truncated on the publisher, either explicitly specified or implicitly collected via CASCADE, minus tables that are not part of the subscription. This will work correctly if all affected tables are part of the same subscription. But if some tables to be truncated on the subscriber have foreign-key links to tables that are not part of the same (or any) subscription, then the application of the truncate action on the subscriber will fail.

  • Large objects (see Chapter 35) are not replicated. There is no workaround for that, other than storing data in normal tables.

  • Replication is only supported by tables, including partitioned tables. Attempts to replicate other types of relations, such as views, materialized views, or foreign tables, will result in an error.

  • When replicating between partitioned tables, the actual replication originates, by default, from the leaf partitions on the publisher, so partitions on the publisher must also exist on the subscriber as valid target tables. (They could either be leaf partitions themselves, or they could be further subpartitioned, or they could even be independent tables.) Publications can also specify that changes are to be replicated using the identity and schema of the partitioned root table instead of that of the individual leaf partitions in which the changes actually originate (see CREATE PUBLICATION).

Prev | Up | Next
31.3. Conflicts | Home | 31.5. Architecture


31.5. Architecture
Prev Up Chapter 31. Logical Replication Home Next

31.5. Architecture

31.5.1. Initial Snapshot

Logical replication starts by copying a snapshot of the data on the publisher database. Once that is done, changes on the publisher are sent to the subscriber as they occur in real time. The subscriber applies data in the order in which commits were made on the publisher so that transactional consistency is guaranteed for the publications within any single subscription.

Logical replication is built with an architecture similar to physical streaming replication (see Section 27.2.5). It is implemented by “walsender” and “apply” processes. The walsender process starts logical decoding (described in Chapter 49) of the WAL and loads the standard logical decoding plugin (pgoutput). The plugin transforms the changes read from WAL to the logical replication protocol (see Section 53.5) and filters the data according to the publication specification. The data is then continuously transferred using the streaming replication protocol to the apply worker, which maps the data to local tables and applies the individual changes as they are received, in correct transactional order.

The apply process on the subscriber database always runs with session_replication_role set to replica, which produces the usual effects on triggers and constraints.

The logical replication apply process currently only fires row triggers, not statement triggers. The initial table synchronization, however, is implemented like a COPY command and thus fires both row and statement triggers for INSERT.

31.5.1. Initial Snapshot

The initial data in existing subscribed tables are snapshotted and copied in a parallel instance of a special kind of apply process. This process will create its own replication slot and copy the existing data. As soon as the copy is finished the table contents will become visible to other backends. Once existing data is copied, the worker enters synchronization mode, which ensures that the table is brought up to a synchronized state with the main apply process by streaming any changes that happened during the initial data copy using standard logical replication. During this synchronization phase, the changes are applied and committed in the same order as they happened on the publisher. Once synchronization is done, control of the replication of the table is given back to the main apply process where replication continues as normal.

Prev | Up | Next
31.4. Restrictions | Home | 31.6. Monitoring


31.6. Monitoring
Prev Up Chapter 31. Logical Replication Home Next

31.6. Monitoring

Because logical replication is based on a similar architecture as physical streaming replication, the monitoring on a publication node is similar to monitoring of a physical replication primary (see Section 27.2.5.2).

The monitoring information about subscription is visible in pg_stat_subscription. This view contains one row for every subscription worker. A subscription can have zero or more active subscription workers depending on its state.

Normally, there is a single apply process running for an enabled subscription. A disabled subscription or a crashed subscription will have zero rows in this view. If the initial data synchronization of any table is in progress, there will be additional workers for the tables being synchronized.

Prev | Up | Next
31.5. Architecture | Home | 31.7. Security


31.7. Security
Prev Up Chapter 31. Logical Replication Home Next

31.7. Security

A user able to modify the schema of subscriber-side tables can execute arbitrary code as a superuser. Limit ownership and TRIGGER privilege on such tables to roles that superusers trust. Moreover, if untrusted users can create tables, use only publications that list tables explicitly. That is to say, create a subscription FOR ALL TABLES only when superusers trust every user permitted to create a non-temp table on the publisher or the subscriber.

The role used for the replication connection must have the REPLICATION attribute (or be a superuser). If the role lacks SUPERUSER and BYPASSRLS, publisher row security policies can execute. If the role does not trust all table owners, include options=-crow_security=off in the connection string; if a table owner then adds a row security policy, that setting will cause replication to halt rather than execute the policy. Access for the role must be configured in pg_hba.conf and it must have the LOGIN attribute.

In order to be able to copy the initial table data, the role used for the replication connection must have the SELECT privilege on a published table (or be a superuser).

To create a publication, the user must have the CREATE privilege in the database.

To add tables to a publication, the user must have ownership rights on the table. To create a publication that publishes all tables automatically, the user must be a superuser.

To create a subscription, the user must be a superuser.

The subscription apply process will run in the local database with the privileges of a superuser.

Privileges are only checked once at the start of a replication connection. They are not re-checked as each change record is read from the publisher, nor are they re-checked for each change when applied.

Prev | Up | Next
31.6. Monitoring | Home | 31.8. Configuration Settings


31.8. Configuration Settings
Prev Up Chapter 31. Logical Replication Home Next

31.8. Configuration Settings

Logical replication requires several configuration options to be set.

On the publisher side, wal_level must be set to logical, and max_replication_slots must be set to at least the number of subscriptions expected to connect, plus some reserve for table synchronization. And max_wal_senders should be set to at least the same as max_replication_slots plus the number of physical replicas that are connected at the same time.

max_replication_slots must also be set on the subscriber. It should be set to at least the number of subscriptions that will be added to the subscriber, plus some reserve for table synchronization. max_logical_replication_workers must be set to at least the number of subscriptions, again plus some reserve for the table synchronization. Additionally the max_worker_processes may need to be adjusted to accommodate for replication workers, at least (max_logical_replication_workers + 1). Note that some extensions and parallel queries also take worker slots from max_worker_processes.

Prev | Up | Next
31.7. Security | Home | 31.9. Quick Setup


31.9. Quick Setup
Prev Up Chapter 31. Logical Replication Home Next

31.9. Quick Setup

First set the configuration options in postgresql.conf:

wal_level = logical

The other required settings have default values that are sufficient for a basic setup.

pg_hba.conf needs to be adjusted to allow replication (the values here depend on your actual network configuration and user you want to use for connecting):

host     all     repuser     0.0.0.0/0     md5

Then on the publisher database:

CREATE PUBLICATION mypub FOR TABLE users, departments;

And on the subscriber database:

CREATE SUBSCRIPTION mysub CONNECTION 'dbname=foo host=bar user=repuser' PUBLICATION mypub;

The above will start the replication process, which synchronizes the initial table contents of the tables users and departments and then starts replicating incremental changes to those tables.

Prev | Up | Next
31.8. Configuration Settings | Home | Chapter 32. Just-in-Time Compilation (JIT)


Chapter 32. Just-in-Time Compilation (JIT)
Prev Up Part III. Server Administration Home Next

Chapter 32. Just-in-Time Compilation (JIT)

Table of Contents

32.1. What Is JIT compilation?
32.1.1. JIT Accelerated Operations
32.1.2. Inlining
32.1.3. Optimization
32.2. When to JIT?
32.3. Configuration
32.4. Extensibility
32.4.1. Inlining Support for Extensions
32.4.2. Pluggable JIT Providers

This chapter explains what just-in-time compilation is, and how it can be configured in PostgreSQL.

Prev | Up | Next
31.9. Quick Setup | Home | 32.1. What Is JIT compilation?


32.1. What Is JIT compilation?
Prev Up Chapter 32. Just-in-Time Compilation (JIT) Home Next

32.1. What Is JIT compilation?

32.1.1. JIT Accelerated Operations
32.1.2. Inlining
32.1.3. Optimization

Just-in-Time (JIT) compilation is the process of turning some form of interpreted program evaluation into a native program, and doing so at run time. For example, instead of using general-purpose code that can evaluate arbitrary SQL expressions to evaluate a particular SQL predicate like WHERE a.col = 3, it is possible to generate a function that is specific to that expression and can be natively executed by the CPU, yielding a speedup.

PostgreSQL has builtin support to perform JIT compilation using LLVM when PostgreSQL is built with --with-llvm.

See src/backend/jit/README for further details.

32.1.1. JIT Accelerated Operations

Currently PostgreSQL's JIT implementation has support for accelerating expression evaluation and tuple deforming. Several other operations could be accelerated in the future.

Expression evaluation is used to evaluate WHERE clauses, target lists, aggregates and projections. It can be accelerated by generating code specific to each case.

Tuple deforming is the process of transforming an on-disk tuple (see Section 70.6.1) into its in-memory representation. It can be accelerated by creating a function specific to the table layout and the number of columns to be extracted.

32.1.2. Inlining

PostgreSQL is very extensible and allows new data types, functions, operators and other database objects to be defined; see Chapter 38. In fact the built-in objects are implemented using nearly the same mechanisms. This extensibility implies some overhead, for example due to function calls (see Section 38.3). To reduce that overhead, JIT compilation can inline the bodies of small functions into the expressions using them. That allows a significant percentage of the overhead to be optimized away.

32.1.3. Optimization

LLVM has support for optimizing generated code. Some of the optimizations are cheap enough to be performed whenever JIT is used, while others are only beneficial for longer-running queries. See https://llvm.org/docs/Passes.html#transform-passes for more details about optimizations.

Prev | Up | Next
Chapter 32. Just-in-Time Compilation (JIT) | Home | 32.2. When to JIT?


32.2. When to JIT?
Prev Up Chapter 32. Just-in-Time Compilation (JIT) Home Next

32.2. When to JIT?

JIT compilation is beneficial primarily for long-running CPU-bound queries. Frequently these will be analytical queries. For short queries the added overhead of performing JIT compilation will often be higher than the time it can save.

To determine whether JIT compilation should be used, the total estimated cost of a query (see Chapter 72 and Section 20.7.2) is used. The estimated cost of the query will be compared with the setting of jit_above_cost. If the cost is higher, JIT compilation will be performed. Two further decisions are then needed. Firstly, if the estimated cost is more than the setting of jit_inline_above_cost, short functions and operators used in the query will be inlined. Secondly, if the estimated cost is more than the setting of jit_optimize_above_cost, expensive optimizations are applied to improve the generated code. Each of these options increases the JIT compilation overhead, but can reduce query execution time considerably.

These cost-based decisions will be made at plan time, not execution time. This means that when prepared statements are in use, and a generic plan is used (see PREPARE), the values of the configuration parameters in effect at prepare time control the decisions, not the settings at execution time.

Note

If jit is set to off, or if no JIT implementation is available (for example because the server was compiled without --with-llvm), JIT will not be performed, even if it would be beneficial based on the above criteria. Setting jit to off has effects at both plan and execution time.

EXPLAIN can be used to see whether JIT is used or not. As an example, here is a query that is not using JIT:

=# EXPLAIN ANALYZE SELECT SUM(relpages) FROM pg_class;
                                                 QUERY PLAN
-------------------------------------------------------------------​------------------------------------------
 Aggregate  (cost=16.27..16.29 rows=1 width=8) (actual time=0.303..0.303 rows=1 loops=1)
   ->  Seq Scan on pg_class  (cost=0.00..15.42 rows=342 width=4) (actual time=0.017..0.111 rows=356 loops=1)
 Planning Time: 0.116 ms
 Execution Time: 0.365 ms
(4 rows)

Given the cost of the plan, it is entirely reasonable that no JIT was used; the cost of JIT would have been bigger than the potential savings. Adjusting the cost limits will lead to JIT use:

=# SET jit_above_cost = 10;
SET
=# EXPLAIN ANALYZE SELECT SUM(relpages) FROM pg_class;
                                                 QUERY PLAN
-------------------------------------------------------------------​------------------------------------------
 Aggregate  (cost=16.27..16.29 rows=1 width=8) (actual time=6.049..6.049 rows=1 loops=1)
   ->  Seq Scan on pg_class  (cost=0.00..15.42 rows=342 width=4) (actual time=0.019..0.052 rows=356 loops=1)
 Planning Time: 0.133 ms
 JIT:
   Functions: 3
   Options: Inlining false, Optimization false, Expressions true, Deforming true
   Timing: Generation 1.259 ms, Inlining 0.000 ms, Optimization 0.797 ms, Emission 5.048 ms, Total 7.104 ms
 Execution Time: 7.416 ms

As visible here, JIT was used, but inlining and expensive optimization were not. If jit_inline_above_cost or jit_optimize_above_cost were also lowered, that would change.

Prev | Up | Next
32.1. What Is JIT compilation? | Home | 32.3. Configuration


32.3. Configuration
Prev Up Chapter 32. Just-in-Time Compilation (JIT) Home Next

32.3. Configuration

The configuration variable jit determines whether JIT compilation is enabled or disabled. If it is enabled, the configuration variables jit_above_cost, jit_inline_above_cost, and jit_optimize_above_cost determine whether JIT compilation is performed for a query, and how much effort is spent doing so.

jit_provider determines which JIT implementation is used. It is rarely required to be changed. See Section 32.4.2.

For development and debugging purposes a few additional configuration parameters exist, as described in Section 20.17.

Prev | Up | Next
32.2. When to JIT? | Home | 32.4. Extensibility


32.4. Extensibility
Prev Up Chapter 32. Just-in-Time Compilation (JIT) Home Next

32.4. Extensibility

32.4.1. Inlining Support for Extensions
32.4.2. Pluggable JIT Providers

32.4.1. Inlining Support for Extensions

PostgreSQL's JIT implementation can inline the bodies of functions of types C and internal, as well as operators based on such functions. To do so for functions in extensions, the definitions of those functions need to be made available. When using PGXS to build an extension against a server that has been compiled with LLVM JIT support, the relevant files will be built and installed automatically.

The relevant files have to be installed into $pkglibdir/bitcode/$extension/ and a summary of them into $pkglibdir/bitcode/$extension.index.bc, where $pkglibdir is the directory returned by pg_config --pkglibdir and $extension is the base name of the extension's shared library.

Note

For functions built into PostgreSQL itself, the bitcode is installed into $pkglibdir/bitcode/postgres.

32.4.2. Pluggable JIT Providers

PostgreSQL provides a JIT implementation based on LLVM. The interface to the JIT provider is pluggable and the provider can be changed without recompiling (although currently, the build process only provides inlining support data for LLVM). The active provider is chosen via the setting jit_provider.

32.4.2.1. JIT Provider Interface

A JIT provider is loaded by dynamically loading the named shared library. The normal library search path is used to locate the library. To provide the required JIT provider callbacks and to indicate that the library is actually a JIT provider, it needs to provide a C function named _PG_jit_provider_init. This function is passed a struct that needs to be filled with the callback function pointers for individual actions:

struct JitProviderCallbacks
{
    JitProviderResetAfterErrorCB reset_after_error;
    JitProviderReleaseContextCB release_context;
    JitProviderCompileExprCB compile_expr;
};

extern void _PG_jit_provider_init(JitProviderCallbacks *cb);

Prev | Up | Next
32.3. Configuration | Home | Chapter 33. Regression Tests


Chapter 33. Regression Tests
Prev Up Part III. Server Administration Home Next

Chapter 33. Regression Tests

Table of Contents

33.1. Running the Tests
33.1.1. Running the Tests Against a Temporary Installation
33.1.2. Running the Tests Against an Existing Installation
33.1.3. Additional Test Suites
33.1.4. Locale and Encoding
33.1.5. Custom Server Settings
33.1.6. Extra Tests
33.1.7. Testing Hot Standby
33.2. Test Evaluation
33.2.1. Error Message Differences
33.2.2. Locale Differences
33.2.3. Date and Time Differences
33.2.4. Floating-Point Differences
33.2.5. Row Ordering Differences
33.2.6. Insufficient Stack Depth
33.2.7. The “random” Test
33.2.8. Configuration Parameters
33.3. Variant Comparison Files
33.4. TAP Tests
33.5. Test Coverage Examination

The regression tests are a comprehensive set of tests for the SQL implementation in PostgreSQL. They test standard SQL operations as well as the extended capabilities of PostgreSQL.

Prev | Up | Next
32.4. Extensibility | Home | 33.1. Running the Tests


33.1. Running the Tests
Prev Up Chapter 33. Regression Tests Home Next

33.1. Running the Tests

33.1.1. Running the Tests Against a Temporary Installation
33.1.2. Running the Tests Against an Existing Installation
33.1.3. Additional Test Suites
33.1.4. Locale and Encoding
33.1.5. Custom Server Settings
33.1.6. Extra Tests
33.1.7. Testing Hot Standby

The regression tests can be run against an already installed and running server, or using a temporary installation within the build tree. Furthermore, there is a “parallel” and a “sequential” mode for running the tests. The sequential method runs each test script alone, while the parallel method starts up multiple server processes to run groups of tests in parallel. Parallel testing adds confidence that interprocess communication and locking are working correctly.

33.1.1. Running the Tests Against a Temporary Installation

To run the parallel regression tests after building but before installation, type:

make check

in the top-level directory. (Or you can change to src/test/regress and run the command there.) At the end you should see something like:

=======================
 All 193 tests passed.
=======================

or otherwise a note about which tests failed. See Section 33.2 below before assuming that a “failure” represents a serious problem.

Because this test method runs a temporary server, it will not work if you did the build as the root user, since the server will not start as root. Recommended procedure is not to do the build as root, or else to perform testing after completing the installation.

If you have configured PostgreSQL to install into a location where an older PostgreSQL installation already exists, and you perform make check before installing the new version, you might find that the tests fail because the new programs try to use the already-installed shared libraries. (Typical symptoms are complaints about undefined symbols.) If you wish to run the tests before overwriting the old installation, you'll need to build with configure --disable-rpath. It is not recommended that you use this option for the final installation, however.

The parallel regression test starts quite a few processes under your user ID. Presently, the maximum concurrency is twenty parallel test scripts, which means forty processes: there's a server process and a psql process for each test script. So if your system enforces a per-user limit on the number of processes, make sure this limit is at least fifty or so, else you might get random-seeming failures in the parallel test. If you are not in a position to raise the limit, you can cut down the degree of parallelism by setting the MAX_CONNECTIONS parameter. For example:

make MAX_CONNECTIONS=10 check

runs no more than ten tests concurrently.

33.1.2. Running the Tests Against an Existing Installation

To run the tests after installation (see Chapter 17), initialize a data directory and start the server as explained in Chapter 19, then type:

make installcheck

or for a parallel test:

make installcheck-parallel

The tests will expect to contact the server at the local host and the default port number, unless directed otherwise by PGHOST and PGPORT environment variables. The tests will be run in a database named regression; any existing database by this name will be dropped.

The tests will also transiently create some cluster-wide objects, such as roles, tablespaces, and subscriptions. These objects will have names beginning with regress_. Beware of using installcheck mode with an installation that has any actual global objects named that way.

33.1.3. Additional Test Suites

The make check and make installcheck commands run only the “core” regression tests, which test built-in functionality of the PostgreSQL server. The source distribution contains many additional test suites, most of them having to do with add-on functionality such as optional procedural languages.

To run all test suites applicable to the modules that have been selected to be built, including the core tests, type one of these commands at the top of the build tree:

make check-world
make installcheck-world

These commands run the tests using temporary servers or an already-installed server, respectively, just as previously explained for make check and make installcheck. Other considerations are the same as previously explained for each method. Note that make check-world builds a separate instance (temporary data directory) for each tested module, so it requires more time and disk space than make installcheck-world.

On a modern machine with multiple CPU cores and no tight operating-system limits, you can make things go substantially faster with parallelism. The recipe that most PostgreSQL developers actually use for running all tests is something like

make check-world -j8 >/dev/null

with a -j limit near to or a bit more than the number of available cores. Discarding stdout eliminates chatter that's not interesting when you just want to verify success. (In case of failure, the stderr messages are usually enough to determine where to look closer.)

Alternatively, you can run individual test suites by typing make check or make installcheck in the appropriate subdirectory of the build tree. Keep in mind that make installcheck assumes you've installed the relevant module(s), not only the core server.

The additional tests that can be invoked this way include:

  • Regression tests for optional procedural languages. These are located under src/pl.

  • Regression tests for contrib modules, located under contrib. Not all contrib modules have tests.

  • Regression tests for the ECPG interface library, located in src/interfaces/ecpg/test.

  • Tests for core-supported authentication methods, located in src/test/authentication. (See below for additional authentication-related tests.)

  • Tests stressing behavior of concurrent sessions, located in src/test/isolation.

  • Tests for crash recovery and physical replication, located in src/test/recovery.

  • Tests for logical replication, located in src/test/subscription.

  • Tests of client programs, located under src/bin.

When using installcheck mode, these tests will create and destroy test databases whose names include regression, for example pl_regression or contrib_regression. Beware of using installcheck mode with an installation that has any non-test databases named that way.

Some of these auxiliary test suites use the TAP infrastructure explained in Section 33.4. The TAP-based tests are run only when PostgreSQL was configured with the option --enable-tap-tests. This is recommended for development, but can be omitted if there is no suitable Perl installation.

Some test suites are not run by default, either because they are not secure to run on a multiuser system or because they require special software. You can decide which test suites to run additionally by setting the make or environment variable PG_TEST_EXTRA to a whitespace-separated list, for example:

make check-world PG_TEST_EXTRA='kerberos ldap ssl'

The following values are currently supported:

kerberos

Runs the test suite under src/test/kerberos. This requires an MIT Kerberos installation and opens TCP/IP listen sockets.

ldap

Runs the test suite under src/test/ldap. This requires an OpenLDAP installation and opens TCP/IP listen sockets.

ssl

Runs the test suite under src/test/ssl. This opens TCP/IP listen sockets.

Tests for features that are not supported by the current build configuration are not run even if they are mentioned in PG_TEST_EXTRA.

In addition, there are tests in src/test/modules which will be run by make check-world but not by make installcheck-world. This is because they install non-production extensions or have other side-effects that are considered undesirable for a production installation. You can use make install and make installcheck in one of those subdirectories if you wish, but it's not recommended to do so with a non-test server.

33.1.4. Locale and Encoding

By default, tests using a temporary installation use the locale defined in the current environment and the corresponding database encoding as determined by initdb. It can be useful to test different locales by setting the appropriate environment variables, for example:

make check LANG=C
make check LC_COLLATE=en_US.utf8 LC_CTYPE=fr_CA.utf8

For implementation reasons, setting LC_ALL does not work for this purpose; all the other locale-related environment variables do work.

When testing against an existing installation, the locale is determined by the existing database cluster and cannot be set separately for the test run.

You can also choose the database encoding explicitly by setting the variable ENCODING, for example:

make check LANG=C ENCODING=EUC_JP

Setting the database encoding this way typically only makes sense if the locale is C; otherwise the encoding is chosen automatically from the locale, and specifying an encoding that does not match the locale will result in an error.

The database encoding can be set for tests against either a temporary or an existing installation, though in the latter case it must be compatible with the installation's locale.

33.1.5. Custom Server Settings

Custom server settings to use when running a regression test suite can be set in the PGOPTIONS environment variable (for settings that allow this):

make check PGOPTIONS="-c force_parallel_mode=regress -c work_mem=50MB"

When running against a temporary installation, custom settings can also be set by supplying a pre-written postgresql.conf:

echo 'log_checkpoints = on' > test_postgresql.conf
echo 'work_mem = 50MB' >> test_postgresql.conf
make check EXTRA_REGRESS_OPTS="--temp-config=test_postgresql.conf"

This can be useful to enable additional logging, adjust resource limits, or enable extra run-time checks such as debug_discard_caches.

33.1.6. Extra Tests

The core regression test suite contains a few test files that are not run by default, because they might be platform-dependent or take a very long time to run. You can run these or other extra test files by setting the variable EXTRA_TESTS. For example, to run the numeric_big test:

make check EXTRA_TESTS=numeric_big

33.1.7. Testing Hot Standby

The source distribution also contains regression tests for the static behavior of Hot Standby. These tests require a running primary server and a running standby server that is accepting new WAL changes from the primary (using either file-based log shipping or streaming replication). Those servers are not automatically created for you, nor is replication setup documented here. Please check the various sections of the documentation devoted to the required commands and related issues.

To run the Hot Standby tests, first create a database called regression on the primary:

psql -h primary -c "CREATE DATABASE regression"

Next, run the preparatory script src/test/regress/sql/hs_primary_setup.sql on the primary in the regression database, for example:

psql -h primary -f src/test/regress/sql/hs_primary_setup.sql regression

Allow these changes to propagate to the standby.

Now arrange for the default database connection to be to the standby server under test (for example, by setting the PGHOST and PGPORT environment variables). Finally, run make standbycheck in the regression directory:

cd src/test/regress
make standbycheck

Some extreme behaviors can also be generated on the primary using the script src/test/regress/sql/hs_primary_extremes.sql to allow the behavior of the standby to be tested.

Prev | Up | Next
Chapter 33. Regression Tests | Home | 33.2. Test Evaluation


33.2. Test Evaluation
Prev Up Chapter 33. Regression Tests Home Next

33.2. Test Evaluation

33.2.1. Error Message Differences
33.2.2. Locale Differences
33.2.3. Date and Time Differences
33.2.4. Floating-Point Differences
33.2.5. Row Ordering Differences
33.2.6. Insufficient Stack Depth
33.2.7. The “random” Test
33.2.8. Configuration Parameters

Some properly installed and fully functional PostgreSQL installations can “fail” some of these regression tests due to platform-specific artifacts such as varying floating-point representation and message wording. The tests are currently evaluated using a simple diff comparison against the outputs generated on a reference system, so the results are sensitive to small system differences. When a test is reported as “failed”, always examine the differences between expected and actual results; you might find that the differences are not significant. Nonetheless, we still strive to maintain accurate reference files across all supported platforms, so it can be expected that all tests pass.

The actual outputs of the regression tests are in files in the src/test/regress/results directory. The test script uses diff to compare each output file against the reference outputs stored in the src/test/regress/expected directory. Any differences are saved for your inspection in src/test/regress/regression.diffs. (When running a test suite other than the core tests, these files of course appear in the relevant subdirectory, not src/test/regress.)

If you don't like the diff options that are used by default, set the environment variable PG_REGRESS_DIFF_OPTS, for instance PG_REGRESS_DIFF_OPTS='-c'. (Or you can run diff yourself, if you prefer.)

If for some reason a particular platform generates a “failure” for a given test, but inspection of the output convinces you that the result is valid, you can add a new comparison file to silence the failure report in future test runs. See Section 33.3 for details.

33.2.1. Error Message Differences

Some of the regression tests involve intentional invalid input values. Error messages can come from either the PostgreSQL code or from the host platform system routines. In the latter case, the messages can vary between platforms, but should reflect similar information. These differences in messages will result in a “failed” regression test that can be validated by inspection.

33.2.2. Locale Differences

If you run the tests against a server that was initialized with a collation-order locale other than C, then there might be differences due to sort order and subsequent failures. The regression test suite is set up to handle this problem by providing alternate result files that together are known to handle a large number of locales.

To run the tests in a different locale when using the temporary-installation method, pass the appropriate locale-related environment variables on the make command line, for example:

make check LANG=de_DE.utf8

(The regression test driver unsets LC_ALL, so it does not work to choose the locale using that variable.) To use no locale, either unset all locale-related environment variables (or set them to C) or use the following special invocation:

make check NO_LOCALE=1

When running the tests against an existing installation, the locale setup is determined by the existing installation. To change it, initialize the database cluster with a different locale by passing the appropriate options to initdb.

In general, it is advisable to try to run the regression tests in the locale setup that is wanted for production use, as this will exercise the locale- and encoding-related code portions that will actually be used in production. Depending on the operating system environment, you might get failures, but then you will at least know what locale-specific behaviors to expect when running real applications.

33.2.3. Date and Time Differences

Most of the date and time results are dependent on the time zone environment. The reference files are generated for time zone PST8PDT (Berkeley, California), and there will be apparent failures if the tests are not run with that time zone setting. The regression test driver sets environment variable PGTZ to PST8PDT, which normally ensures proper results.

33.2.4. Floating-Point Differences

Some of the tests involve computing 64-bit floating-point numbers (double precision) from table columns. Differences in results involving mathematical functions of double precision columns have been observed. The float8 and geometry tests are particularly prone to small differences across platforms, or even with different compiler optimization settings. Human eyeball comparison is needed to determine the real significance of these differences which are usually 10 places to the right of the decimal point.

Some systems display minus zero as -0, while others just show 0.

Some systems signal errors from pow() and exp() differently from the mechanism expected by the current PostgreSQL code.

33.2.5. Row Ordering Differences

You might see differences in which the same rows are output in a different order than what appears in the expected file. In most cases this is not, strictly speaking, a bug. Most of the regression test scripts are not so pedantic as to use an ORDER BY for every single SELECT, and so their result row orderings are not well-defined according to the SQL specification. In practice, since we are looking at the same queries being executed on the same data by the same software, we usually get the same result ordering on all platforms, so the lack of ORDER BY is not a problem. Some queries do exhibit cross-platform ordering differences, however. When testing against an already-installed server, ordering differences can also be caused by non-C locale settings or non-default parameter settings, such as custom values of work_mem or the planner cost parameters.

Therefore, if you see an ordering difference, it's not something to worry about, unless the query does have an ORDER BY that your result is violating. However, please report it anyway, so that we can add an ORDER BY to that particular query to eliminate the bogus “failure” in future releases.

You might wonder why we don't order all the regression test queries explicitly to get rid of this issue once and for all. The reason is that that would make the regression tests less useful, not more, since they'd tend to exercise query plan types that produce ordered results to the exclusion of those that don't.

33.2.6. Insufficient Stack Depth

If the errors test results in a server crash at the select infinite_recurse() command, it means that the platform's limit on process stack size is smaller than the max_stack_depth parameter indicates. This can be fixed by running the server under a higher stack size limit (4MB is recommended with the default value of max_stack_depth). If you are unable to do that, an alternative is to reduce the value of max_stack_depth.

On platforms supporting getrlimit(), the server should automatically choose a safe value of max_stack_depth; so unless you've manually overridden this setting, a failure of this kind is a reportable bug.

33.2.7. The “random” Test

The random test script is intended to produce random results. In very rare cases, this causes that regression test to fail. Typing:

diff results/random.out expected/random.out

should produce only one or a few lines of differences. You need not worry unless the random test fails repeatedly.

33.2.8. Configuration Parameters

When running the tests against an existing installation, some non-default parameter settings could cause the tests to fail. For example, changing parameters such as enable_seqscan or enable_indexscan could cause plan changes that would affect the results of tests that use EXPLAIN.

Prev | Up | Next
33.1. Running the Tests | Home | 33.3. Variant Comparison Files


33.3. Variant Comparison Files
Prev Up Chapter 33. Regression Tests Home Next

33.3. Variant Comparison Files

Since some of the tests inherently produce environment-dependent results, we have provided ways to specify alternate “expected” result files. Each regression test can have several comparison files showing possible results on different platforms. There are two independent mechanisms for determining which comparison file is used for each test.

The first mechanism allows comparison files to be selected for specific platforms. There is a mapping file, src/test/regress/resultmap, that defines which comparison file to use for each platform. To eliminate bogus test “failures” for a particular platform, you first choose or make a variant result file, and then add a line to the resultmap file.

Each line in the mapping file is of the form

testname:output:platformpattern=comparisonfilename

The test name is just the name of the particular regression test module. The output value indicates which output file to check. For the standard regression tests, this is always out. The value corresponds to the file extension of the output file. The platform pattern is a pattern in the style of the Unix tool expr (that is, a regular expression with an implicit ^ anchor at the start). It is matched against the platform name as printed by config.guess. The comparison file name is the base name of the substitute result comparison file.

For example: some systems lack a working strtof function, for which our workaround causes rounding errors in the float4 regression test. Therefore, we provide a variant comparison file, float4-misrounded-input.out, which includes the results to be expected on these systems. To silence the bogus “failure” message on HP-UX 10 platforms, resultmap includes:

float4:out:hppa.*-hp-hpux10.*=float4-misrounded-input.out

which will trigger on any machine where the output of config.guess matches hppa.*-hp-hpux10.*. Other lines in resultmap select the variant comparison file for other platforms where it's appropriate.

The second selection mechanism for variant comparison files is much more automatic: it simply uses the “best match” among several supplied comparison files. The regression test driver script considers both the standard comparison file for a test, testname.out, and variant files named testname_digit.out (where the digit is any single digit 0-9). If any such file is an exact match, the test is considered to pass; otherwise, the one that generates the shortest diff is used to create the failure report. (If resultmap includes an entry for the particular test, then the base testname is the substitute name given in resultmap.)

For example, for the char test, the comparison file char.out contains results that are expected in the C and POSIX locales, while the file char_1.out contains results sorted as they appear in many other locales.

The best-match mechanism was devised to cope with locale-dependent results, but it can be used in any situation where the test results cannot be predicted easily from the platform name alone. A limitation of this mechanism is that the test driver cannot tell which variant is actually “correct” for the current environment; it will just pick the variant that seems to work best. Therefore it is safest to use this mechanism only for variant results that you are willing to consider equally valid in all contexts.

Prev | Up | Next
33.2. Test Evaluation | Home | 33.4. TAP Tests


33.4. TAP Tests
Prev Up Chapter 33. Regression Tests Home Next

33.4. TAP Tests

Various tests, particularly the client program tests under src/bin, use the Perl TAP tools and are run using the Perl testing program prove. You can pass command-line options to prove by setting the make variable PROVE_FLAGS, for example:

make -C src/bin check PROVE_FLAGS='--timer'

See the manual page of prove for more information.

The make variable PROVE_TESTS can be used to define a whitespace-separated list of paths relative to the Makefile invoking prove to run the specified subset of tests instead of the default t/*.pl. For example:

make check PROVE_TESTS='t/001_test1.pl t/003_test3.pl'

The TAP tests require the Perl module IPC::Run. This module is available from CPAN or an operating system package. They also require PostgreSQL to be configured with the option --enable-tap-tests.

Generically speaking, the TAP tests will test the executables in a previously-installed installation tree if you say make installcheck, or will build a new local installation tree from current sources if you say make check. In either case they will initialize a local instance (data directory) and transiently run a server in it. Some of these tests run more than one server. Thus, these tests can be fairly resource-intensive.

It's important to realize that the TAP tests will start test server(s) even when you say make installcheck; this is unlike the traditional non-TAP testing infrastructure, which expects to use an already-running test server in that case. Some PostgreSQL subdirectories contain both traditional-style and TAP-style tests, meaning that make installcheck will produce a mix of results from temporary servers and the already-running test server.

Prev | Up | Next
33.3. Variant Comparison Files | Home | 33.5. Test Coverage Examination


33.5. Test Coverage Examination
Prev Up Chapter 33. Regression Tests Home Next

33.5. Test Coverage Examination

The PostgreSQL source code can be compiled with coverage testing instrumentation, so that it becomes possible to examine which parts of the code are covered by the regression tests or any other test suite that is run with the code. This is currently supported when compiling with GCC, and it requires the gcov and lcov programs.

A typical workflow looks like this:

./configure --enable-coverage ... OTHER OPTIONS ...
make
make check # or other test suite
make coverage-html

Then point your HTML browser to coverage/index.html.

If you don't have lcov or prefer text output over an HTML report, you can run

make coverage

instead of make coverage-html, which will produce .gcov output files for each source file relevant to the test. (make coverage and make coverage-html will overwrite each other's files, so mixing them might be confusing.)

You can run several different tests before making the coverage report; the execution counts will accumulate. If you want to reset the execution counts between test runs, run:

make coverage-clean

You can run the make coverage-html or make coverage command in a subdirectory if you want a coverage report for only a portion of the code tree.

Use make distclean to clean up when done.

Prev | Up | Next
33.4. TAP Tests | Home | Part IV. Client Interfaces


Part IV. Client Interfaces
Prev Up PostgreSQL 14.4 Documentation Home Next

Part IV. Client Interfaces

This part describes the client programming interfaces distributed with PostgreSQL. Each of these chapters can be read independently. Note that there are many other programming interfaces for client programs that are distributed separately and contain their own documentation (Appendix H lists some of the more popular ones). Readers of this part should be familiar with using SQL commands to manipulate and query the database (see Part II) and of course with the programming language that the interface uses.

Table of Contents

34. libpq — C Library
34.1. Database Connection Control Functions
34.2. Connection Status Functions
34.3. Command Execution Functions
34.4. Asynchronous Command Processing
34.5. Pipeline Mode
34.6. Retrieving Query Results Row-by-Row
34.7. Canceling Queries in Progress
34.8. The Fast-Path Interface
34.9. Asynchronous Notification
34.10. Functions Associated with the COPY Command
34.11. Control Functions
34.12. Miscellaneous Functions
34.13. Notice Processing
34.14. Event System
34.15. Environment Variables
34.16. The Password File
34.17. The Connection Service File
34.18. LDAP Lookup of Connection Parameters
34.19. SSL Support
34.20. Behavior in Threaded Programs
34.21. Building libpq Programs
34.22. Example Programs
35. Large Objects
35.1. Introduction
35.2. Implementation Features
35.3. Client Interfaces
35.4. Server-Side Functions
35.5. Example Program
36. ECPG — Embedded SQL in C
36.1. The Concept
36.2. Managing Database Connections
36.3. Running SQL Commands
36.4. Using Host Variables
36.5. Dynamic SQL
36.6. pgtypes Library
36.7. Using Descriptor Areas
36.8. Error Handling
36.9. Preprocessor Directives
36.10. Processing Embedded SQL Programs
36.11. Library Functions
36.12. Large Objects
36.13. C++ Applications
36.14. Embedded SQL Commands
36.15. Informix Compatibility Mode
36.16. Internals
37. The Information Schema
37.1. The Schema
37.2. Data Types
37.3. information_schema_catalog_name
37.4. administrable_role_​authorizations
37.5. applicable_roles
37.6. attributes
37.7. character_sets
37.8. check_constraint_routine_usage
37.9. check_constraints
37.10. collations
37.11. collation_character_set_​applicability
37.12. column_column_usage
37.13. column_domain_usage
37.14. column_options
37.15. column_privileges
37.16. column_udt_usage
37.17. columns
37.18. constraint_column_usage
37.19. constraint_table_usage
37.20. data_type_privileges
37.21. domain_constraints
37.22. domain_udt_usage
37.23. domains
37.24. element_types
37.25. enabled_roles
37.26. foreign_data_wrapper_options
37.27. foreign_data_wrappers
37.28. foreign_server_options
37.29. foreign_servers
37.30. foreign_table_options
37.31. foreign_tables
37.32. key_column_usage
37.33. parameters
37.34. referential_constraints
37.35. role_column_grants
37.36. role_routine_grants
37.37. role_table_grants
37.38. role_udt_grants
37.39. role_usage_grants
37.40. routine_column_usage
37.41. routine_privileges
37.42. routine_routine_usage
37.43. routine_sequence_usage
37.44. routine_table_usage
37.45. routines
37.46. schemata
37.47. sequences
37.48. sql_features
37.49. sql_implementation_info
37.50. sql_parts
37.51. sql_sizing
37.52. table_constraints
37.53. table_privileges
37.54. tables
37.55. transforms
37.56. triggered_update_columns
37.57. triggers
37.58. udt_privileges
37.59. usage_privileges
37.60. user_defined_types
37.61. user_mapping_options
37.62. user_mappings
37.63. view_column_usage
37.64. view_routine_usage
37.65. view_table_usage
37.66. views
Prev | Up | Next
33.5. Test Coverage Examination | Home | Chapter 34. libpq — C Library


Chapter 34. libpq — C Library
Prev Up Part IV. Client Interfaces Home Next

Chapter 34. libpq — C Library

Table of Contents

34.1. Database Connection Control Functions
34.1.1. Connection Strings
34.1.2. Parameter Key Words
34.2. Connection Status Functions
34.3. Command Execution Functions
34.3.1. Main Functions
34.3.2. Retrieving Query Result Information
34.3.3. Retrieving Other Result Information
34.3.4. Escaping Strings for Inclusion in SQL Commands
34.4. Asynchronous Command Processing
34.5. Pipeline Mode
34.5.1. Using Pipeline Mode
34.5.2. Functions Associated with Pipeline Mode
34.5.3. When to Use Pipeline Mode
34.6. Retrieving Query Results Row-by-Row
34.7. Canceling Queries in Progress
34.8. The Fast-Path Interface
34.9. Asynchronous Notification
34.10. Functions Associated with the COPY Command
34.10.1. Functions for Sending COPY Data
34.10.2. Functions for Receiving COPY Data
34.10.3. Obsolete Functions for COPY
34.11. Control Functions
34.12. Miscellaneous Functions
34.13. Notice Processing
34.14. Event System
34.14.1. Event Types
34.14.2. Event Callback Procedure
34.14.3. Event Support Functions
34.14.4. Event Example
34.15. Environment Variables
34.16. The Password File
34.17. The Connection Service File
34.18. LDAP Lookup of Connection Parameters
34.19. SSL Support
34.19.1. Client Verification of Server Certificates
34.19.2. Client Certificates
34.19.3. Protection Provided in Different Modes
34.19.4. SSL Client File Usage
34.19.5. SSL Library Initialization
34.20. Behavior in Threaded Programs
34.21. Building libpq Programs
34.22. Example Programs

libpq is the C application programmer's interface to PostgreSQL. libpq is a set of library functions that allow client programs to pass queries to the PostgreSQL backend server and to receive the results of these queries.

libpq is also the underlying engine for several other PostgreSQL application interfaces, including those written for C++, Perl, Python, Tcl and ECPG. So some aspects of libpq's behavior will be important to you if you use one of those packages. In particular, Section 34.15, Section 34.16 and Section 34.19 describe behavior that is visible to the user of any application that uses libpq.

Some short programs are included at the end of this chapter (Section 34.22) to show how to write programs that use libpq. There are also several complete examples of libpq applications in the directory src/test/examples in the source code distribution.

Client programs that use libpq must include the header file libpq-fe.h and must link with the libpq library.

Prev | Up | Next
Part IV. Client Interfaces | Home | 34.1. Database Connection Control Functions


34.1. Database Connection Control Functions
Prev Up Chapter 34. libpq — C Library Home Next

34.1. Database Connection Control Functions

34.1.1. Connection Strings
34.1.2. Parameter Key Words

The following functions deal with making a connection to a PostgreSQL backend server. An application program can have several backend connections open at one time. (One reason to do that is to access more than one database.) Each connection is represented by a PGconn object, which is obtained from the function PQconnectdb, PQconnectdbParams, or PQsetdbLogin. Note that these functions will always return a non-null object pointer, unless perhaps there is too little memory even to allocate the PGconn object. The PQstatus function should be called to check the return value for a successful connection before queries are sent via the connection object.

Warning

If untrusted users have access to a database that has not adopted a secure schema usage pattern, begin each session by removing publicly-writable schemas from search_path. One can set parameter key word options to value -csearch_path=. Alternately, one can issue PQexec(conn, "SELECT pg_catalog.set_config('search_path', '', false)") after connecting. This consideration is not specific to libpq; it applies to every interface for executing arbitrary SQL commands.

Warning

On Unix, forking a process with open libpq connections can lead to unpredictable results because the parent and child processes share the same sockets and operating system resources. For this reason, such usage is not recommended, though doing an exec from the child process to load a new executable is safe.

PQconnectdbParams

Makes a new connection to the database server.

PGconn *PQconnectdbParams(const char * const *keywords,
                          const char * const *values,
                          int expand_dbname);

This function opens a new database connection using the parameters taken from two NULL-terminated arrays. The first, keywords, is defined as an array of strings, each one being a key word. The second, values, gives the value for each key word. Unlike PQsetdbLogin below, the parameter set can be extended without changing the function signature, so use of this function (or its nonblocking analogs PQconnectStartParams and PQconnectPoll) is preferred for new application programming.

The currently recognized parameter key words are listed in Section 34.1.2.

The passed arrays can be empty to use all default parameters, or can contain one or more parameter settings. They must be matched in length. Processing will stop at the first NULL entry in the keywords array. Also, if the values entry associated with a non-NULL keywords entry is NULL or an empty string, that entry is ignored and processing continues with the next pair of array entries.

When expand_dbname is non-zero, the value for the first dbname key word is checked to see if it is a connection string. If so, it is “expanded” into the individual connection parameters extracted from the string. The value is considered to be a connection string, rather than just a database name, if it contains an equal sign (=) or it begins with a URI scheme designator. (More details on connection string formats appear in Section 34.1.1.) Only the first occurrence of dbname is treated in this way; any subsequent dbname parameter is processed as a plain database name.

In general the parameter arrays are processed from start to end. If any key word is repeated, the last value (that is not NULL or empty) is used. This rule applies in particular when a key word found in a connection string conflicts with one appearing in the keywords array. Thus, the programmer may determine whether array entries can override or be overridden by values taken from a connection string. Array entries appearing before an expanded dbname entry can be overridden by fields of the connection string, and in turn those fields are overridden by array entries appearing after dbname (but, again, only if those entries supply non-empty values).

After processing all the array entries and any expanded connection string, any connection parameters that remain unset are filled with default values. If an unset parameter's corresponding environment variable (see Section 34.15) is set, its value is used. If the environment variable is not set either, then the parameter's built-in default value is used.

PQconnectdb

Makes a new connection to the database server.

PGconn *PQconnectdb(const char *conninfo);

This function opens a new database connection using the parameters taken from the string conninfo.

The passed string can be empty to use all default parameters, or it can contain one or more parameter settings separated by whitespace, or it can contain a URI. See Section 34.1.1 for details.

PQsetdbLogin

Makes a new connection to the database server.

PGconn *PQsetdbLogin(const char *pghost,
                     const char *pgport,
                     const char *pgoptions,
                     const char *pgtty,
                     const char *dbName,
                     const char *login,
                     const char *pwd);

This is the predecessor of PQconnectdb with a fixed set of parameters. It has the same functionality except that the missing parameters will always take on default values. Write NULL or an empty string for any one of the fixed parameters that is to be defaulted.

If the dbName contains an = sign or has a valid connection URI prefix, it is taken as a conninfo string in exactly the same way as if it had been passed to PQconnectdb, and the remaining parameters are then applied as specified for PQconnectdbParams.

pgtty is no longer used and any value passed will be ignored.

PQsetdb

Makes a new connection to the database server.

PGconn *PQsetdb(char *pghost,
                char *pgport,
                char *pgoptions,
                char *pgtty,
                char *dbName);

This is a macro that calls PQsetdbLogin with null pointers for the login and pwd parameters. It is provided for backward compatibility with very old programs.

PQconnectStartParams
PQconnectStart
PQconnectPoll

Make a connection to the database server in a nonblocking manner.

PGconn *PQconnectStartParams(const char * const *keywords,
                             const char * const *values,
                             int expand_dbname);

PGconn *PQconnectStart(const char *conninfo);

PostgresPollingStatusType PQconnectPoll(PGconn *conn);

These three functions are used to open a connection to a database server such that your application's thread of execution is not blocked on remote I/O whilst doing so. The point of this approach is that the waits for I/O to complete can occur in the application's main loop, rather than down inside PQconnectdbParams or PQconnectdb, and so the application can manage this operation in parallel with other activities.

With PQconnectStartParams, the database connection is made using the parameters taken from the keywords and values arrays, and controlled by expand_dbname, as described above for PQconnectdbParams.

With PQconnectStart, the database connection is made using the parameters taken from the string conninfo as described above for PQconnectdb.

Neither PQconnectStartParams nor PQconnectStart nor PQconnectPoll will block, so long as a number of restrictions are met:

  • The hostaddr parameter must be used appropriately to prevent DNS queries from being made. See the documentation of this parameter in Section 34.1.2 for details.

  • If you call PQtrace, ensure that the stream object into which you trace will not block.

  • You must ensure that the socket is in the appropriate state before calling PQconnectPoll, as described below.

To begin a nonblocking connection request, call PQconnectStart or PQconnectStartParams. If the result is null, then libpq has been unable to allocate a new PGconn structure. Otherwise, a valid PGconn pointer is returned (though not yet representing a valid connection to the database). Next call PQstatus(conn). If the result is CONNECTION_BAD, the connection attempt has already failed, typically because of invalid connection parameters.

If PQconnectStart or PQconnectStartParams succeeds, the next stage is to poll libpq so that it can proceed with the connection sequence. Use PQsocket(conn) to obtain the descriptor of the socket underlying the database connection. (Caution: do not assume that the socket remains the same across PQconnectPoll calls.) Loop thus: If PQconnectPoll(conn) last returned PGRES_POLLING_READING, wait until the socket is ready to read (as indicated by select(), poll(), or similar system function). Then call PQconnectPoll(conn) again. Conversely, if PQconnectPoll(conn) last returned PGRES_POLLING_WRITING, wait until the socket is ready to write, then call PQconnectPoll(conn) again. On the first iteration, i.e., if you have yet to call PQconnectPoll, behave as if it last returned PGRES_POLLING_WRITING. Continue this loop until PQconnectPoll(conn) returns PGRES_POLLING_FAILED, indicating the connection procedure has failed, or PGRES_POLLING_OK, indicating the connection has been successfully made.

At any time during connection, the status of the connection can be checked by calling PQstatus. If this call returns CONNECTION_BAD, then the connection procedure has failed; if the call returns CONNECTION_OK, then the connection is ready. Both of these states are equally detectable from the return value of PQconnectPoll, described above. Other states might also occur during (and only during) an asynchronous connection procedure. These indicate the current stage of the connection procedure and might be useful to provide feedback to the user for example. These statuses are:

CONNECTION_STARTED

Waiting for connection to be made.

CONNECTION_MADE

Connection OK; waiting to send.

CONNECTION_AWAITING_RESPONSE

Waiting for a response from the server.

CONNECTION_AUTH_OK

Received authentication; waiting for backend start-up to finish.

CONNECTION_SSL_STARTUP

Negotiating SSL encryption.

CONNECTION_SETENV

Negotiating environment-driven parameter settings.

CONNECTION_CHECK_WRITABLE

Checking if connection is able to handle write transactions.

CONNECTION_CONSUME

Consuming any remaining response messages on connection.

Note that, although these constants will remain (in order to maintain compatibility), an application should never rely upon these occurring in a particular order, or at all, or on the status always being one of these documented values. An application might do something like this:

switch(PQstatus(conn))
{
        case CONNECTION_STARTED:
            feedback = "Connecting...";
            break;

        case CONNECTION_MADE:
            feedback = "Connected to server...";
            break;
.
.
.
        default:
            feedback = "Connecting...";
}

The connect_timeout connection parameter is ignored when using PQconnectPoll; it is the application's responsibility to decide whether an excessive amount of time has elapsed. Otherwise, PQconnectStart followed by a PQconnectPoll loop is equivalent to PQconnectdb.

Note that when PQconnectStart or PQconnectStartParams returns a non-null pointer, you must call PQfinish when you are finished with it, in order to dispose of the structure and any associated memory blocks. This must be done even if the connection attempt fails or is abandoned.

PQconndefaults

Returns the default connection options.

PQconninfoOption *PQconndefaults(void);

typedef struct
{
    char   *keyword;   /* The keyword of the option */
    char   *envvar;    /* Fallback environment variable name */
    char   *compiled;  /* Fallback compiled in default value */
    char   *val;       /* Option's current value, or NULL */
    char   *label;     /* Label for field in connect dialog */
    char   *dispchar;  /* Indicates how to display this field
                          in a connect dialog. Values are:
                          ""        Display entered value as is
                          "*"       Password field - hide value
                          "D"       Debug option - don't show by default */
    int     dispsize;  /* Field size in characters for dialog */
} PQconninfoOption;

Returns a connection options array. This can be used to determine all possible PQconnectdb options and their current default values. The return value points to an array of PQconninfoOption structures, which ends with an entry having a null keyword pointer. The null pointer is returned if memory could not be allocated. Note that the current default values (val fields) will depend on environment variables and other context. A missing or invalid service file will be silently ignored. Callers must treat the connection options data as read-only.

After processing the options array, free it by passing it to PQconninfoFree. If this is not done, a small amount of memory is leaked for each call to PQconndefaults.

PQconninfo

Returns the connection options used by a live connection.

PQconninfoOption *PQconninfo(PGconn *conn);

Returns a connection options array. This can be used to determine all possible PQconnectdb options and the values that were used to connect to the server. The return value points to an array of PQconninfoOption structures, which ends with an entry having a null keyword pointer. All notes above for PQconndefaults also apply to the result of PQconninfo.

PQconninfoParse

Returns parsed connection options from the provided connection string.

PQconninfoOption *PQconninfoParse(const char *conninfo, char **errmsg);

Parses a connection string and returns the resulting options as an array; or returns NULL if there is a problem with the connection string. This function can be used to extract the PQconnectdb options in the provided connection string. The return value points to an array of PQconninfoOption structures, which ends with an entry having a null keyword pointer.

All legal options will be present in the result array, but the PQconninfoOption for any option not present in the connection string will have val set to NULL; default values are not inserted.

If errmsg is not NULL, then *errmsg is set to NULL on success, else to a malloc'd error string explaining the problem. (It is also possible for *errmsg to be set to NULL and the function to return NULL; this indicates an out-of-memory condition.)

After processing the options array, free it by passing it to PQconninfoFree. If this is not done, some memory is leaked for each call to PQconninfoParse. Conversely, if an error occurs and errmsg is not NULL, be sure to free the error string using PQfreemem.

PQfinish

Closes the connection to the server. Also frees memory used by the PGconn object.

void PQfinish(PGconn *conn);

Note that even if the server connection attempt fails (as indicated by PQstatus), the application should call PQfinish to free the memory used by the PGconn object. The PGconn pointer must not be used again after PQfinish has been called.

PQreset

Resets the communication channel to the server.

void PQreset(PGconn *conn);

This function will close the connection to the server and attempt to establish a new connection, using all the same parameters previously used. This might be useful for error recovery if a working connection is lost.

PQresetStart
PQresetPoll

Reset the communication channel to the server, in a nonblocking manner.

int PQresetStart(PGconn *conn);

PostgresPollingStatusType PQresetPoll(PGconn *conn);

These functions will close the connection to the server and attempt to establish a new connection, using all the same parameters previously used. This can be useful for error recovery if a working connection is lost. They differ from PQreset (above) in that they act in a nonblocking manner. These functions suffer from the same restrictions as PQconnectStartParams, PQconnectStart and PQconnectPoll.

To initiate a connection reset, call PQresetStart. If it returns 0, the reset has failed. If it returns 1, poll the reset using PQresetPoll in exactly the same way as you would create the connection using PQconnectPoll.

PQpingParams

PQpingParams reports the status of the server. It accepts connection parameters identical to those of PQconnectdbParams, described above. It is not necessary to supply correct user name, password, or database name values to obtain the server status; however, if incorrect values are provided, the server will log a failed connection attempt.

PGPing PQpingParams(const char * const *keywords,
                    const char * const *values,
                    int expand_dbname);

The function returns one of the following values:

PQPING_OK

The server is running and appears to be accepting connections.

PQPING_REJECT

The server is running but is in a state that disallows connections (startup, shutdown, or crash recovery).

PQPING_NO_RESPONSE

The server could not be contacted. This might indicate that the server is not running, or that there is something wrong with the given connection parameters (for example, wrong port number), or that there is a network connectivity problem (for example, a firewall blocking the connection request).

PQPING_NO_ATTEMPT

No attempt was made to contact the server, because the supplied parameters were obviously incorrect or there was some client-side problem (for example, out of memory).

PQping

PQping reports the status of the server. It accepts connection parameters identical to those of PQconnectdb, described above. It is not necessary to supply correct user name, password, or database name values to obtain the server status; however, if incorrect values are provided, the server will log a failed connection attempt.

PGPing PQping(const char *conninfo);

The return values are the same as for PQpingParams.

PQsetSSLKeyPassHook_OpenSSL

PQsetSSLKeyPassHook_OpenSSL lets an application override libpq's default handling of encrypted client certificate key files using sslpassword or interactive prompting.

void PQsetSSLKeyPassHook_OpenSSL(PQsslKeyPassHook_OpenSSL_type hook);

The application passes a pointer to a callback function with signature:

int callback_fn(char *buf, int size, PGconn *conn);

which libpq will then call instead of its default PQdefaultSSLKeyPassHook_OpenSSL handler. The callback should determine the password for the key and copy it to result-buffer buf of size size. The string in buf must be null-terminated. The callback must return the length of the password stored in buf excluding the null terminator. On failure, the callback should set buf[0] = '\0' and return 0. See PQdefaultSSLKeyPassHook_OpenSSL in libpq's source code for an example.

If the user specified an explicit key location, its path will be in conn->sslkey when the callback is invoked. This will be empty if the default key path is being used. For keys that are engine specifiers, it is up to engine implementations whether they use the OpenSSL password callback or define their own handling.

The app callback may choose to delegate unhandled cases to PQdefaultSSLKeyPassHook_OpenSSL, or call it first and try something else if it returns 0, or completely override it.

The callback must not escape normal flow control with exceptions, longjmp(...), etc. It must return normally.

PQgetSSLKeyPassHook_OpenSSL

PQgetSSLKeyPassHook_OpenSSL returns the current client certificate key password hook, or NULL if none has been set.

PQsslKeyPassHook_OpenSSL_type PQgetSSLKeyPassHook_OpenSSL(void);

34.1.1. Connection Strings

Several libpq functions parse a user-specified string to obtain connection parameters. There are two accepted formats for these strings: plain keyword/value strings and URIs. URIs generally follow RFC 3986, except that multi-host connection strings are allowed as further described below.

34.1.1.1. Keyword/Value Connection Strings

In the keyword/value format, each parameter setting is in the form keyword = value, with space(s) between settings. Spaces around a setting's equal sign are optional. To write an empty value, or a value containing spaces, surround it with single quotes, for example keyword = 'a value'. Single quotes and backslashes within a value must be escaped with a backslash, i.e., \' and \\.

Example:

host=localhost port=5432 dbname=mydb connect_timeout=10

The recognized parameter key words are listed in Section 34.1.2.

34.1.1.2. Connection URIs

The general form for a connection URI is:

postgresql://[userspec@][hostspec][/dbname][?paramspec]

where userspec is:

user[:password]

and hostspec is:

[host][:port][,...]

and paramspec is:

name=value[&...]

The URI scheme designator can be either postgresql:// or postgres://. Each of the remaining URI parts is optional. The following examples illustrate valid URI syntax:

postgresql://
postgresql://localhost
postgresql://localhost:5433
postgresql://localhost/mydb
postgresql://user@localhost
postgresql://user:secret@localhost
postgresql://other@localhost/otherdb?connect_timeout=10&application_name=myapp
postgresql://host1:123,host2:456/somedb?target_session_attrs=any&application_name=myapp

Values that would normally appear in the hierarchical part of the URI can alternatively be given as named parameters. For example:

postgresql:///mydb?host=localhost&port=5433

All named parameters must match key words listed in Section 34.1.2, except that for compatibility with JDBC connection URIs, instances of ssl=true are translated into sslmode=require.

The connection URI needs to be encoded with percent-encoding if it includes symbols with special meaning in any of its parts. Here is an example where the equal sign (=) is replaced with %3D and the space character with %20:

postgresql://user@localhost:5433/mydb?options=-c%20synchronous_commit%3Doff

The host part may be either a host name or an IP address. To specify an IPv6 address, enclose it in square brackets:

postgresql://[2001:db8::1234]/database

The host part is interpreted as described for the parameter host. In particular, a Unix-domain socket connection is chosen if the host part is either empty or looks like an absolute path name, otherwise a TCP/IP connection is initiated. Note, however, that the slash is a reserved character in the hierarchical part of the URI. So, to specify a non-standard Unix-domain socket directory, either omit the host part of the URI and specify the host as a named parameter, or percent-encode the path in the host part of the URI:

postgresql:///dbname?host=/var/lib/postgresql
postgresql://%2Fvar%2Flib%2Fpostgresql/dbname

It is possible to specify multiple host components, each with an optional port component, in a single URI. A URI of the form postgresql://host1:port1,host2:port2,host3:port3/ is equivalent to a connection string of the form host=host1,host2,host3 port=port1,port2,port3. As further described below, each host will be tried in turn until a connection is successfully established.

34.1.1.3. Specifying Multiple Hosts

It is possible to specify multiple hosts to connect to, so that they are tried in the given order. In the Keyword/Value format, the host, hostaddr, and port options accept comma-separated lists of values. The same number of elements must be given in each option that is specified, such that e.g., the first hostaddr corresponds to the first host name, the second hostaddr corresponds to the second host name, and so forth. As an exception, if only one port is specified, it applies to all the hosts.

In the connection URI format, you can list multiple host:port pairs separated by commas in the host component of the URI.

In either format, a single host name can translate to multiple network addresses. A common example of this is a host that has both an IPv4 and an IPv6 address.

When multiple hosts are specified, or when a single host name is translated to multiple addresses, all the hosts and addresses will be tried in order, until one succeeds. If none of the hosts can be reached, the connection fails. If a connection is established successfully, but authentication fails, the remaining hosts in the list are not tried.

If a password file is used, you can have different passwords for different hosts. All the other connection options are the same for every host in the list; it is not possible to e.g., specify different usernames for different hosts.

34.1.2. Parameter Key Words

The currently recognized parameter key words are:

host

Name of host to connect to. If a host name looks like an absolute path name, it specifies Unix-domain communication rather than TCP/IP communication; the value is the name of the directory in which the socket file is stored. (On Unix, an absolute path name begins with a slash. On Windows, paths starting with drive letters are also recognized.) If the host name starts with @, it is taken as a Unix-domain socket in the abstract namespace (currently supported on Linux and Windows). The default behavior when host is not specified, or is empty, is to connect to a Unix-domain socket in /tmp (or whatever socket directory was specified when PostgreSQL was built). On Windows and on machines without Unix-domain sockets, the default is to connect to localhost.

A comma-separated list of host names is also accepted, in which case each host name in the list is tried in order; an empty item in the list selects the default behavior as explained above. See Section 34.1.1.3 for details.

hostaddr

Numeric IP address of host to connect to. This should be in the standard IPv4 address format, e.g., 172.28.40.9. If your machine supports IPv6, you can also use those addresses. TCP/IP communication is always used when a nonempty string is specified for this parameter. If this parameter is not specified, the value of host will be looked up to find the corresponding IP address — or, if host specifies an IP address, that value will be used directly.

Using hostaddr allows the application to avoid a host name look-up, which might be important in applications with time constraints. However, a host name is required for GSSAPI or SSPI authentication methods, as well as for verify-full SSL certificate verification. The following rules are used:

  • If host is specified without hostaddr, a host name lookup occurs. (When using PQconnectPoll, the lookup occurs when PQconnectPoll first considers this host name, and it may cause PQconnectPoll to block for a significant amount of time.)

  • If hostaddr is specified without host, the value for hostaddr gives the server network address. The connection attempt will fail if the authentication method requires a host name.

  • If both host and hostaddr are specified, the value for hostaddr gives the server network address. The value for host is ignored unless the authentication method requires it, in which case it will be used as the host name.

Note that authentication is likely to fail if host is not the name of the server at network address hostaddr. Also, when both host and hostaddr are specified, host is used to identify the connection in a password file (see Section 34.16).

A comma-separated list of hostaddr values is also accepted, in which case each host in the list is tried in order. An empty item in the list causes the corresponding host name to be used, or the default host name if that is empty as well. See Section 34.1.1.3 for details.

Without either a host name or host address, libpq will connect using a local Unix-domain socket; or on Windows and on machines without Unix-domain sockets, it will attempt to connect to localhost.

port

Port number to connect to at the server host, or socket file name extension for Unix-domain connections. If multiple hosts were given in the host or hostaddr parameters, this parameter may specify a comma-separated list of ports of the same length as the host list, or it may specify a single port number to be used for all hosts. An empty string, or an empty item in a comma-separated list, specifies the default port number established when PostgreSQL was built.

dbname

The database name. Defaults to be the same as the user name. In certain contexts, the value is checked for extended formats; see Section 34.1.1 for more details on those.

user

PostgreSQL user name to connect as. Defaults to be the same as the operating system name of the user running the application.

password

Password to be used if the server demands password authentication.

passfile

Specifies the name of the file used to store passwords (see Section 34.16). Defaults to ~/.pgpass, or %APPDATA%\postgresql\pgpass.conf on Microsoft Windows. (No error is reported if this file does not exist.)

channel_binding

This option controls the client's use of channel binding. A setting of require means that the connection must employ channel binding, prefer means that the client will choose channel binding if available, and disable prevents the use of channel binding. The default is prefer if PostgreSQL is compiled with SSL support; otherwise the default is disable.

Channel binding is a method for the server to authenticate itself to the client. It is only supported over SSL connections with PostgreSQL 11 or later servers using the SCRAM authentication method.

connect_timeout

Maximum time to wait while connecting, in seconds (write as a decimal integer, e.g., 10). Zero, negative, or not specified means wait indefinitely. The minimum allowed timeout is 2 seconds, therefore a value of 1 is interpreted as 2. This timeout applies separately to each host name or IP address. For example, if you specify two hosts and connect_timeout is 5, each host will time out if no connection is made within 5 seconds, so the total time spent waiting for a connection might be up to 10 seconds.

client_encoding

This sets the client_encoding configuration parameter for this connection. In addition to the values accepted by the corresponding server option, you can use auto to determine the right encoding from the current locale in the client (LC_CTYPE environment variable on Unix systems).

options

Specifies command-line options to send to the server at connection start. For example, setting this to -c geqo=off sets the session's value of the geqo parameter to off. Spaces within this string are considered to separate command-line arguments, unless escaped with a backslash (\); write \\ to represent a literal backslash. For a detailed discussion of the available options, consult Chapter 20.

application_name

Specifies a value for the application_name configuration parameter.

fallback_application_name

Specifies a fallback value for the application_name configuration parameter. This value will be used if no value has been given for application_name via a connection parameter or the PGAPPNAME environment variable. Specifying a fallback name is useful in generic utility programs that wish to set a default application name but allow it to be overridden by the user.

keepalives

Controls whether client-side TCP keepalives are used. The default value is 1, meaning on, but you can change this to 0, meaning off, if keepalives are not wanted. This parameter is ignored for connections made via a Unix-domain socket.

keepalives_idle

Controls the number of seconds of inactivity after which TCP should send a keepalive message to the server. A value of zero uses the system default. This parameter is ignored for connections made via a Unix-domain socket, or if keepalives are disabled. It is only supported on systems where TCP_KEEPIDLE or an equivalent socket option is available, and on Windows; on other systems, it has no effect.

keepalives_interval

Controls the number of seconds after which a TCP keepalive message that is not acknowledged by the server should be retransmitted. A value of zero uses the system default. This parameter is ignored for connections made via a Unix-domain socket, or if keepalives are disabled. It is only supported on systems where TCP_KEEPINTVL or an equivalent socket option is available, and on Windows; on other systems, it has no effect.

keepalives_count

Controls the number of TCP keepalives that can be lost before the client's connection to the server is considered dead. A value of zero uses the system default. This parameter is ignored for connections made via a Unix-domain socket, or if keepalives are disabled. It is only supported on systems where TCP_KEEPCNT or an equivalent socket option is available; on other systems, it has no effect.

tcp_user_timeout

Controls the number of milliseconds that transmitted data may remain unacknowledged before a connection is forcibly closed. A value of zero uses the system default. This parameter is ignored for connections made via a Unix-domain socket. It is only supported on systems where TCP_USER_TIMEOUT is available; on other systems, it has no effect.

tty

Ignored (formerly, this specified where to send server debug output).

replication

This option determines whether the connection should use the replication protocol instead of the normal protocol. This is what PostgreSQL replication connections as well as tools such as pg_basebackup use internally, but it can also be used by third-party applications. For a description of the replication protocol, consult Section 53.4.

The following values, which are case-insensitive, are supported:

true, on, yes, 1

The connection goes into physical replication mode.

database

The connection goes into logical replication mode, connecting to the database specified in the dbname parameter.

false, off, no, 0

The connection is a regular one, which is the default behavior.

In physical or logical replication mode, only the simple query protocol can be used.

gssencmode

This option determines whether or with what priority a secure GSS TCP/IP connection will be negotiated with the server. There are three modes:

disable

only try a non-GSSAPI-encrypted connection

prefer (default)

if there are GSSAPI credentials present (i.e., in a credentials cache), first try a GSSAPI-encrypted connection; if that fails or there are no credentials, try a non-GSSAPI-encrypted connection. This is the default when PostgreSQL has been compiled with GSSAPI support.

require

only try a GSSAPI-encrypted connection

gssencmode is ignored for Unix domain socket communication. If PostgreSQL is compiled without GSSAPI support, using the require option will cause an error, while prefer will be accepted but libpq will not actually attempt a GSSAPI-encrypted connection.

sslmode

This option determines whether or with what priority a secure SSL TCP/IP connection will be negotiated with the server. There are six modes:

disable

only try a non-SSL connection

allow

first try a non-SSL connection; if that fails, try an SSL connection

prefer (default)

first try an SSL connection; if that fails, try a non-SSL connection

require

only try an SSL connection. If a root CA file is present, verify the certificate in the same way as if verify-ca was specified

verify-ca

only try an SSL connection, and verify that the server certificate is issued by a trusted certificate authority (CA)

verify-full

only try an SSL connection, verify that the server certificate is issued by a trusted CA and that the requested server host name matches that in the certificate

See Section 34.19 for a detailed description of how these options work.

sslmode is ignored for Unix domain socket communication. If PostgreSQL is compiled without SSL support, using options require, verify-ca, or verify-full will cause an error, while options allow and prefer will be accepted but libpq will not actually attempt an SSL connection.

Note that if GSSAPI encryption is possible, that will be used in preference to SSL encryption, regardless of the value of sslmode. To force use of SSL encryption in an environment that has working GSSAPI infrastructure (such as a Kerberos server), also set gssencmode to disable.

requiressl

This option is deprecated in favor of the sslmode setting.

If set to 1, an SSL connection to the server is required (this is equivalent to sslmode require). libpq will then refuse to connect if the server does not accept an SSL connection. If set to 0 (default), libpq will negotiate the connection type with the server (equivalent to sslmode prefer). This option is only available if PostgreSQL is compiled with SSL support.

sslcompression

If set to 1, data sent over SSL connections will be compressed. If set to 0, compression will be disabled. The default is 0. This parameter is ignored if a connection without SSL is made.

SSL compression is nowadays considered insecure and its use is no longer recommended. OpenSSL 1.1.0 disables compression by default, and many operating system distributions disable it in prior versions as well, so setting this parameter to on will not have any effect if the server does not accept compression. PostgreSQL 14 disables compression completely in the backend.

If security is not a primary concern, compression can improve throughput if the network is the bottleneck. Disabling compression can improve response time and throughput if CPU performance is the limiting factor.

sslcert

This parameter specifies the file name of the client SSL certificate, replacing the default ~/.postgresql/postgresql.crt. This parameter is ignored if an SSL connection is not made.

sslkey

This parameter specifies the location for the secret key used for the client certificate. It can either specify a file name that will be used instead of the default ~/.postgresql/postgresql.key, or it can specify a key obtained from an external “engine” (engines are OpenSSL loadable modules). An external engine specification should consist of a colon-separated engine name and an engine-specific key identifier. This parameter is ignored if an SSL connection is not made.

sslpassword

This parameter specifies the password for the secret key specified in sslkey, allowing client certificate private keys to be stored in encrypted form on disk even when interactive passphrase input is not practical.

Specifying this parameter with any non-empty value suppresses the Enter PEM pass phrase: prompt that OpenSSL will emit by default when an encrypted client certificate key is provided to libpq.

If the key is not encrypted this parameter is ignored. The parameter has no effect on keys specified by OpenSSL engines unless the engine uses the OpenSSL password callback mechanism for prompts.

There is no environment variable equivalent to this option, and no facility for looking it up in .pgpass. It can be used in a service file connection definition. Users with more sophisticated uses should consider using OpenSSL engines and tools like PKCS#11 or USB crypto offload devices.

sslrootcert

This parameter specifies the name of a file containing SSL certificate authority (CA) certificate(s). If the file exists, the server's certificate will be verified to be signed by one of these authorities. The default is ~/.postgresql/root.crt.

sslcrl

This parameter specifies the file name of the SSL server certificate revocation list (CRL). Certificates listed in this file, if it exists, will be rejected while attempting to authenticate the server's certificate. If neither sslcrl nor sslcrldir is set, this setting is taken as ~/.postgresql/root.crl.

sslcrldir

This parameter specifies the directory name of the SSL server certificate revocation list (CRL). Certificates listed in the files in this directory, if it exists, will be rejected while attempting to authenticate the server's certificate.

The directory needs to be prepared with the OpenSSL command openssl rehash or c_rehash. See its documentation for details.

Both sslcrl and sslcrldir can be specified together.

sslsni

If set to 1 (default), libpq sets the TLS extension “Server Name Indication” (SNI) on SSL-enabled connections. By setting this parameter to 0, this is turned off.

The Server Name Indication can be used by SSL-aware proxies to route connections without having to decrypt the SSL stream. (Note that this requires a proxy that is aware of the PostgreSQL protocol handshake, not just any SSL proxy.) However, SNI makes the destination host name appear in cleartext in the network traffic, so it might be undesirable in some cases.

requirepeer

This parameter specifies the operating-system user name of the server, for example requirepeer=postgres. When making a Unix-domain socket connection, if this parameter is set, the client checks at the beginning of the connection that the server process is running under the specified user name; if it is not, the connection is aborted with an error. This parameter can be used to provide server authentication similar to that available with SSL certificates on TCP/IP connections. (Note that if the Unix-domain socket is in /tmp or another publicly writable location, any user could start a server listening there. Use this parameter to ensure that you are connected to a server run by a trusted user.) This option is only supported on platforms for which the peer authentication method is implemented; see Section 21.9.

ssl_min_protocol_version

This parameter specifies the minimum SSL/TLS protocol version to allow for the connection. Valid values are TLSv1, TLSv1.1, TLSv1.2 and TLSv1.3. The supported protocols depend on the version of OpenSSL used, older versions not supporting the most modern protocol versions. If not specified, the default is TLSv1.2, which satisfies industry best practices as of this writing.

ssl_max_protocol_version

This parameter specifies the maximum SSL/TLS protocol version to allow for the connection. Valid values are TLSv1, TLSv1.1, TLSv1.2 and TLSv1.3. The supported protocols depend on the version of OpenSSL used, older versions not supporting the most modern protocol versions. If not set, this parameter is ignored and the connection will use the maximum bound defined by the backend, if set. Setting the maximum protocol version is mainly useful for testing or if some component has issues working with a newer protocol.

krbsrvname

Kerberos service name to use when authenticating with GSSAPI. This must match the service name specified in the server configuration for Kerberos authentication to succeed. (See also Section 21.6.) The default value is normally postgres, but that can be changed when building PostgreSQL via the --with-krb-srvnam option of configure. In most environments, this parameter never needs to be changed. Some Kerberos implementations might require a different service name, such as Microsoft Active Directory which requires the service name to be in upper case (POSTGRES).

gsslib

GSS library to use for GSSAPI authentication. Currently this is disregarded except on Windows builds that include both GSSAPI and SSPI support. In that case, set this to gssapi to cause libpq to use the GSSAPI library for authentication instead of the default SSPI.

service

Service name to use for additional parameters. It specifies a service name in pg_service.conf that holds additional connection parameters. This allows applications to specify only a service name so connection parameters can be centrally maintained. See Section 34.17.

target_session_attrs

This option determines whether the session must have certain properties to be acceptable. It's typically used in combination with multiple host names to select the first acceptable alternative among several hosts. There are six modes:

any (default)

any successful connection is acceptable

read-write

session must accept read-write transactions by default (that is, the server must not be in hot standby mode and the default_transaction_read_only parameter must be off)

read-only

session must not accept read-write transactions by default (the converse)

primary

server must not be in hot standby mode

standby

server must be in hot standby mode

prefer-standby

first try to find a standby server, but if none of the listed hosts is a standby server, try again in any mode

Prev | Up | Next
Chapter 34. libpq — C Library | Home | 34.2. Connection Status Functions


34.2. Connection Status Functions
Prev Up Chapter 34. libpq — C Library Home Next

34.2. Connection Status Functions

These functions can be used to interrogate the status of an existing database connection object.

Tip

libpq application programmers should be careful to maintain the PGconn abstraction. Use the accessor functions described below to get at the contents of PGconn. Reference to internal PGconn fields using libpq-int.h is not recommended because they are subject to change in the future.

The following functions return parameter values established at connection. These values are fixed for the life of the connection. If a multi-host connection string is used, the values of PQhost, PQport, and PQpass can change if a new connection is established using the same PGconn object. Other values are fixed for the lifetime of the PGconn object.

PQdb

Returns the database name of the connection.

char *PQdb(const PGconn *conn);

PQuser

Returns the user name of the connection.

char *PQuser(const PGconn *conn);

PQpass

Returns the password of the connection.

char *PQpass(const PGconn *conn);

PQpass will return either the password specified in the connection parameters, or if there was none and the password was obtained from the password file, it will return that. In the latter case, if multiple hosts were specified in the connection parameters, it is not possible to rely on the result of PQpass until the connection is established. The status of the connection can be checked using the function PQstatus.

PQhost

Returns the server host name of the active connection. This can be a host name, an IP address, or a directory path if the connection is via Unix socket. (The path case can be distinguished because it will always be an absolute path, beginning with /.)

char *PQhost(const PGconn *conn);

If the connection parameters specified both host and hostaddr, then PQhost will return the host information. If only hostaddr was specified, then that is returned. If multiple hosts were specified in the connection parameters, PQhost returns the host actually connected to.

PQhost returns NULL if the conn argument is NULL. Otherwise, if there is an error producing the host information (perhaps if the connection has not been fully established or there was an error), it returns an empty string.

If multiple hosts were specified in the connection parameters, it is not possible to rely on the result of PQhost until the connection is established. The status of the connection can be checked using the function PQstatus.

PQhostaddr

Returns the server IP address of the active connection. This can be the address that a host name resolved to, or an IP address provided through the hostaddr parameter.

char *PQhostaddr(const PGconn *conn);

PQhostaddr returns NULL if the conn argument is NULL. Otherwise, if there is an error producing the host information (perhaps if the connection has not been fully established or there was an error), it returns an empty string.

PQport

Returns the port of the active connection.

char *PQport(const PGconn *conn);

If multiple ports were specified in the connection parameters, PQport returns the port actually connected to.

PQport returns NULL if the conn argument is NULL. Otherwise, if there is an error producing the port information (perhaps if the connection has not been fully established or there was an error), it returns an empty string.

If multiple ports were specified in the connection parameters, it is not possible to rely on the result of PQport until the connection is established. The status of the connection can be checked using the function PQstatus.

PQtty

This function no longer does anything, but it remains for backwards compatibility. The function always return an empty string, or NULL if the conn argument is NULL.

char *PQtty(const PGconn *conn);

PQoptions

Returns the command-line options passed in the connection request.

char *PQoptions(const PGconn *conn);

The following functions return status data that can change as operations are executed on the PGconn object.

PQstatus

Returns the status of the connection.

ConnStatusType PQstatus(const PGconn *conn);

The status can be one of a number of values. However, only two of these are seen outside of an asynchronous connection procedure: CONNECTION_OK and CONNECTION_BAD. A good connection to the database has the status CONNECTION_OK. A failed connection attempt is signaled by status CONNECTION_BAD. Ordinarily, an OK status will remain so until PQfinish, but a communications failure might result in the status changing to CONNECTION_BAD prematurely. In that case the application could try to recover by calling PQreset.

See the entry for PQconnectStartParams, PQconnectStart and PQconnectPoll with regards to other status codes that might be returned.

PQtransactionStatus

Returns the current in-transaction status of the server.

PGTransactionStatusType PQtransactionStatus(const PGconn *conn);

The status can be PQTRANS_IDLE (currently idle), PQTRANS_ACTIVE (a command is in progress), PQTRANS_INTRANS (idle, in a valid transaction block), or PQTRANS_INERROR (idle, in a failed transaction block). PQTRANS_UNKNOWN is reported if the connection is bad. PQTRANS_ACTIVE is reported only when a query has been sent to the server and not yet completed.

PQparameterStatus

Looks up a current parameter setting of the server.

const char *PQparameterStatus(const PGconn *conn, const char *paramName);

Certain parameter values are reported by the server automatically at connection startup or whenever their values change. PQparameterStatus can be used to interrogate these settings. It returns the current value of a parameter if known, or NULL if the parameter is not known.

Parameters reported as of the current release include server_version, server_encoding, client_encoding, application_name, default_transaction_read_only, in_hot_standby, is_superuser, session_authorization, DateStyle, IntervalStyle, TimeZone, integer_datetimes, and standard_conforming_strings. (server_encoding, TimeZone, and integer_datetimes were not reported by releases before 8.0; standard_conforming_strings was not reported by releases before 8.1; IntervalStyle was not reported by releases before 8.4; application_name was not reported by releases before 9.0; default_transaction_read_only and in_hot_standby were not reported by releases before 14.) Note that server_version, server_encoding and integer_datetimes cannot change after startup.

If no value for standard_conforming_strings is reported, applications can assume it is off, that is, backslashes are treated as escapes in string literals. Also, the presence of this parameter can be taken as an indication that the escape string syntax (E'...') is accepted.

Although the returned pointer is declared const, it in fact points to mutable storage associated with the PGconn structure. It is unwise to assume the pointer will remain valid across queries.

PQprotocolVersion

Interrogates the frontend/backend protocol being used.

int PQprotocolVersion(const PGconn *conn);

Applications might wish to use this function to determine whether certain features are supported. Currently, the possible values are 3 (3.0 protocol), or zero (connection bad). The protocol version will not change after connection startup is complete, but it could theoretically change during a connection reset. The 3.0 protocol is supported by PostgreSQL server versions 7.4 and above.

PQserverVersion

Returns an integer representing the server version.

int PQserverVersion(const PGconn *conn);

Applications might use this function to determine the version of the database server they are connected to. The result is formed by multiplying the server's major version number by 10000 and adding the minor version number. For example, version 10.1 will be returned as 100001, and version 11.0 will be returned as 110000. Zero is returned if the connection is bad.

Prior to major version 10, PostgreSQL used three-part version numbers in which the first two parts together represented the major version. For those versions, PQserverVersion uses two digits for each part; for example version 9.1.5 will be returned as 90105, and version 9.2.0 will be returned as 90200.

Therefore, for purposes of determining feature compatibility, applications should divide the result of PQserverVersion by 100 not 10000 to determine a logical major version number. In all release series, only the last two digits differ between minor releases (bug-fix releases).

PQerrorMessage

Returns the error message most recently generated by an operation on the connection.

char *PQerrorMessage(const PGconn *conn);

Nearly all libpq functions will set a message for PQerrorMessage if they fail. Note that by libpq convention, a nonempty PQerrorMessage result can consist of multiple lines, and will include a trailing newline. The caller should not free the result directly. It will be freed when the associated PGconn handle is passed to PQfinish. The result string should not be expected to remain the same across operations on the PGconn structure.

PQsocket

Obtains the file descriptor number of the connection socket to the server. A valid descriptor will be greater than or equal to 0; a result of -1 indicates that no server connection is currently open. (This will not change during normal operation, but could change during connection setup or reset.)

int PQsocket(const PGconn *conn);

PQbackendPID

Returns the process ID (PID) of the backend process handling this connection.

int PQbackendPID(const PGconn *conn);

The backend PID is useful for debugging purposes and for comparison to NOTIFY messages (which include the PID of the notifying backend process). Note that the PID belongs to a process executing on the database server host, not the local host!

PQconnectionNeedsPassword

Returns true (1) if the connection authentication method required a password, but none was available. Returns false (0) if not.

int PQconnectionNeedsPassword(const PGconn *conn);

This function can be applied after a failed connection attempt to decide whether to prompt the user for a password.

PQconnectionUsedPassword

Returns true (1) if the connection authentication method used a password. Returns false (0) if not.

int PQconnectionUsedPassword(const PGconn *conn);

This function can be applied after either a failed or successful connection attempt to detect whether the server demanded a password.

The following functions return information related to SSL. This information usually doesn't change after a connection is established.

PQsslInUse

Returns true (1) if the connection uses SSL, false (0) if not.

int PQsslInUse(const PGconn *conn);

PQsslAttribute

Returns SSL-related information about the connection.

const char *PQsslAttribute(const PGconn *conn, const char *attribute_name);

The list of available attributes varies depending on the SSL library being used, and the type of connection. If an attribute is not available, returns NULL.

The following attributes are commonly available:

library

Name of the SSL implementation in use. (Currently, only "OpenSSL" is implemented)

protocol

SSL/TLS version in use. Common values are "TLSv1", "TLSv1.1" and "TLSv1.2", but an implementation may return other strings if some other protocol is used.

key_bits

Number of key bits used by the encryption algorithm.

cipher

A short name of the ciphersuite used, e.g., "DHE-RSA-DES-CBC3-SHA". The names are specific to each SSL implementation.

compression

If SSL compression is in use, returns the name of the compression algorithm, or "on" if compression is used but the algorithm is not known. If compression is not in use, returns "off".

PQsslAttributeNames

Return an array of SSL attribute names available. The array is terminated by a NULL pointer.

const char * const * PQsslAttributeNames(const PGconn *conn);

PQsslStruct

Return a pointer to an SSL-implementation-specific object describing the connection.

void *PQsslStruct(const PGconn *conn, const char *struct_name);

The struct(s) available depend on the SSL implementation in use. For OpenSSL, there is one struct, available under the name "OpenSSL", and it returns a pointer to the OpenSSL SSL struct. To use this function, code along the following lines could be used:

#include <libpq-fe.h>
#include <openssl/ssl.h>

...

    SSL *ssl;

    dbconn = PQconnectdb(...);
    ...

    ssl = PQsslStruct(dbconn, "OpenSSL");
    if (ssl)
    {
        /* use OpenSSL functions to access ssl */
    }

This structure can be used to verify encryption levels, check server certificates, and more. Refer to the OpenSSL documentation for information about this structure.

PQgetssl

Returns the SSL structure used in the connection, or null if SSL is not in use.

void *PQgetssl(const PGconn *conn);

This function is equivalent to PQsslStruct(conn, "OpenSSL"). It should not be used in new applications, because the returned struct is specific to OpenSSL and will not be available if another SSL implementation is used. To check if a connection uses SSL, call PQsslInUse instead, and for more details about the connection, use PQsslAttribute.

Prev | Up | Next
34.1. Database Connection Control Functions | Home | 34.3. Command Execution Functions


34.3. Command Execution Functions
Prev Up Chapter 34. libpq — C Library Home Next

34.3. Command Execution Functions

34.3.1. Main Functions
34.3.2. Retrieving Query Result Information
34.3.3. Retrieving Other Result Information
34.3.4. Escaping Strings for Inclusion in SQL Commands

Once a connection to a database server has been successfully established, the functions described here are used to perform SQL queries and commands.

34.3.1. Main Functions

PQexec

Submits a command to the server and waits for the result.

PGresult *PQexec(PGconn *conn, const char *command);

Returns a PGresult pointer or possibly a null pointer. A non-null pointer will generally be returned except in out-of-memory conditions or serious errors such as inability to send the command to the server. The PQresultStatus function should be called to check the return value for any errors (including the value of a null pointer, in which case it will return PGRES_FATAL_ERROR). Use PQerrorMessage to get more information about such errors.

The command string can include multiple SQL commands (separated by semicolons). Multiple queries sent in a single PQexec call are processed in a single transaction, unless there are explicit BEGIN/COMMIT commands included in the query string to divide it into multiple transactions. (See Section 53.2.2.1 for more details about how the server handles multi-query strings.) Note however that the returned PGresult structure describes only the result of the last command executed from the string. Should one of the commands fail, processing of the string stops with it and the returned PGresult describes the error condition.

PQexecParams

Submits a command to the server and waits for the result, with the ability to pass parameters separately from the SQL command text.

PGresult *PQexecParams(PGconn *conn,
                       const char *command,
                       int nParams,
                       const Oid *paramTypes,
                       const char * const *paramValues,
                       const int *paramLengths,
                       const int *paramFormats,
                       int resultFormat);

PQexecParams is like PQexec, but offers additional functionality: parameter values can be specified separately from the command string proper, and query results can be requested in either text or binary format.

The function arguments are:

conn

The connection object to send the command through.

command

The SQL command string to be executed. If parameters are used, they are referred to in the command string as $1, $2, etc.

nParams

The number of parameters supplied; it is the length of the arrays paramTypes[], paramValues[], paramLengths[], and paramFormats[]. (The array pointers can be NULL when nParams is zero.)

paramTypes[]

Specifies, by OID, the data types to be assigned to the parameter symbols. If paramTypes is NULL, or any particular element in the array is zero, the server infers a data type for the parameter symbol in the same way it would do for an untyped literal string.

paramValues[]

Specifies the actual values of the parameters. A null pointer in this array means the corresponding parameter is null; otherwise the pointer points to a zero-terminated text string (for text format) or binary data in the format expected by the server (for binary format).

paramLengths[]

Specifies the actual data lengths of binary-format parameters. It is ignored for null parameters and text-format parameters. The array pointer can be null when there are no binary parameters.

paramFormats[]

Specifies whether parameters are text (put a zero in the array entry for the corresponding parameter) or binary (put a one in the array entry for the corresponding parameter). If the array pointer is null then all parameters are presumed to be text strings.

Values passed in binary format require knowledge of the internal representation expected by the backend. For example, integers must be passed in network byte order. Passing numeric values requires knowledge of the server storage format, as implemented in src/backend/utils/adt/numeric.c::numeric_send() and src/backend/utils/adt/numeric.c::numeric_recv().

resultFormat

Specify zero to obtain results in text format, or one to obtain results in binary format. (There is not currently a provision to obtain different result columns in different formats, although that is possible in the underlying protocol.)

The primary advantage of PQexecParams over PQexec is that parameter values can be separated from the command string, thus avoiding the need for tedious and error-prone quoting and escaping.

Unlike PQexec, PQexecParams allows at most one SQL command in the given string. (There can be semicolons in it, but not more than one nonempty command.) This is a limitation of the underlying protocol, but has some usefulness as an extra defense against SQL-injection attacks.

Tip

Specifying parameter types via OIDs is tedious, particularly if you prefer not to hard-wire particular OID values into your program. However, you can avoid doing so even in cases where the server by itself cannot determine the type of the parameter, or chooses a different type than you want. In the SQL command text, attach an explicit cast to the parameter symbol to show what data type you will send. For example:

SELECT * FROM mytable WHERE x = $1::bigint;

This forces parameter $1 to be treated as bigint, whereas by default it would be assigned the same type as x. Forcing the parameter type decision, either this way or by specifying a numeric type OID, is strongly recommended when sending parameter values in binary format, because binary format has less redundancy than text format and so there is less chance that the server will detect a type mismatch mistake for you.

PQprepare

Submits a request to create a prepared statement with the given parameters, and waits for completion.

PGresult *PQprepare(PGconn *conn,
                    const char *stmtName,
                    const char *query,
                    int nParams,
                    const Oid *paramTypes);

PQprepare creates a prepared statement for later execution with PQexecPrepared. This feature allows commands to be executed repeatedly without being parsed and planned each time; see PREPARE for details.

The function creates a prepared statement named stmtName from the query string, which must contain a single SQL command. stmtName can be "" to create an unnamed statement, in which case any pre-existing unnamed statement is automatically replaced; otherwise it is an error if the statement name is already defined in the current session. If any parameters are used, they are referred to in the query as $1, $2, etc. nParams is the number of parameters for which types are pre-specified in the array paramTypes[]. (The array pointer can be NULL when nParams is zero.) paramTypes[] specifies, by OID, the data types to be assigned to the parameter symbols. If paramTypes is NULL, or any particular element in the array is zero, the server assigns a data type to the parameter symbol in the same way it would do for an untyped literal string. Also, the query can use parameter symbols with numbers higher than nParams; data types will be inferred for these symbols as well. (See PQdescribePrepared for a means to find out what data types were inferred.)

As with PQexec, the result is normally a PGresult object whose contents indicate server-side success or failure. A null result indicates out-of-memory or inability to send the command at all. Use PQerrorMessage to get more information about such errors.

Prepared statements for use with PQexecPrepared can also be created by executing SQL PREPARE statements. Also, although there is no libpq function for deleting a prepared statement, the SQL DEALLOCATE statement can be used for that purpose.

PQexecPrepared

Sends a request to execute a prepared statement with given parameters, and waits for the result.

PGresult *PQexecPrepared(PGconn *conn,
                         const char *stmtName,
                         int nParams,
                         const char * const *paramValues,
                         const int *paramLengths,
                         const int *paramFormats,
                         int resultFormat);

PQexecPrepared is like PQexecParams, but the command to be executed is specified by naming a previously-prepared statement, instead of giving a query string. This feature allows commands that will be used repeatedly to be parsed and planned just once, rather than each time they are executed. The statement must have been prepared previously in the current session.

The parameters are identical to PQexecParams, except that the name of a prepared statement is given instead of a query string, and the paramTypes[] parameter is not present (it is not needed since the prepared statement's parameter types were determined when it was created).

PQdescribePrepared

Submits a request to obtain information about the specified prepared statement, and waits for completion.

PGresult *PQdescribePrepared(PGconn *conn, const char *stmtName);

PQdescribePrepared allows an application to obtain information about a previously prepared statement.

stmtName can be "" or NULL to reference the unnamed statement, otherwise it must be the name of an existing prepared statement. On success, a PGresult with status PGRES_COMMAND_OK is returned. The functions PQnparams and PQparamtype can be applied to this PGresult to obtain information about the parameters of the prepared statement, and the functions PQnfields, PQfname, PQftype, etc provide information about the result columns (if any) of the statement.

PQdescribePortal

Submits a request to obtain information about the specified portal, and waits for completion.

PGresult *PQdescribePortal(PGconn *conn, const char *portalName);

PQdescribePortal allows an application to obtain information about a previously created portal. (libpq does not provide any direct access to portals, but you can use this function to inspect the properties of a cursor created with a DECLARE CURSOR SQL command.)

portalName can be "" or NULL to reference the unnamed portal, otherwise it must be the name of an existing portal. On success, a PGresult with status PGRES_COMMAND_OK is returned. The functions PQnfields, PQfname, PQftype, etc can be applied to the PGresult to obtain information about the result columns (if any) of the portal.

The PGresult structure encapsulates the result returned by the server. libpq application programmers should be careful to maintain the PGresult abstraction. Use the accessor functions below to get at the contents of PGresult. Avoid directly referencing the fields of the PGresult structure because they are subject to change in the future.

PQresultStatus

Returns the result status of the command.

ExecStatusType PQresultStatus(const PGresult *res);

PQresultStatus can return one of the following values:

PGRES_EMPTY_QUERY

The string sent to the server was empty.

PGRES_COMMAND_OK

Successful completion of a command returning no data.

PGRES_TUPLES_OK

Successful completion of a command returning data (such as a SELECT or SHOW).

PGRES_COPY_OUT

Copy Out (from server) data transfer started.

PGRES_COPY_IN

Copy In (to server) data transfer started.

PGRES_BAD_RESPONSE

The server's response was not understood.

PGRES_NONFATAL_ERROR

A nonfatal error (a notice or warning) occurred.

PGRES_FATAL_ERROR

A fatal error occurred.

PGRES_COPY_BOTH

Copy In/Out (to and from server) data transfer started. This feature is currently used only for streaming replication, so this status should not occur in ordinary applications.

PGRES_SINGLE_TUPLE

The PGresult contains a single result tuple from the current command. This status occurs only when single-row mode has been selected for the query (see Section 34.6).

PGRES_PIPELINE_SYNC

The PGresult represents a synchronization point in pipeline mode, requested by PQpipelineSync. This status occurs only when pipeline mode has been selected.

PGRES_PIPELINE_ABORTED

The PGresult represents a pipeline that has received an error from the server. PQgetResult must be called repeatedly, and each time it will return this status code until the end of the current pipeline, at which point it will return PGRES_PIPELINE_SYNC and normal processing can resume.

If the result status is PGRES_TUPLES_OK or PGRES_SINGLE_TUPLE, then the functions described below can be used to retrieve the rows returned by the query. Note that a SELECT command that happens to retrieve zero rows still shows PGRES_TUPLES_OK. PGRES_COMMAND_OK is for commands that can never return rows (INSERT or UPDATE without a RETURNING clause, etc.). A response of PGRES_EMPTY_QUERY might indicate a bug in the client software.

A result of status PGRES_NONFATAL_ERROR will never be returned directly by PQexec or other query execution functions; results of this kind are instead passed to the notice processor (see Section 34.13).

PQresStatus

Converts the enumerated type returned by PQresultStatus into a string constant describing the status code. The caller should not free the result.

char *PQresStatus(ExecStatusType status);

PQresultErrorMessage

Returns the error message associated with the command, or an empty string if there was no error.

char *PQresultErrorMessage(const PGresult *res);

If there was an error, the returned string will include a trailing newline. The caller should not free the result directly. It will be freed when the associated PGresult handle is passed to PQclear.

Immediately following a PQexec or PQgetResult call, PQerrorMessage (on the connection) will return the same string as PQresultErrorMessage (on the result). However, a PGresult will retain its error message until destroyed, whereas the connection's error message will change when subsequent operations are done. Use PQresultErrorMessage when you want to know the status associated with a particular PGresult; use PQerrorMessage when you want to know the status from the latest operation on the connection.

PQresultVerboseErrorMessage

Returns a reformatted version of the error message associated with a PGresult object.

char *PQresultVerboseErrorMessage(const PGresult *res,
                                  PGVerbosity verbosity,
                                  PGContextVisibility show_context);

In some situations a client might wish to obtain a more detailed version of a previously-reported error. PQresultVerboseErrorMessage addresses this need by computing the message that would have been produced by PQresultErrorMessage if the specified verbosity settings had been in effect for the connection when the given PGresult was generated. If the PGresult is not an error result, “PGresult is not an error result” is reported instead. The returned string includes a trailing newline.

Unlike most other functions for extracting data from a PGresult, the result of this function is a freshly allocated string. The caller must free it using PQfreemem() when the string is no longer needed.

A NULL return is possible if there is insufficient memory.

PQresultErrorField

Returns an individual field of an error report.

char *PQresultErrorField(const PGresult *res, int fieldcode);

fieldcode is an error field identifier; see the symbols listed below. NULL is returned if the PGresult is not an error or warning result, or does not include the specified field. Field values will normally not include a trailing newline. The caller should not free the result directly. It will be freed when the associated PGresult handle is passed to PQclear.

The following field codes are available:

PG_DIAG_SEVERITY

The severity; the field contents are ERROR, FATAL, or PANIC (in an error message), or WARNING, NOTICE, DEBUG, INFO, or LOG (in a notice message), or a localized translation of one of these. Always present.

PG_DIAG_SEVERITY_NONLOCALIZED

The severity; the field contents are ERROR, FATAL, or PANIC (in an error message), or WARNING, NOTICE, DEBUG, INFO, or LOG (in a notice message). This is identical to the PG_DIAG_SEVERITY field except that the contents are never localized. This is present only in reports generated by PostgreSQL versions 9.6 and later.

PG_DIAG_SQLSTATE

The SQLSTATE code for the error. The SQLSTATE code identifies the type of error that has occurred; it can be used by front-end applications to perform specific operations (such as error handling) in response to a particular database error. For a list of the possible SQLSTATE codes, see Appendix A. This field is not localizable, and is always present.

PG_DIAG_MESSAGE_PRIMARY

The primary human-readable error message (typically one line). Always present.

PG_DIAG_MESSAGE_DETAIL

Detail: an optional secondary error message carrying more detail about the problem. Might run to multiple lines.

PG_DIAG_MESSAGE_HINT

Hint: an optional suggestion what to do about the problem. This is intended to differ from detail in that it offers advice (potentially inappropriate) rather than hard facts. Might run to multiple lines.

PG_DIAG_STATEMENT_POSITION

A string containing a decimal integer indicating an error cursor position as an index into the original statement string. The first character has index 1, and positions are measured in characters not bytes.

PG_DIAG_INTERNAL_POSITION

This is defined the same as the PG_DIAG_STATEMENT_POSITION field, but it is used when the cursor position refers to an internally generated command rather than the one submitted by the client. The PG_DIAG_INTERNAL_QUERY field will always appear when this field appears.

PG_DIAG_INTERNAL_QUERY

The text of a failed internally-generated command. This could be, for example, an SQL query issued by a PL/pgSQL function.

PG_DIAG_CONTEXT

An indication of the context in which the error occurred. Presently this includes a call stack traceback of active procedural language functions and internally-generated queries. The trace is one entry per line, most recent first.

PG_DIAG_SCHEMA_NAME

If the error was associated with a specific database object, the name of the schema containing that object, if any.

PG_DIAG_TABLE_NAME

If the error was associated with a specific table, the name of the table. (Refer to the schema name field for the name of the table's schema.)

PG_DIAG_COLUMN_NAME

If the error was associated with a specific table column, the name of the column. (Refer to the schema and table name fields to identify the table.)

PG_DIAG_DATATYPE_NAME

If the error was associated with a specific data type, the name of the data type. (Refer to the schema name field for the name of the data type's schema.)

PG_DIAG_CONSTRAINT_NAME

If the error was associated with a specific constraint, the name of the constraint. Refer to fields listed above for the associated table or domain. (For this purpose, indexes are treated as constraints, even if they weren't created with constraint syntax.)

PG_DIAG_SOURCE_FILE

The file name of the source-code location where the error was reported.

PG_DIAG_SOURCE_LINE

The line number of the source-code location where the error was reported.

PG_DIAG_SOURCE_FUNCTION

The name of the source-code function reporting the error.

Note

The fields for schema name, table name, column name, data type name, and constraint name are supplied only for a limited number of error types; see Appendix A. Do not assume that the presence of any of these fields guarantees the presence of another field. Core error sources observe the interrelationships noted above, but user-defined functions may use these fields in other ways. In the same vein, do not assume that these fields denote contemporary objects in the current database.

The client is responsible for formatting displayed information to meet its needs; in particular it should break long lines as needed. Newline characters appearing in the error message fields should be treated as paragraph breaks, not line breaks.

Errors generated internally by libpq will have severity and primary message, but typically no other fields.

Note that error fields are only available from PGresult objects, not PGconn objects; there is no PQerrorField function.

PQclear

Frees the storage associated with a PGresult. Every command result should be freed via PQclear when it is no longer needed.

void PQclear(PGresult *res);

You can keep a PGresult object around for as long as you need it; it does not go away when you issue a new command, nor even if you close the connection. To get rid of it, you must call PQclear. Failure to do this will result in memory leaks in your application.

34.3.2. Retrieving Query Result Information

These functions are used to extract information from a PGresult object that represents a successful query result (that is, one that has status PGRES_TUPLES_OK or PGRES_SINGLE_TUPLE). They can also be used to extract information from a successful Describe operation: a Describe's result has all the same column information that actual execution of the query would provide, but it has zero rows. For objects with other status values, these functions will act as though the result has zero rows and zero columns.

PQntuples

Returns the number of rows (tuples) in the query result. (Note that PGresult objects are limited to no more than INT_MAX rows, so an int result is sufficient.)

int PQntuples(const PGresult *res);

PQnfields

Returns the number of columns (fields) in each row of the query result.

int PQnfields(const PGresult *res);

PQfname

Returns the column name associated with the given column number. Column numbers start at 0. The caller should not free the result directly. It will be freed when the associated PGresult handle is passed to PQclear.

char *PQfname(const PGresult *res,
              int column_number);

NULL is returned if the column number is out of range.

PQfnumber

Returns the column number associated with the given column name.

int PQfnumber(const PGresult *res,
              const char *column_name);

-1 is returned if the given name does not match any column.

The given name is treated like an identifier in an SQL command, that is, it is downcased unless double-quoted. For example, given a query result generated from the SQL command:

SELECT 1 AS FOO, 2 AS "BAR";

we would have the results:

PQfname(res, 0)              foo
PQfname(res, 1)              BAR
PQfnumber(res, "FOO")        0
PQfnumber(res, "foo")        0
PQfnumber(res, "BAR")        -1
PQfnumber(res, "\"BAR\"")    1

PQftable

Returns the OID of the table from which the given column was fetched. Column numbers start at 0.

Oid PQftable(const PGresult *res,
             int column_number);

InvalidOid is returned if the column number is out of range, or if the specified column is not a simple reference to a table column. You can query the system table pg_class to determine exactly which table is referenced.

The type Oid and the constant InvalidOid will be defined when you include the libpq header file. They will both be some integer type.

PQftablecol

Returns the column number (within its table) of the column making up the specified query result column. Query-result column numbers start at 0, but table columns have nonzero numbers.

int PQftablecol(const PGresult *res,
                int column_number);

Zero is returned if the column number is out of range, or if the specified column is not a simple reference to a table column.

PQfformat

Returns the format code indicating the format of the given column. Column numbers start at 0.

int PQfformat(const PGresult *res,
              int column_number);

Format code zero indicates textual data representation, while format code one indicates binary representation. (Other codes are reserved for future definition.)

PQftype

Returns the data type associated with the given column number. The integer returned is the internal OID number of the type. Column numbers start at 0.

Oid PQftype(const PGresult *res,
            int column_number);

You can query the system table pg_type to obtain the names and properties of the various data types. The OIDs of the built-in data types are defined in the file catalog/pg_type_d.h in the PostgreSQL installation's include directory.

PQfmod

Returns the type modifier of the column associated with the given column number. Column numbers start at 0.

int PQfmod(const PGresult *res,
           int column_number);

The interpretation of modifier values is type-specific; they typically indicate precision or size limits. The value -1 is used to indicate “no information available”. Most data types do not use modifiers, in which case the value is always -1.

PQfsize

Returns the size in bytes of the column associated with the given column number. Column numbers start at 0.

int PQfsize(const PGresult *res,
            int column_number);

PQfsize returns the space allocated for this column in a database row, in other words the size of the server's internal representation of the data type. (Accordingly, it is not really very useful to clients.) A negative value indicates the data type is variable-length.

PQbinaryTuples

Returns 1 if the PGresult contains binary data and 0 if it contains text data.

int PQbinaryTuples(const PGresult *res);

This function is deprecated (except for its use in connection with COPY), because it is possible for a single PGresult to contain text data in some columns and binary data in others. PQfformat is preferred. PQbinaryTuples returns 1 only if all columns of the result are binary (format 1).

PQgetvalue

Returns a single field value of one row of a PGresult. Row and column numbers start at 0. The caller should not free the result directly. It will be freed when the associated PGresult handle is passed to PQclear.

char *PQgetvalue(const PGresult *res,
                 int row_number,
                 int column_number);

For data in text format, the value returned by PQgetvalue is a null-terminated character string representation of the field value. For data in binary format, the value is in the binary representation determined by the data type's typsend and typreceive functions. (The value is actually followed by a zero byte in this case too, but that is not ordinarily useful, since the value is likely to contain embedded nulls.)

An empty string is returned if the field value is null. See PQgetisnull to distinguish null values from empty-string values.

The pointer returned by PQgetvalue points to storage that is part of the PGresult structure. One should not modify the data it points to, and one must explicitly copy the data into other storage if it is to be used past the lifetime of the PGresult structure itself.

PQgetisnull

Tests a field for a null value. Row and column numbers start at 0.

int PQgetisnull(const PGresult *res,
                int row_number,
                int column_number);

This function returns 1 if the field is null and 0 if it contains a non-null value. (Note that PQgetvalue will return an empty string, not a null pointer, for a null field.)

PQgetlength

Returns the actual length of a field value in bytes. Row and column numbers start at 0.

int PQgetlength(const PGresult *res,
                int row_number,
                int column_number);

This is the actual data length for the particular data value, that is, the size of the object pointed to by PQgetvalue. For text data format this is the same as strlen(). For binary format this is essential information. Note that one should not rely on PQfsize to obtain the actual data length.

PQnparams

Returns the number of parameters of a prepared statement.

int PQnparams(const PGresult *res);

This function is only useful when inspecting the result of PQdescribePrepared. For other types of queries it will return zero.

PQparamtype

Returns the data type of the indicated statement parameter. Parameter numbers start at 0.

Oid PQparamtype(const PGresult *res, int param_number);

This function is only useful when inspecting the result of PQdescribePrepared. For other types of queries it will return zero.

PQprint

Prints out all the rows and, optionally, the column names to the specified output stream.

void PQprint(FILE *fout,      /* output stream */
             const PGresult *res,
             const PQprintOpt *po);
typedef struct
{
    pqbool  header;      /* print output field headings and row count */
    pqbool  align;       /* fill align the fields */
    pqbool  standard;    /* old brain dead format */
    pqbool  html3;       /* output HTML tables */
    pqbool  expanded;    /* expand tables */
    pqbool  pager;       /* use pager for output if needed */
    char    *fieldSep;   /* field separator */
    char    *tableOpt;   /* attributes for HTML table element */
    char    *caption;    /* HTML table caption */
    char    **fieldName; /* null-terminated array of replacement field names */
} PQprintOpt;

This function was formerly used by psql to print query results, but this is no longer the case. Note that it assumes all the data is in text format.

34.3.3. Retrieving Other Result Information

These functions are used to extract other information from PGresult objects.

PQcmdStatus

Returns the command status tag from the SQL command that generated the PGresult.

char *PQcmdStatus(PGresult *res);

Commonly this is just the name of the command, but it might include additional data such as the number of rows processed. The caller should not free the result directly. It will be freed when the associated PGresult handle is passed to PQclear.

PQcmdTuples

Returns the number of rows affected by the SQL command.

char *PQcmdTuples(PGresult *res);

This function returns a string containing the number of rows affected by the SQL statement that generated the PGresult. This function can only be used following the execution of a SELECT, CREATE TABLE AS, INSERT, UPDATE, DELETE, MOVE, FETCH, or COPY statement, or an EXECUTE of a prepared query that contains an INSERT, UPDATE, or DELETE statement. If the command that generated the PGresult was anything else, PQcmdTuples returns an empty string. The caller should not free the return value directly. It will be freed when the associated PGresult handle is passed to PQclear.

PQoidValue

Returns the OID of the inserted row, if the SQL command was an INSERT that inserted exactly one row into a table that has OIDs, or a EXECUTE of a prepared query containing a suitable INSERT statement. Otherwise, this function returns InvalidOid. This function will also return InvalidOid if the table affected by the INSERT statement does not contain OIDs.

Oid PQoidValue(const PGresult *res);

PQoidStatus

This function is deprecated in favor of PQoidValue and is not thread-safe. It returns a string with the OID of the inserted row, while PQoidValue returns the OID value.

char *PQoidStatus(const PGresult *res);

34.3.4. Escaping Strings for Inclusion in SQL Commands

PQescapeLiteral

char *PQescapeLiteral(PGconn *conn, const char *str, size_t length);

PQescapeLiteral escapes a string for use within an SQL command. This is useful when inserting data values as literal constants in SQL commands. Certain characters (such as quotes and backslashes) must be escaped to prevent them from being interpreted specially by the SQL parser. PQescapeLiteral performs this operation.

PQescapeLiteral returns an escaped version of the str parameter in memory allocated with malloc(). This memory should be freed using PQfreemem() when the result is no longer needed. A terminating zero byte is not required, and should not be counted in length. (If a terminating zero byte is found before length bytes are processed, PQescapeLiteral stops at the zero; the behavior is thus rather like strncpy.) The return string has all special characters replaced so that they can be properly processed by the PostgreSQL string literal parser. A terminating zero byte is also added. The single quotes that must surround PostgreSQL string literals are included in the result string.

On error, PQescapeLiteral returns NULL and a suitable message is stored in the conn object.

Tip

It is especially important to do proper escaping when handling strings that were received from an untrustworthy source. Otherwise there is a security risk: you are vulnerable to “SQL injection” attacks wherein unwanted SQL commands are fed to your database.

Note that it is neither necessary nor correct to do escaping when a data value is passed as a separate parameter in PQexecParams or its sibling routines.

PQescapeIdentifier

char *PQescapeIdentifier(PGconn *conn, const char *str, size_t length);

PQescapeIdentifier escapes a string for use as an SQL identifier, such as a table, column, or function name. This is useful when a user-supplied identifier might contain special characters that would otherwise not be interpreted as part of the identifier by the SQL parser, or when the identifier might contain upper case characters whose case should be preserved.

PQescapeIdentifier returns a version of the str parameter escaped as an SQL identifier in memory allocated with malloc(). This memory must be freed using PQfreemem() when the result is no longer needed. A terminating zero byte is not required, and should not be counted in length. (If a terminating zero byte is found before length bytes are processed, PQescapeIdentifier stops at the zero; the behavior is thus rather like strncpy.) The return string has all special characters replaced so that it will be properly processed as an SQL identifier. A terminating zero byte is also added. The return string will also be surrounded by double quotes.

On error, PQescapeIdentifier returns NULL and a suitable message is stored in the conn object.

Tip

As with string literals, to prevent SQL injection attacks, SQL identifiers must be escaped when they are received from an untrustworthy source.

PQescapeStringConn

size_t PQescapeStringConn(PGconn *conn,
                          char *to, const char *from, size_t length,
                          int *error);

PQescapeStringConn escapes string literals, much like PQescapeLiteral. Unlike PQescapeLiteral, the caller is responsible for providing an appropriately sized buffer. Furthermore, PQescapeStringConn does not generate the single quotes that must surround PostgreSQL string literals; they should be provided in the SQL command that the result is inserted into. The parameter from points to the first character of the string that is to be escaped, and the length parameter gives the number of bytes in this string. A terminating zero byte is not required, and should not be counted in length. (If a terminating zero byte is found before length bytes are processed, PQescapeStringConn stops at the zero; the behavior is thus rather like strncpy.) to shall point to a buffer that is able to hold at least one more byte than twice the value of length, otherwise the behavior is undefined. Behavior is likewise undefined if the to and from strings overlap.

If the error parameter is not NULL, then *error is set to zero on success, nonzero on error. Presently the only possible error conditions involve invalid multibyte encoding in the source string. The output string is still generated on error, but it can be expected that the server will reject it as malformed. On error, a suitable message is stored in the conn object, whether or not error is NULL.

PQescapeStringConn returns the number of bytes written to to, not including the terminating zero byte.

PQescapeString

PQescapeString is an older, deprecated version of PQescapeStringConn.

size_t PQescapeString (char *to, const char *from, size_t length);

The only difference from PQescapeStringConn is that PQescapeString does not take PGconn or error parameters. Because of this, it cannot adjust its behavior depending on the connection properties (such as character encoding) and therefore it might give the wrong results. Also, it has no way to report error conditions.

PQescapeString can be used safely in client programs that work with only one PostgreSQL connection at a time (in this case it can find out what it needs to know “behind the scenes”). In other contexts it is a security hazard and should be avoided in favor of PQescapeStringConn.

PQescapeByteaConn

Escapes binary data for use within an SQL command with the type bytea. As with PQescapeStringConn, this is only used when inserting data directly into an SQL command string.

unsigned char *PQescapeByteaConn(PGconn *conn,
                                 const unsigned char *from,
                                 size_t from_length,
                                 size_t *to_length);

Certain byte values must be escaped when used as part of a bytea literal in an SQL statement. PQescapeByteaConn escapes bytes using either hex encoding or backslash escaping. See Section 8.4 for more information.

The from parameter points to the first byte of the string that is to be escaped, and the from_length parameter gives the number of bytes in this binary string. (A terminating zero byte is neither necessary nor counted.) The to_length parameter points to a variable that will hold the resultant escaped string length. This result string length includes the terminating zero byte of the result.

PQescapeByteaConn returns an escaped version of the from parameter binary string in memory allocated with malloc(). This memory should be freed using PQfreemem() when the result is no longer needed. The return string has all special characters replaced so that they can be properly processed by the PostgreSQL string literal parser, and the bytea input function. A terminating zero byte is also added. The single quotes that must surround PostgreSQL string literals are not part of the result string.

On error, a null pointer is returned, and a suitable error message is stored in the conn object. Currently, the only possible error is insufficient memory for the result string.

PQescapeBytea

PQescapeBytea is an older, deprecated version of PQescapeByteaConn.

unsigned char *PQescapeBytea(const unsigned char *from,
                             size_t from_length,
                             size_t *to_length);

The only difference from PQescapeByteaConn is that PQescapeBytea does not take a PGconn parameter. Because of this, PQescapeBytea can only be used safely in client programs that use a single PostgreSQL connection at a time (in this case it can find out what it needs to know “behind the scenes”). It might give the wrong results if used in programs that use multiple database connections (use PQescapeByteaConn in such cases).

PQunescapeBytea

Converts a string representation of binary data into binary data — the reverse of PQescapeBytea. This is needed when retrieving bytea data in text format, but not when retrieving it in binary format.

unsigned char *PQunescapeBytea(const unsigned char *from, size_t *to_length);

The from parameter points to a string such as might be returned by PQgetvalue when applied to a bytea column. PQunescapeBytea converts this string representation into its binary representation. It returns a pointer to a buffer allocated with malloc(), or NULL on error, and puts the size of the buffer in to_length. The result must be freed using PQfreemem when it is no longer needed.

This conversion is not exactly the inverse of PQescapeBytea, because the string is not expected to be “escaped” when received from PQgetvalue. In particular this means there is no need for string quoting considerations, and so no need for a PGconn parameter.

Prev | Up | Next
34.2. Connection Status Functions | Home | 34.4. Asynchronous Command Processing


34.4. Asynchronous Command Processing
Prev Up Chapter 34. libpq — C Library Home Next

34.4. Asynchronous Command Processing

The PQexec function is adequate for submitting commands in normal, synchronous applications. It has a few deficiencies, however, that can be of importance to some users:

  • PQexec waits for the command to be completed. The application might have other work to do (such as maintaining a user interface), in which case it won't want to block waiting for the response.

  • Since the execution of the client application is suspended while it waits for the result, it is hard for the application to decide that it would like to try to cancel the ongoing command. (It can be done from a signal handler, but not otherwise.)

  • PQexec can return only one PGresult structure. If the submitted command string contains multiple SQL commands, all but the last PGresult are discarded by PQexec.

  • PQexec always collects the command's entire result, buffering it in a single PGresult. While this simplifies error-handling logic for the application, it can be impractical for results containing many rows.

Applications that do not like these limitations can instead use the underlying functions that PQexec is built from: PQsendQuery and PQgetResult. There are also PQsendQueryParams, PQsendPrepare, PQsendQueryPrepared, PQsendDescribePrepared, and PQsendDescribePortal, which can be used with PQgetResult to duplicate the functionality of PQexecParams, PQprepare, PQexecPrepared, PQdescribePrepared, and PQdescribePortal respectively.

PQsendQuery

Submits a command to the server without waiting for the result(s). 1 is returned if the command was successfully dispatched and 0 if not (in which case, use PQerrorMessage to get more information about the failure).

int PQsendQuery(PGconn *conn, const char *command);

After successfully calling PQsendQuery, call PQgetResult one or more times to obtain the results. PQsendQuery cannot be called again (on the same connection) until PQgetResult has returned a null pointer, indicating that the command is done.

In pipeline mode, command strings containing more than one SQL command are disallowed.

PQsendQueryParams

Submits a command and separate parameters to the server without waiting for the result(s).

int PQsendQueryParams(PGconn *conn,
                      const char *command,
                      int nParams,
                      const Oid *paramTypes,
                      const char * const *paramValues,
                      const int *paramLengths,
                      const int *paramFormats,
                      int resultFormat);

This is equivalent to PQsendQuery except that query parameters can be specified separately from the query string. The function's parameters are handled identically to PQexecParams. Like PQexecParams, it allows only one command in the query string.

PQsendPrepare

Sends a request to create a prepared statement with the given parameters, without waiting for completion.

int PQsendPrepare(PGconn *conn,
                  const char *stmtName,
                  const char *query,
                  int nParams,
                  const Oid *paramTypes);

This is an asynchronous version of PQprepare: it returns 1 if it was able to dispatch the request, and 0 if not. After a successful call, call PQgetResult to determine whether the server successfully created the prepared statement. The function's parameters are handled identically to PQprepare.

PQsendQueryPrepared

Sends a request to execute a prepared statement with given parameters, without waiting for the result(s).

int PQsendQueryPrepared(PGconn *conn,
                        const char *stmtName,
                        int nParams,
                        const char * const *paramValues,
                        const int *paramLengths,
                        const int *paramFormats,
                        int resultFormat);

This is similar to PQsendQueryParams, but the command to be executed is specified by naming a previously-prepared statement, instead of giving a query string. The function's parameters are handled identically to PQexecPrepared.

PQsendDescribePrepared

Submits a request to obtain information about the specified prepared statement, without waiting for completion.

int PQsendDescribePrepared(PGconn *conn, const char *stmtName);

This is an asynchronous version of PQdescribePrepared: it returns 1 if it was able to dispatch the request, and 0 if not. After a successful call, call PQgetResult to obtain the results. The function's parameters are handled identically to PQdescribePrepared.

PQsendDescribePortal

Submits a request to obtain information about the specified portal, without waiting for completion.

int PQsendDescribePortal(PGconn *conn, const char *portalName);

This is an asynchronous version of PQdescribePortal: it returns 1 if it was able to dispatch the request, and 0 if not. After a successful call, call PQgetResult to obtain the results. The function's parameters are handled identically to PQdescribePortal.

PQgetResult

Waits for the next result from a prior PQsendQuery, PQsendQueryParams, PQsendPrepare, PQsendQueryPrepared, PQsendDescribePrepared, PQsendDescribePortal, or PQpipelineSync call, and returns it. A null pointer is returned when the command is complete and there will be no more results.

PGresult *PQgetResult(PGconn *conn);

PQgetResult must be called repeatedly until it returns a null pointer, indicating that the command is done. (If called when no command is active, PQgetResult will just return a null pointer at once.) Each non-null result from PQgetResult should be processed using the same PGresult accessor functions previously described. Don't forget to free each result object with PQclear when done with it. Note that PQgetResult will block only if a command is active and the necessary response data has not yet been read by PQconsumeInput .

In pipeline mode, PQgetResult will return normally unless an error occurs; for any subsequent query sent after the one that caused the error until (and excluding) the next synchronization point, a special result of type PGRES_PIPELINE_ABORTED will be returned, and a null pointer will be returned after it. When the pipeline synchronization point is reached, a result of type PGRES_PIPELINE_SYNC will be returned. The result of the next query after the synchronization point follows immediately (that is, no null pointer is returned after the synchronization point.)

Note

Even when PQresultStatus indicates a fatal error, PQgetResult should be called until it returns a null pointer, to allow libpq to process the error information completely.

Using PQsendQuery and PQgetResult solves one of PQexec's problems: If a command string contains multiple SQL commands, the results of those commands can be obtained individually. (This allows a simple form of overlapped processing, by the way: the client can be handling the results of one command while the server is still working on later queries in the same command string.)

Another frequently-desired feature that can be obtained with PQsendQuery and PQgetResult is retrieving large query results a row at a time. This is discussed in Section 34.6.

By itself, calling PQgetResult will still cause the client to block until the server completes the next SQL command. This can be avoided by proper use of two more functions:

PQconsumeInput

If input is available from the server, consume it.

int PQconsumeInput(PGconn *conn);

PQconsumeInput normally returns 1 indicating “no error”, but returns 0 if there was some kind of trouble (in which case PQerrorMessage can be consulted). Note that the result does not say whether any input data was actually collected. After calling PQconsumeInput , the application can check PQisBusy and/or PQnotifies to see if their state has changed.

PQconsumeInput can be called even if the application is not prepared to deal with a result or notification just yet. The function will read available data and save it in a buffer, thereby causing a select() read-ready indication to go away. The application can thus use PQconsumeInput to clear the select() condition immediately, and then examine the results at leisure.

PQisBusy

Returns 1 if a command is busy, that is, PQgetResult would block waiting for input. A 0 return indicates that PQgetResult can be called with assurance of not blocking.

int PQisBusy(PGconn *conn);

PQisBusy will not itself attempt to read data from the server; therefore PQconsumeInput must be invoked first, or the busy state will never end.

A typical application using these functions will have a main loop that uses select() or poll() to wait for all the conditions that it must respond to. One of the conditions will be input available from the server, which in terms of select() means readable data on the file descriptor identified by PQsocket. When the main loop detects input ready, it should call PQconsumeInput to read the input. It can then call PQisBusy, followed by PQgetResult if PQisBusy returns false (0). It can also call PQnotifies to detect NOTIFY messages (see Section 34.9).

A client that uses PQsendQuery/PQgetResult can also attempt to cancel a command that is still being processed by the server; see Section 34.7. But regardless of the return value of PQcancel, the application must continue with the normal result-reading sequence using PQgetResult. A successful cancellation will simply cause the command to terminate sooner than it would have otherwise.

By using the functions described above, it is possible to avoid blocking while waiting for input from the database server. However, it is still possible that the application will block waiting to send output to the server. This is relatively uncommon but can happen if very long SQL commands or data values are sent. (It is much more probable if the application sends data via COPY IN, however.) To prevent this possibility and achieve completely nonblocking database operation, the following additional functions can be used.

PQsetnonblocking

Sets the nonblocking status of the connection.

int PQsetnonblocking(PGconn *conn, int arg);

Sets the state of the connection to nonblocking if arg is 1, or blocking if arg is 0. Returns 0 if OK, -1 if error.

In the nonblocking state, calls to PQsendQuery, PQputline, PQputnbytes, PQputCopyData, and PQendcopy will not block but instead return an error if they need to be called again.

Note that PQexec does not honor nonblocking mode; if it is called, it will act in blocking fashion anyway.

PQisnonblocking

Returns the blocking status of the database connection.

int PQisnonblocking(const PGconn *conn);

Returns 1 if the connection is set to nonblocking mode and 0 if blocking.

PQflush

Attempts to flush any queued output data to the server. Returns 0 if successful (or if the send queue is empty), -1 if it failed for some reason, or 1 if it was unable to send all the data in the send queue yet (this case can only occur if the connection is nonblocking).

int PQflush(PGconn *conn);

After sending any command or data on a nonblocking connection, call PQflush. If it returns 1, wait for the socket to become read- or write-ready. If it becomes write-ready, call PQflush again. If it becomes read-ready, call PQconsumeInput , then call PQflush again. Repeat until PQflush returns 0. (It is necessary to check for read-ready and drain the input with PQconsumeInput , because the server can block trying to send us data, e.g., NOTICE messages, and won't read our data until we read its.) Once PQflush returns 0, wait for the socket to be read-ready and then read the response as described above.

Prev | Up | Next
34.3. Command Execution Functions | Home | 34.5. Pipeline Mode


34.5. Pipeline Mode
Prev Up Chapter 34. libpq — C Library Home Next

34.5. Pipeline Mode

34.5.1. Using Pipeline Mode
34.5.2. Functions Associated with Pipeline Mode
34.5.3. When to Use Pipeline Mode

libpq pipeline mode allows applications to send a query without having to read the result of the previously sent query. Taking advantage of the pipeline mode, a client will wait less for the server, since multiple queries/results can be sent/received in a single network transaction.

While pipeline mode provides a significant performance boost, writing clients using the pipeline mode is more complex because it involves managing a queue of pending queries and finding which result corresponds to which query in the queue.

Pipeline mode also generally consumes more memory on both the client and server, though careful and aggressive management of the send/receive queue can mitigate this. This applies whether or not the connection is in blocking or non-blocking mode.

While the pipeline API was introduced in PostgreSQL 14, it is a client-side feature which doesn't require special server support and works on any server that supports the v3 extended query protocol.

34.5.1. Using Pipeline Mode

To issue pipelines, the application must switch the connection into pipeline mode, which is done with PQenterPipelineMode. PQpipelineStatus can be used to test whether pipeline mode is active. In pipeline mode, only asynchronous operations are permitted, command strings containing multiple SQL commands are disallowed, and so is COPY. Using synchronous command execution functions such as PQfn, PQexec, PQexecParams, PQprepare, PQexecPrepared, PQdescribePrepared, PQdescribePortal, is an error condition. Once all dispatched commands have had their results processed, and the end pipeline result has been consumed, the application may return to non-pipelined mode with PQexitPipelineMode.

Note

It is best to use pipeline mode with libpq in non-blocking mode. If used in blocking mode it is possible for a client/server deadlock to occur. [15]

34.5.1.1. Issuing Queries

After entering pipeline mode, the application dispatches requests using PQsendQuery, PQsendQueryParams, or its prepared-query sibling PQsendQueryPrepared. These requests are queued on the client-side until flushed to the server; this occurs when PQpipelineSync is used to establish a synchronization point in the pipeline, or when PQflush is called. The functions PQsendPrepare, PQsendDescribePrepared, and PQsendDescribePortal also work in pipeline mode. Result processing is described below.

The server executes statements, and returns results, in the order the client sends them. The server will begin executing the commands in the pipeline immediately, not waiting for the end of the pipeline. Note that results are buffered on the server side; the server flushes that buffer when a synchronization point is established with PQpipelineSync, or when PQsendFlushRequest is called. If any statement encounters an error, the server aborts the current transaction and does not execute any subsequent command in the queue until the next synchronization point; a PGRES_PIPELINE_ABORTED result is produced for each such command. (This remains true even if the commands in the pipeline would rollback the transaction.) Query processing resumes after the synchronization point.

It's fine for one operation to depend on the results of a prior one; for example, one query may define a table that the next query in the same pipeline uses. Similarly, an application may create a named prepared statement and execute it with later statements in the same pipeline.

34.5.1.2. Processing Results

To process the result of one query in a pipeline, the application calls PQgetResult repeatedly and handles each result until PQgetResult returns null. The result from the next query in the pipeline may then be retrieved using PQgetResult again and the cycle repeated. The application handles individual statement results as normal. When the results of all the queries in the pipeline have been returned, PQgetResult returns a result containing the status value PGRES_PIPELINE_SYNC

The client may choose to defer result processing until the complete pipeline has been sent, or interleave that with sending further queries in the pipeline; see Section 34.5.1.4.

To enter single-row mode, call PQsetSingleRowMode before retrieving results with PQgetResult. This mode selection is effective only for the query currently being processed. For more information on the use of PQsetSingleRowMode, refer to Section 34.6.

PQgetResult behaves the same as for normal asynchronous processing except that it may contain the new PGresult types PGRES_PIPELINE_SYNC and PGRES_PIPELINE_ABORTED. PGRES_PIPELINE_SYNC is reported exactly once for each PQpipelineSync at the corresponding point in the pipeline. PGRES_PIPELINE_ABORTED is emitted in place of a normal query result for the first error and all subsequent results until the next PGRES_PIPELINE_SYNC; see Section 34.5.1.3.

PQisBusy, PQconsumeInput, etc operate as normal when processing pipeline results. In particular, a call to PQisBusy in the middle of a pipeline returns 0 if the results for all the queries issued so far have been consumed.

libpq does not provide any information to the application about the query currently being processed (except that PQgetResult returns null to indicate that we start returning the results of next query). The application must keep track of the order in which it sent queries, to associate them with their corresponding results. Applications will typically use a state machine or a FIFO queue for this.

34.5.1.3. Error Handling

From the client's perspective, after PQresultStatus returns PGRES_FATAL_ERROR, the pipeline is flagged as aborted. PQresultStatus will report a PGRES_PIPELINE_ABORTED result for each remaining queued operation in an aborted pipeline. The result for PQpipelineSync is reported as PGRES_PIPELINE_SYNC to signal the end of the aborted pipeline and resumption of normal result processing.

The client must process results with PQgetResult during error recovery.

If the pipeline used an implicit transaction, then operations that have already executed are rolled back and operations that were queued to follow the failed operation are skipped entirely. The same behavior holds if the pipeline starts and commits a single explicit transaction (i.e. the first statement is BEGIN and the last is COMMIT) except that the session remains in an aborted transaction state at the end of the pipeline. If a pipeline contains multiple explicit transactions, all transactions that committed prior to the error remain committed, the currently in-progress transaction is aborted, and all subsequent operations are skipped completely, including subsequent transactions. If a pipeline synchronization point occurs with an explicit transaction block in aborted state, the next pipeline will become aborted immediately unless the next command puts the transaction in normal mode with ROLLBACK.

Note

The client must not assume that work is committed when it sends a COMMIT — only when the corresponding result is received to confirm the commit is complete. Because errors arrive asynchronously, the application needs to be able to restart from the last received committed change and resend work done after that point if something goes wrong.

34.5.1.4. Interleaving Result Processing and Query Dispatch

To avoid deadlocks on large pipelines the client should be structured around a non-blocking event loop using operating system facilities such as select, poll, WaitForMultipleObjectEx, etc.

The client application should generally maintain a queue of work remaining to be dispatched and a queue of work that has been dispatched but not yet had its results processed. When the socket is writable it should dispatch more work. When the socket is readable it should read results and process them, matching them up to the next entry in its corresponding results queue. Based on available memory, results from the socket should be read frequently: there's no need to wait until the pipeline end to read the results. Pipelines should be scoped to logical units of work, usually (but not necessarily) one transaction per pipeline. There's no need to exit pipeline mode and re-enter it between pipelines, or to wait for one pipeline to finish before sending the next.

An example using select() and a simple state machine to track sent and received work is in src/test/modules/libpq_pipeline/libpq_pipeline.c in the PostgreSQL source distribution.

34.5.2. Functions Associated with Pipeline Mode

PQpipelineStatus

Returns the current pipeline mode status of the libpq connection.

PGpipelineStatus PQpipelineStatus(const PGconn *conn);

PQpipelineStatus can return one of the following values:

PQ_PIPELINE_ON

The libpq connection is in pipeline mode.

PQ_PIPELINE_OFF

The libpq connection is not in pipeline mode.

PQ_PIPELINE_ABORTED

The libpq connection is in pipeline mode and an error occurred while processing the current pipeline. The aborted flag is cleared when PQgetResult returns a result of type PGRES_PIPELINE_SYNC.

PQenterPipelineMode

Causes a connection to enter pipeline mode if it is currently idle or already in pipeline mode.

int PQenterPipelineMode(PGconn *conn);

Returns 1 for success. Returns 0 and has no effect if the connection is not currently idle, i.e., it has a result ready, or it is waiting for more input from the server, etc. This function does not actually send anything to the server, it just changes the libpq connection state.

PQexitPipelineMode

Causes a connection to exit pipeline mode if it is currently in pipeline mode with an empty queue and no pending results.

int PQexitPipelineMode(PGconn *conn);

Returns 1 for success. Returns 1 and takes no action if not in pipeline mode. If the current statement isn't finished processing, or PQgetResult has not been called to collect results from all previously sent query, returns 0 (in which case, use PQerrorMessage to get more information about the failure).

PQpipelineSync

Marks a synchronization point in a pipeline by sending a sync message and flushing the send buffer. This serves as the delimiter of an implicit transaction and an error recovery point; see Section 34.5.1.3.

int PQpipelineSync(PGconn *conn);

Returns 1 for success. Returns 0 if the connection is not in pipeline mode or sending a sync message failed.

PQsendFlushRequest

Sends a request for the server to flush its output buffer.

int PQsendFlushRequest(PGconn *conn);

Returns 1 for success. Returns 0 on any failure.

The server flushes its output buffer automatically as a result of PQpipelineSync being called, or on any request when not in pipeline mode; this function is useful to cause the server to flush its output buffer in pipeline mode without establishing a synchronization point. Note that the request is not itself flushed to the server automatically; use PQflush if necessary.

34.5.3. When to Use Pipeline Mode

Much like asynchronous query mode, there is no meaningful performance overhead when using pipeline mode. It increases client application complexity, and extra caution is required to prevent client/server deadlocks, but pipeline mode can offer considerable performance improvements, in exchange for increased memory usage from leaving state around longer.

Pipeline mode is most useful when the server is distant, i.e., network latency (“ping time”) is high, and also when many small operations are being performed in rapid succession. There is usually less benefit in using pipelined commands when each query takes many multiples of the client/server round-trip time to execute. A 100-statement operation run on a server 300 ms round-trip-time away would take 30 seconds in network latency alone without pipelining; with pipelining it may spend as little as 0.3 s waiting for results from the server.

Use pipelined commands when your application does lots of small INSERT, UPDATE and DELETE operations that can't easily be transformed into operations on sets, or into a COPY operation.

Pipeline mode is not useful when information from one operation is required by the client to produce the next operation. In such cases, the client would have to introduce a synchronization point and wait for a full client/server round-trip to get the results it needs. However, it's often possible to adjust the client design to exchange the required information server-side. Read-modify-write cycles are especially good candidates; for example:

BEGIN;
SELECT x FROM mytable WHERE id = 42 FOR UPDATE;
-- result: x=2
-- client adds 1 to x:
UPDATE mytable SET x = 3 WHERE id = 42;
COMMIT;

could be much more efficiently done with:

UPDATE mytable SET x = x + 1 WHERE id = 42;

Pipelining is less useful, and more complex, when a single pipeline contains multiple transactions (see Section 34.5.1.3).

[15] The client will block trying to send queries to the server, but the server will block trying to send results to the client from queries it has already processed. This only occurs when the client sends enough queries to fill both its output buffer and the server's receive buffer before it switches to processing input from the server, but it's hard to predict exactly when that will happen.

Prev | Up | Next
34.4. Asynchronous Command Processing | Home | 34.6. Retrieving Query Results Row-by-Row


34.6. Retrieving Query Results Row-by-Row
Prev Up Chapter 34. libpq — C Library Home Next

34.6. Retrieving Query Results Row-by-Row

Ordinarily, libpq collects an SQL command's entire result and returns it to the application as a single PGresult. This can be unworkable for commands that return a large number of rows. For such cases, applications can use PQsendQuery and PQgetResult in single-row mode. In this mode, the result row(s) are returned to the application one at a time, as they are received from the server.

To enter single-row mode, call PQsetSingleRowMode immediately after a successful call of PQsendQuery (or a sibling function). This mode selection is effective only for the currently executing query. Then call PQgetResult repeatedly, until it returns null, as documented in Section 34.4. If the query returns any rows, they are returned as individual PGresult objects, which look like normal query results except for having status code PGRES_SINGLE_TUPLE instead of PGRES_TUPLES_OK. After the last row, or immediately if the query returns zero rows, a zero-row object with status PGRES_TUPLES_OK is returned; this is the signal that no more rows will arrive. (But note that it is still necessary to continue calling PQgetResult until it returns null.) All of these PGresult objects will contain the same row description data (column names, types, etc) that an ordinary PGresult object for the query would have. Each object should be freed with PQclear as usual.

When using pipeline mode, single-row mode needs to be activated for each query in the pipeline before retrieving results for that query with PQgetResult. See Section 34.5 for more information.

PQsetSingleRowMode

Select single-row mode for the currently-executing query.

int PQsetSingleRowMode(PGconn *conn);

This function can only be called immediately after PQsendQuery or one of its sibling functions, before any other operation on the connection such as PQconsumeInput or PQgetResult. If called at the correct time, the function activates single-row mode for the current query and returns 1. Otherwise the mode stays unchanged and the function returns 0. In any case, the mode reverts to normal after completion of the current query.

Caution

While processing a query, the server may return some rows and then encounter an error, causing the query to be aborted. Ordinarily, libpq discards any such rows and reports only the error. But in single-row mode, those rows will have already been returned to the application. Hence, the application will see some PGRES_SINGLE_TUPLE PGresult objects followed by a PGRES_FATAL_ERROR object. For proper transactional behavior, the application must be designed to discard or undo whatever has been done with the previously-processed rows, if the query ultimately fails.

Prev | Up | Next
34.5. Pipeline Mode | Home | 34.7. Canceling Queries in Progress


34.7. Canceling Queries in Progress
Prev Up Chapter 34. libpq — C Library Home Next

34.7. Canceling Queries in Progress

A client application can request cancellation of a command that is still being processed by the server, using the functions described in this section.

PQgetCancel

Creates a data structure containing the information needed to cancel a command issued through a particular database connection.

PGcancel *PQgetCancel(PGconn *conn);

PQgetCancel creates a PGcancel object given a PGconn connection object. It will return NULL if the given conn is NULL or an invalid connection. The PGcancel object is an opaque structure that is not meant to be accessed directly by the application; it can only be passed to PQcancel or PQfreeCancel.

PQfreeCancel

Frees a data structure created by PQgetCancel.

void PQfreeCancel(PGcancel *cancel);

PQfreeCancel frees a data object previously created by PQgetCancel.

PQcancel

Requests that the server abandon processing of the current command.

int PQcancel(PGcancel *cancel, char *errbuf, int errbufsize);

The return value is 1 if the cancel request was successfully dispatched and 0 if not. If not, errbuf is filled with an explanatory error message. errbuf must be a char array of size errbufsize (the recommended size is 256 bytes).

Successful dispatch is no guarantee that the request will have any effect, however. If the cancellation is effective, the current command will terminate early and return an error result. If the cancellation fails (say, because the server was already done processing the command), then there will be no visible result at all.

PQcancel can safely be invoked from a signal handler, if the errbuf is a local variable in the signal handler. The PGcancel object is read-only as far as PQcancel is concerned, so it can also be invoked from a thread that is separate from the one manipulating the PGconn object.

PQrequestCancel

PQrequestCancel is a deprecated variant of PQcancel.

int PQrequestCancel(PGconn *conn);

Requests that the server abandon processing of the current command. It operates directly on the PGconn object, and in case of failure stores the error message in the PGconn object (whence it can be retrieved by PQerrorMessage). Although the functionality is the same, this approach creates hazards for multiple-thread programs and signal handlers, since it is possible that overwriting the PGconn's error message will mess up the operation currently in progress on the connection.

Prev | Up | Next
34.6. Retrieving Query Results Row-by-Row | Home | 34.8. The Fast-Path Interface


34.8. The Fast-Path Interface
Prev Up Chapter 34. libpq — C Library Home Next

34.8. The Fast-Path Interface

PostgreSQL provides a fast-path interface to send simple function calls to the server.

Tip

This interface is somewhat obsolete, as one can achieve similar performance and greater functionality by setting up a prepared statement to define the function call. Then, executing the statement with binary transmission of parameters and results substitutes for a fast-path function call.

The function PQfn requests execution of a server function via the fast-path interface:

PGresult *PQfn(PGconn *conn,
               int fnid,
               int *result_buf,
               int *result_len,
               int result_is_int,
               const PQArgBlock *args,
               int nargs);

typedef struct
{
    int len;
    int isint;
    union
    {
        int *ptr;
        int integer;
    } u;
} PQArgBlock;

The fnid argument is the OID of the function to be executed. args and nargs define the parameters to be passed to the function; they must match the declared function argument list. When the isint field of a parameter structure is true, the u.integer value is sent to the server as an integer of the indicated length (this must be 2 or 4 bytes); proper byte-swapping occurs. When isint is false, the indicated number of bytes at *u.ptr are sent with no processing; the data must be in the format expected by the server for binary transmission of the function's argument data type. (The declaration of u.ptr as being of type int * is historical; it would be better to consider it void *.) result_buf points to the buffer in which to place the function's return value. The caller must have allocated sufficient space to store the return value. (There is no check!) The actual result length in bytes will be returned in the integer pointed to by result_len. If a 2- or 4-byte integer result is expected, set result_is_int to 1, otherwise set it to 0. Setting result_is_int to 1 causes libpq to byte-swap the value if necessary, so that it is delivered as a proper int value for the client machine; note that a 4-byte integer is delivered into *result_buf for either allowed result size. When result_is_int is 0, the binary-format byte string sent by the server is returned unmodified. (In this case it's better to consider result_buf as being of type void *.)

PQfn always returns a valid PGresult pointer, with status PGRES_COMMAND_OK for success or PGRES_FATAL_ERROR if some problem was encountered. The result status should be checked before the result is used. The caller is responsible for freeing the PGresult with PQclear when it is no longer needed.

To pass a NULL argument to the function, set the len field of that parameter structure to -1; the isint and u fields are then irrelevant.

If the function returns NULL, *result_len is set to -1, and *result_buf is not modified.

Note that it is not possible to handle set-valued results when using this interface. Also, the function must be a plain function, not an aggregate, window function, or procedure.

Prev | Up | Next
34.7. Canceling Queries in Progress | Home | 34.9. Asynchronous Notification


34.9. Asynchronous Notification
Prev Up Chapter 34. libpq — C Library Home Next

34.9. Asynchronous Notification

PostgreSQL offers asynchronous notification via the LISTEN and NOTIFY commands. A client session registers its interest in a particular notification channel with the LISTEN command (and can stop listening with the UNLISTEN command). All sessions listening on a particular channel will be notified asynchronously when a NOTIFY command with that channel name is executed by any session. A “payload” string can be passed to communicate additional data to the listeners.

libpq applications submit LISTEN, UNLISTEN, and NOTIFY commands as ordinary SQL commands. The arrival of NOTIFY messages can subsequently be detected by calling PQnotifies.

The function PQnotifies returns the next notification from a list of unhandled notification messages received from the server. It returns a null pointer if there are no pending notifications. Once a notification is returned from PQnotifies, it is considered handled and will be removed from the list of notifications.

PGnotify *PQnotifies(PGconn *conn);

typedef struct pgNotify
{
    char *relname;              /* notification channel name */
    int  be_pid;                /* process ID of notifying server process */
    char *extra;                /* notification payload string */
} PGnotify;

After processing a PGnotify object returned by PQnotifies, be sure to free it with PQfreemem. It is sufficient to free the PGnotify pointer; the relname and extra fields do not represent separate allocations. (The names of these fields are historical; in particular, channel names need not have anything to do with relation names.)

Example 34.2 gives a sample program that illustrates the use of asynchronous notification.

PQnotifies does not actually read data from the server; it just returns messages previously absorbed by another libpq function. In ancient releases of libpq, the only way to ensure timely receipt of NOTIFY messages was to constantly submit commands, even empty ones, and then check PQnotifies after each PQexec. While this still works, it is deprecated as a waste of processing power.

A better way to check for NOTIFY messages when you have no useful commands to execute is to call PQconsumeInput , then check PQnotifies. You can use select() to wait for data to arrive from the server, thereby using no CPU power unless there is something to do. (See PQsocket to obtain the file descriptor number to use with select().) Note that this will work OK whether you submit commands with PQsendQuery/PQgetResult or simply use PQexec. You should, however, remember to check PQnotifies after each PQgetResult or PQexec, to see if any notifications came in during the processing of the command.

Prev | Up | Next
34.8. The Fast-Path Interface | Home | 34.10. Functions Associated with the COPY Command


34.10. Functions Associated with the COPY Command
Prev Up Chapter 34. libpq — C Library Home Next

34.10. Functions Associated with the COPY Command

34.10.1. Functions for Sending COPY Data
34.10.2. Functions for Receiving COPY Data
34.10.3. Obsolete Functions for COPY

The COPY command in PostgreSQL has options to read from or write to the network connection used by libpq. The functions described in this section allow applications to take advantage of this capability by supplying or consuming copied data.

The overall process is that the application first issues the SQL COPY command via PQexec or one of the equivalent functions. The response to this (if there is no error in the command) will be a PGresult object bearing a status code of PGRES_COPY_OUT or PGRES_COPY_IN (depending on the specified copy direction). The application should then use the functions of this section to receive or transmit data rows. When the data transfer is complete, another PGresult object is returned to indicate success or failure of the transfer. Its status will be PGRES_COMMAND_OK for success or PGRES_FATAL_ERROR if some problem was encountered. At this point further SQL commands can be issued via PQexec. (It is not possible to execute other SQL commands using the same connection while the COPY operation is in progress.)

If a COPY command is issued via PQexec in a string that could contain additional commands, the application must continue fetching results via PQgetResult after completing the COPY sequence. Only when PQgetResult returns NULL is it certain that the PQexec command string is done and it is safe to issue more commands.

The functions of this section should be executed only after obtaining a result status of PGRES_COPY_OUT or PGRES_COPY_IN from PQexec or PQgetResult.

A PGresult object bearing one of these status values carries some additional data about the COPY operation that is starting. This additional data is available using functions that are also used in connection with query results:

PQnfields

Returns the number of columns (fields) to be copied.

PQbinaryTuples

0 indicates the overall copy format is textual (rows separated by newlines, columns separated by separator characters, etc). 1 indicates the overall copy format is binary. See COPY for more information.

PQfformat

Returns the format code (0 for text, 1 for binary) associated with each column of the copy operation. The per-column format codes will always be zero when the overall copy format is textual, but the binary format can support both text and binary columns. (However, as of the current implementation of COPY, only binary columns appear in a binary copy; so the per-column formats always match the overall format at present.)

34.10.1. Functions for Sending COPY Data

These functions are used to send data during COPY FROM STDIN. They will fail if called when the connection is not in COPY_IN state.

PQputCopyData

Sends data to the server during COPY_IN state.

int PQputCopyData(PGconn *conn,
                  const char *buffer,
                  int nbytes);

Transmits the COPY data in the specified buffer, of length nbytes, to the server. The result is 1 if the data was queued, zero if it was not queued because of full buffers (this will only happen in nonblocking mode), or -1 if an error occurred. (Use PQerrorMessage to retrieve details if the return value is -1. If the value is zero, wait for write-ready and try again.)

The application can divide the COPY data stream into buffer loads of any convenient size. Buffer-load boundaries have no semantic significance when sending. The contents of the data stream must match the data format expected by the COPY command; see COPY for details.

PQputCopyEnd

Sends end-of-data indication to the server during COPY_IN state.

int PQputCopyEnd(PGconn *conn,
                 const char *errormsg);

Ends the COPY_IN operation successfully if errormsg is NULL. If errormsg is not NULL then the COPY is forced to fail, with the string pointed to by errormsg used as the error message. (One should not assume that this exact error message will come back from the server, however, as the server might have already failed the COPY for its own reasons.)

The result is 1 if the termination message was sent; or in nonblocking mode, this may only indicate that the termination message was successfully queued. (In nonblocking mode, to be certain that the data has been sent, you should next wait for write-ready and call PQflush, repeating until it returns zero.) Zero indicates that the function could not queue the termination message because of full buffers; this will only happen in nonblocking mode. (In this case, wait for write-ready and try the PQputCopyEnd call again.) If a hard error occurs, -1 is returned; you can use PQerrorMessage to retrieve details.

After successfully calling PQputCopyEnd, call PQgetResult to obtain the final result status of the COPY command. One can wait for this result to be available in the usual way. Then return to normal operation.

34.10.2. Functions for Receiving COPY Data

These functions are used to receive data during COPY TO STDOUT. They will fail if called when the connection is not in COPY_OUT state.

PQgetCopyData

Receives data from the server during COPY_OUT state.

int PQgetCopyData(PGconn *conn,
                  char **buffer,
                  int async);

Attempts to obtain another row of data from the server during a COPY. Data is always returned one data row at a time; if only a partial row is available, it is not returned. Successful return of a data row involves allocating a chunk of memory to hold the data. The buffer parameter must be non-NULL. *buffer is set to point to the allocated memory, or to NULL in cases where no buffer is returned. A non-NULL result buffer should be freed using PQfreemem when no longer needed.

When a row is successfully returned, the return value is the number of data bytes in the row (this will always be greater than zero). The returned string is always null-terminated, though this is probably only useful for textual COPY. A result of zero indicates that the COPY is still in progress, but no row is yet available (this is only possible when async is true). A result of -1 indicates that the COPY is done. A result of -2 indicates that an error occurred (consult PQerrorMessage for the reason).

When async is true (not zero), PQgetCopyData will not block waiting for input; it will return zero if the COPY is still in progress but no complete row is available. (In this case wait for read-ready and then call PQconsumeInput before calling PQgetCopyData again.) When async is false (zero), PQgetCopyData will block until data is available or the operation completes.

After PQgetCopyData returns -1, call PQgetResult to obtain the final result status of the COPY command. One can wait for this result to be available in the usual way. Then return to normal operation.

34.10.3. Obsolete Functions for COPY

These functions represent older methods of handling COPY. Although they still work, they are deprecated due to poor error handling, inconvenient methods of detecting end-of-data, and lack of support for binary or nonblocking transfers.

PQgetline

Reads a newline-terminated line of characters (transmitted by the server) into a buffer string of size length.

int PQgetline(PGconn *conn,
              char *buffer,
              int length);

This function copies up to length-1 characters into the buffer and converts the terminating newline into a zero byte. PQgetline returns EOF at the end of input, 0 if the entire line has been read, and 1 if the buffer is full but the terminating newline has not yet been read.

Note that the application must check to see if a new line consists of the two characters \., which indicates that the server has finished sending the results of the COPY command. If the application might receive lines that are more than length-1 characters long, care is needed to be sure it recognizes the \. line correctly (and does not, for example, mistake the end of a long data line for a terminator line).

PQgetlineAsync

Reads a row of COPY data (transmitted by the server) into a buffer without blocking.

int PQgetlineAsync(PGconn *conn,
                   char *buffer,
                   int bufsize);

This function is similar to PQgetline, but it can be used by applications that must read COPY data asynchronously, that is, without blocking. Having issued the COPY command and gotten a PGRES_COPY_OUT response, the application should call PQconsumeInput and PQgetlineAsync until the end-of-data signal is detected.

Unlike PQgetline, this function takes responsibility for detecting end-of-data.

On each call, PQgetlineAsync will return data if a complete data row is available in libpq's input buffer. Otherwise, no data is returned until the rest of the row arrives. The function returns -1 if the end-of-copy-data marker has been recognized, or 0 if no data is available, or a positive number giving the number of bytes of data returned. If -1 is returned, the caller must next call PQendcopy, and then return to normal processing.

The data returned will not extend beyond a data-row boundary. If possible a whole row will be returned at one time. But if the buffer offered by the caller is too small to hold a row sent by the server, then a partial data row will be returned. With textual data this can be detected by testing whether the last returned byte is \n or not. (In a binary COPY, actual parsing of the COPY data format will be needed to make the equivalent determination.) The returned string is not null-terminated. (If you want to add a terminating null, be sure to pass a bufsize one smaller than the room actually available.)

PQputline

Sends a null-terminated string to the server. Returns 0 if OK and EOF if unable to send the string.

int PQputline(PGconn *conn,
              const char *string);

The COPY data stream sent by a series of calls to PQputline has the same format as that returned by PQgetlineAsync, except that applications are not obliged to send exactly one data row per PQputline call; it is okay to send a partial line or multiple lines per call.

Note

Before PostgreSQL protocol 3.0, it was necessary for the application to explicitly send the two characters \. as a final line to indicate to the server that it had finished sending COPY data. While this still works, it is deprecated and the special meaning of \. can be expected to be removed in a future release. It is sufficient to call PQendcopy after having sent the actual data.

PQputnbytes

Sends a non-null-terminated string to the server. Returns 0 if OK and EOF if unable to send the string.

int PQputnbytes(PGconn *conn,
                const char *buffer,
                int nbytes);

This is exactly like PQputline, except that the data buffer need not be null-terminated since the number of bytes to send is specified directly. Use this procedure when sending binary data.

PQendcopy

Synchronizes with the server.

int PQendcopy(PGconn *conn);

This function waits until the server has finished the copying. It should either be issued when the last string has been sent to the server using PQputline or when the last string has been received from the server using PQgetline. It must be issued or the server will get “out of sync” with the client. Upon return from this function, the server is ready to receive the next SQL command. The return value is 0 on successful completion, nonzero otherwise. (Use PQerrorMessage to retrieve details if the return value is nonzero.)

When using PQgetResult, the application should respond to a PGRES_COPY_OUT result by executing PQgetline repeatedly, followed by PQendcopy after the terminator line is seen. It should then return to the PQgetResult loop until PQgetResult returns a null pointer. Similarly a PGRES_COPY_IN result is processed by a series of PQputline calls followed by PQendcopy, then return to the PQgetResult loop. This arrangement will ensure that a COPY command embedded in a series of SQL commands will be executed correctly.

Older applications are likely to submit a COPY via PQexec and assume that the transaction is done after PQendcopy. This will work correctly only if the COPY is the only SQL command in the command string.

Prev | Up | Next
34.9. Asynchronous Notification | Home | 34.11. Control Functions


34.11. Control Functions
Prev Up Chapter 34. libpq — C Library Home Next

34.11. Control Functions

These functions control miscellaneous details of libpq's behavior.

PQclientEncoding

Returns the client encoding.

int PQclientEncoding(const PGconn *conn);

Note that it returns the encoding ID, not a symbolic string such as EUC_JP. If unsuccessful, it returns -1. To convert an encoding ID to an encoding name, you can use:

char *pg_encoding_to_char(int encoding_id);

PQsetClientEncoding

Sets the client encoding.

int PQsetClientEncoding(PGconn *conn, const char *encoding);

conn is a connection to the server, and encoding is the encoding you want to use. If the function successfully sets the encoding, it returns 0, otherwise -1. The current encoding for this connection can be determined by using PQclientEncoding.

PQsetErrorVerbosity

Determines the verbosity of messages returned by PQerrorMessage and PQresultErrorMessage.

typedef enum
{
    PQERRORS_TERSE,
    PQERRORS_DEFAULT,
    PQERRORS_VERBOSE,
    PQERRORS_SQLSTATE
} PGVerbosity;

PGVerbosity PQsetErrorVerbosity(PGconn *conn, PGVerbosity verbosity);

PQsetErrorVerbosity sets the verbosity mode, returning the connection's previous setting. In TERSE mode, returned messages include severity, primary text, and position only; this will normally fit on a single line. The DEFAULT mode produces messages that include the above plus any detail, hint, or context fields (these might span multiple lines). The VERBOSE mode includes all available fields. The SQLSTATE mode includes only the error severity and the SQLSTATE error code, if one is available (if not, the output is like TERSE mode).

Changing the verbosity setting does not affect the messages available from already-existing PGresult objects, only subsequently-created ones. (But see PQresultVerboseErrorMessage if you want to print a previous error with a different verbosity.)

PQsetErrorContextVisibility

Determines the handling of CONTEXT fields in messages returned by PQerrorMessage and PQresultErrorMessage.

typedef enum
{
    PQSHOW_CONTEXT_NEVER,
    PQSHOW_CONTEXT_ERRORS,
    PQSHOW_CONTEXT_ALWAYS
} PGContextVisibility;

PGContextVisibility PQsetErrorContextVisibility(PGconn *conn, PGContextVisibility show_context);

PQsetErrorContextVisibility sets the context display mode, returning the connection's previous setting. This mode controls whether the CONTEXT field is included in messages. The NEVER mode never includes CONTEXT, while ALWAYS always includes it if available. In ERRORS mode (the default), CONTEXT fields are included only in error messages, not in notices and warnings. (However, if the verbosity setting is TERSE or SQLSTATE, CONTEXT fields are omitted regardless of the context display mode.)

Changing this mode does not affect the messages available from already-existing PGresult objects, only subsequently-created ones. (But see PQresultVerboseErrorMessage if you want to print a previous error with a different display mode.)

PQtrace

Enables tracing of the client/server communication to a debugging file stream.

void PQtrace(PGconn *conn, FILE *stream);

Each line consists of: an optional timestamp, a direction indicator (F for messages from client to server or B for messages from server to client), message length, message type, and message contents. Non-message contents fields (timestamp, direction, length and message type) are separated by a tab. Message contents are separated by a space. Protocol strings are enclosed in double quotes, while strings used as data values are enclosed in single quotes. Non-printable chars are printed as hexadecimal escapes. Further message-type-specific detail can be found in Section 53.7.

Note

On Windows, if the libpq library and an application are compiled with different flags, this function call will crash the application because the internal representation of the FILE pointers differ. Specifically, multithreaded/single-threaded, release/debug, and static/dynamic flags should be the same for the library and all applications using that library.

PQsetTraceFlags

Controls the tracing behavior of client/server communication.

void PQsetTraceFlags(PGconn *conn, int flags);

flags contains flag bits describing the operating mode of tracing. If flags contains PQTRACE_SUPPRESS_TIMESTAMPS, then the timestamp is not included when printing each message. If flags contains PQTRACE_REGRESS_MODE, then some fields are redacted when printing each message, such as object OIDs, to make the output more convenient to use in testing frameworks. This function must be called after calling PQtrace.

PQuntrace

Disables tracing started by PQtrace.

void PQuntrace(PGconn *conn);

Prev | Up | Next
34.10. Functions Associated with the COPY Command | Home | 34.12. Miscellaneous Functions


34.12. Miscellaneous Functions
Prev Up Chapter 34. libpq — C Library Home Next

34.12. Miscellaneous Functions

As always, there are some functions that just don't fit anywhere.

PQfreemem

Frees memory allocated by libpq.

void PQfreemem(void *ptr);

Frees memory allocated by libpq, particularly PQescapeByteaConn, PQescapeBytea, PQunescapeBytea, and PQnotifies. It is particularly important that this function, rather than free(), be used on Microsoft Windows. This is because allocating memory in a DLL and releasing it in the application works only if multithreaded/single-threaded, release/debug, and static/dynamic flags are the same for the DLL and the application. On non-Microsoft Windows platforms, this function is the same as the standard library function free().

PQconninfoFree

Frees the data structures allocated by PQconndefaults or PQconninfoParse.

void PQconninfoFree(PQconninfoOption *connOptions);

A simple PQfreemem will not do for this, since the array contains references to subsidiary strings.

PQencryptPasswordConn

Prepares the encrypted form of a PostgreSQL password.

char *PQencryptPasswordConn(PGconn *conn, const char *passwd, const char *user, const char *algorithm);

This function is intended to be used by client applications that wish to send commands like ALTER USER joe PASSWORD 'pwd'. It is good practice not to send the original cleartext password in such a command, because it might be exposed in command logs, activity displays, and so on. Instead, use this function to convert the password to encrypted form before it is sent.

The passwd and user arguments are the cleartext password, and the SQL name of the user it is for. algorithm specifies the encryption algorithm to use to encrypt the password. Currently supported algorithms are md5 and scram-sha-256 (on and off are also accepted as aliases for md5, for compatibility with older server versions). Note that support for scram-sha-256 was introduced in PostgreSQL version 10, and will not work correctly with older server versions. If algorithm is NULL, this function will query the server for the current value of the password_encryption setting. That can block, and will fail if the current transaction is aborted, or if the connection is busy executing another query. If you wish to use the default algorithm for the server but want to avoid blocking, query password_encryption yourself before calling PQencryptPasswordConn, and pass that value as the algorithm.

The return value is a string allocated by malloc. The caller can assume the string doesn't contain any special characters that would require escaping. Use PQfreemem to free the result when done with it. On error, returns NULL, and a suitable message is stored in the connection object.

PQencryptPassword

Prepares the md5-encrypted form of a PostgreSQL password.

char *PQencryptPassword(const char *passwd, const char *user);

PQencryptPassword is an older, deprecated version of PQencryptPasswordConn. The difference is that PQencryptPassword does not require a connection object, and md5 is always used as the encryption algorithm.

PQmakeEmptyPGresult

Constructs an empty PGresult object with the given status.

PGresult *PQmakeEmptyPGresult(PGconn *conn, ExecStatusType status);

This is libpq's internal function to allocate and initialize an empty PGresult object. This function returns NULL if memory could not be allocated. It is exported because some applications find it useful to generate result objects (particularly objects with error status) themselves. If conn is not null and status indicates an error, the current error message of the specified connection is copied into the PGresult. Also, if conn is not null, any event procedures registered in the connection are copied into the PGresult. (They do not get PGEVT_RESULTCREATE calls, but see PQfireResultCreateEvents.) Note that PQclear should eventually be called on the object, just as with a PGresult returned by libpq itself.

PQfireResultCreateEvents

Fires a PGEVT_RESULTCREATE event (see Section 34.14) for each event procedure registered in the PGresult object. Returns non-zero for success, zero if any event procedure fails.

int PQfireResultCreateEvents(PGconn *conn, PGresult *res);

The conn argument is passed through to event procedures but not used directly. It can be NULL if the event procedures won't use it.

Event procedures that have already received a PGEVT_RESULTCREATE or PGEVT_RESULTCOPY event for this object are not fired again.

The main reason that this function is separate from PQmakeEmptyPGresult is that it is often appropriate to create a PGresult and fill it with data before invoking the event procedures.

PQcopyResult

Makes a copy of a PGresult object. The copy is not linked to the source result in any way and PQclear must be called when the copy is no longer needed. If the function fails, NULL is returned.

PGresult *PQcopyResult(const PGresult *src, int flags);

This is not intended to make an exact copy. The returned result is always put into PGRES_TUPLES_OK status, and does not copy any error message in the source. (It does copy the command status string, however.) The flags argument determines what else is copied. It is a bitwise OR of several flags. PG_COPYRES_ATTRS specifies copying the source result's attributes (column definitions). PG_COPYRES_TUPLES specifies copying the source result's tuples. (This implies copying the attributes, too.) PG_COPYRES_NOTICEHOOKS specifies copying the source result's notify hooks. PG_COPYRES_EVENTS specifies copying the source result's events. (But any instance data associated with the source is not copied.)

PQsetResultAttrs

Sets the attributes of a PGresult object.

int PQsetResultAttrs(PGresult *res, int numAttributes, PGresAttDesc *attDescs);

The provided attDescs are copied into the result. If the attDescs pointer is NULL or numAttributes is less than one, the request is ignored and the function succeeds. If res already contains attributes, the function will fail. If the function fails, the return value is zero. If the function succeeds, the return value is non-zero.

PQsetvalue

Sets a tuple field value of a PGresult object.

int PQsetvalue(PGresult *res, int tup_num, int field_num, char *value, int len);

The function will automatically grow the result's internal tuples array as needed. However, the tup_num argument must be less than or equal to PQntuples, meaning this function can only grow the tuples array one tuple at a time. But any field of any existing tuple can be modified in any order. If a value at field_num already exists, it will be overwritten. If len is -1 or value is NULL, the field value will be set to an SQL null value. The value is copied into the result's private storage, thus is no longer needed after the function returns. If the function fails, the return value is zero. If the function succeeds, the return value is non-zero.

PQresultAlloc

Allocate subsidiary storage for a PGresult object.

void *PQresultAlloc(PGresult *res, size_t nBytes);

Any memory allocated with this function will be freed when res is cleared. If the function fails, the return value is NULL. The result is guaranteed to be adequately aligned for any type of data, just as for malloc.

PQresultMemorySize

Retrieves the number of bytes allocated for a PGresult object.

size_t PQresultMemorySize(const PGresult *res);

This value is the sum of all malloc requests associated with the PGresult object, that is, all the space that will be freed by PQclear. This information can be useful for managing memory consumption.

PQlibVersion

Return the version of libpq that is being used.

int PQlibVersion(void);

The result of this function can be used to determine, at run time, whether specific functionality is available in the currently loaded version of libpq. The function can be used, for example, to determine which connection options are available in PQconnectdb.

The result is formed by multiplying the library's major version number by 10000 and adding the minor version number. For example, version 10.1 will be returned as 100001, and version 11.0 will be returned as 110000.

Prior to major version 10, PostgreSQL used three-part version numbers in which the first two parts together represented the major version. For those versions, PQlibVersion uses two digits for each part; for example version 9.1.5 will be returned as 90105, and version 9.2.0 will be returned as 90200.

Therefore, for purposes of determining feature compatibility, applications should divide the result of PQlibVersion by 100 not 10000 to determine a logical major version number. In all release series, only the last two digits differ between minor releases (bug-fix releases).

Note

This function appeared in PostgreSQL version 9.1, so it cannot be used to detect required functionality in earlier versions, since calling it will create a link dependency on version 9.1 or later.

Prev | Up | Next
34.11. Control Functions | Home | 34.13. Notice Processing


34.13. Notice Processing
Prev Up Chapter 34. libpq — C Library Home Next

34.13. Notice Processing

Notice and warning messages generated by the server are not returned by the query execution functions, since they do not imply failure of the query. Instead they are passed to a notice handling function, and execution continues normally after the handler returns. The default notice handling function prints the message on stderr, but the application can override this behavior by supplying its own handling function.

For historical reasons, there are two levels of notice handling, called the notice receiver and notice processor. The default behavior is for the notice receiver to format the notice and pass a string to the notice processor for printing. However, an application that chooses to provide its own notice receiver will typically ignore the notice processor layer and just do all the work in the notice receiver.

The function PQsetNoticeReceiver sets or examines the current notice receiver for a connection object. Similarly, PQsetNoticeProcessor sets or examines the current notice processor.

typedef void (*PQnoticeReceiver) (void *arg, const PGresult *res);

PQnoticeReceiver
PQsetNoticeReceiver(PGconn *conn,
                    PQnoticeReceiver proc,
                    void *arg);

typedef void (*PQnoticeProcessor) (void *arg, const char *message);

PQnoticeProcessor
PQsetNoticeProcessor(PGconn *conn,
                     PQnoticeProcessor proc,
                     void *arg);

Each of these functions returns the previous notice receiver or processor function pointer, and sets the new value. If you supply a null function pointer, no action is taken, but the current pointer is returned.

When a notice or warning message is received from the server, or generated internally by libpq, the notice receiver function is called. It is passed the message in the form of a PGRES_NONFATAL_ERROR PGresult. (This allows the receiver to extract individual fields using PQresultErrorField, or obtain a complete preformatted message using PQresultErrorMessage or PQresultVerboseErrorMessage.) The same void pointer passed to PQsetNoticeReceiver is also passed. (This pointer can be used to access application-specific state if needed.)

The default notice receiver simply extracts the message (using PQresultErrorMessage) and passes it to the notice processor.

The notice processor is responsible for handling a notice or warning message given in text form. It is passed the string text of the message (including a trailing newline), plus a void pointer that is the same one passed to PQsetNoticeProcessor. (This pointer can be used to access application-specific state if needed.)

The default notice processor is simply:

static void
defaultNoticeProcessor(void *arg, const char *message)
{
    fprintf(stderr, "%s", message);
}

Once you have set a notice receiver or processor, you should expect that that function could be called as long as either the PGconn object or PGresult objects made from it exist. At creation of a PGresult, the PGconn's current notice handling pointers are copied into the PGresult for possible use by functions like PQgetvalue.

Prev | Up | Next
34.12. Miscellaneous Functions | Home | 34.14. Event System


34.14. Event System
Prev Up Chapter 34. libpq — C Library Home Next

34.14. Event System

34.14.1. Event Types
34.14.2. Event Callback Procedure
34.14.3. Event Support Functions
34.14.4. Event Example

libpq's event system is designed to notify registered event handlers about interesting libpq events, such as the creation or destruction of PGconn and PGresult objects. A principal use case is that this allows applications to associate their own data with a PGconn or PGresult and ensure that that data is freed at an appropriate time.

Each registered event handler is associated with two pieces of data, known to libpq only as opaque void * pointers. There is a pass-through pointer that is provided by the application when the event handler is registered with a PGconn. The pass-through pointer never changes for the life of the PGconn and all PGresults generated from it; so if used, it must point to long-lived data. In addition there is an instance data pointer, which starts out NULL in every PGconn and PGresult. This pointer can be manipulated using the PQinstanceData, PQsetInstanceData, PQresultInstanceData and PQsetResultInstanceData functions. Note that unlike the pass-through pointer, instance data of a PGconn is not automatically inherited by PGresults created from it. libpq does not know what pass-through and instance data pointers point to (if anything) and will never attempt to free them — that is the responsibility of the event handler.

34.14.1. Event Types

The enum PGEventId names the types of events handled by the event system. All its values have names beginning with PGEVT. For each event type, there is a corresponding event info structure that carries the parameters passed to the event handlers. The event types are:

PGEVT_REGISTER

The register event occurs when PQregisterEventProc is called. It is the ideal time to initialize any instanceData an event procedure may need. Only one register event will be fired per event handler per connection. If the event procedure fails, the registration is aborted.

typedef struct
{
    PGconn *conn;
} PGEventRegister;

When a PGEVT_REGISTER event is received, the evtInfo pointer should be cast to a PGEventRegister *. This structure contains a PGconn that should be in the CONNECTION_OK status; guaranteed if one calls PQregisterEventProc right after obtaining a good PGconn. When returning a failure code, all cleanup must be performed as no PGEVT_CONNDESTROY event will be sent.

PGEVT_CONNRESET

The connection reset event is fired on completion of PQreset or PQresetPoll. In both cases, the event is only fired if the reset was successful. If the event procedure fails, the entire connection reset will fail; the PGconn is put into CONNECTION_BAD status and PQresetPoll will return PGRES_POLLING_FAILED.

typedef struct
{
    PGconn *conn;
} PGEventConnReset;

When a PGEVT_CONNRESET event is received, the evtInfo pointer should be cast to a PGEventConnReset *. Although the contained PGconn was just reset, all event data remains unchanged. This event should be used to reset/reload/requery any associated instanceData. Note that even if the event procedure fails to process PGEVT_CONNRESET, it will still receive a PGEVT_CONNDESTROY event when the connection is closed.

PGEVT_CONNDESTROY

The connection destroy event is fired in response to PQfinish. It is the event procedure's responsibility to properly clean up its event data as libpq has no ability to manage this memory. Failure to clean up will lead to memory leaks.

typedef struct
{
    PGconn *conn;
} PGEventConnDestroy;

When a PGEVT_CONNDESTROY event is received, the evtInfo pointer should be cast to a PGEventConnDestroy *. This event is fired prior to PQfinish performing any other cleanup. The return value of the event procedure is ignored since there is no way of indicating a failure from PQfinish. Also, an event procedure failure should not abort the process of cleaning up unwanted memory.

PGEVT_RESULTCREATE

The result creation event is fired in response to any query execution function that generates a result, including PQgetResult. This event will only be fired after the result has been created successfully.

typedef struct
{
    PGconn *conn;
    PGresult *result;
} PGEventResultCreate;

When a PGEVT_RESULTCREATE event is received, the evtInfo pointer should be cast to a PGEventResultCreate *. The conn is the connection used to generate the result. This is the ideal place to initialize any instanceData that needs to be associated with the result. If the event procedure fails, the result will be cleared and the failure will be propagated. The event procedure must not try to PQclear the result object for itself. When returning a failure code, all cleanup must be performed as no PGEVT_RESULTDESTROY event will be sent.

PGEVT_RESULTCOPY

The result copy event is fired in response to PQcopyResult. This event will only be fired after the copy is complete. Only event procedures that have successfully handled the PGEVT_RESULTCREATE or PGEVT_RESULTCOPY event for the source result will receive PGEVT_RESULTCOPY events.

typedef struct
{
    const PGresult *src;
    PGresult *dest;
} PGEventResultCopy;

When a PGEVT_RESULTCOPY event is received, the evtInfo pointer should be cast to a PGEventResultCopy *. The src result is what was copied while the dest result is the copy destination. This event can be used to provide a deep copy of instanceData, since PQcopyResult cannot do that. If the event procedure fails, the entire copy operation will fail and the dest result will be cleared. When returning a failure code, all cleanup must be performed as no PGEVT_RESULTDESTROY event will be sent for the destination result.

PGEVT_RESULTDESTROY

The result destroy event is fired in response to a PQclear. It is the event procedure's responsibility to properly clean up its event data as libpq has no ability to manage this memory. Failure to clean up will lead to memory leaks.

typedef struct
{
    PGresult *result;
} PGEventResultDestroy;

When a PGEVT_RESULTDESTROY event is received, the evtInfo pointer should be cast to a PGEventResultDestroy *. This event is fired prior to PQclear performing any other cleanup. The return value of the event procedure is ignored since there is no way of indicating a failure from PQclear. Also, an event procedure failure should not abort the process of cleaning up unwanted memory.

34.14.2. Event Callback Procedure

PGEventProc

PGEventProc is a typedef for a pointer to an event procedure, that is, the user callback function that receives events from libpq. The signature of an event procedure must be

int eventproc(PGEventId evtId, void *evtInfo, void *passThrough)

The evtId parameter indicates which PGEVT event occurred. The evtInfo pointer must be cast to the appropriate structure type to obtain further information about the event. The passThrough parameter is the pointer provided to PQregisterEventProc when the event procedure was registered. The function should return a non-zero value if it succeeds and zero if it fails.

A particular event procedure can be registered only once in any PGconn. This is because the address of the procedure is used as a lookup key to identify the associated instance data.

Caution

On Windows, functions can have two different addresses: one visible from outside a DLL and another visible from inside the DLL. One should be careful that only one of these addresses is used with libpq's event-procedure functions, else confusion will result. The simplest rule for writing code that will work is to ensure that event procedures are declared static. If the procedure's address must be available outside its own source file, expose a separate function to return the address.

34.14.3. Event Support Functions

PQregisterEventProc

Registers an event callback procedure with libpq.

int PQregisterEventProc(PGconn *conn, PGEventProc proc,
                        const char *name, void *passThrough);

An event procedure must be registered once on each PGconn you want to receive events about. There is no limit, other than memory, on the number of event procedures that can be registered with a connection. The function returns a non-zero value if it succeeds and zero if it fails.

The proc argument will be called when a libpq event is fired. Its memory address is also used to lookup instanceData. The name argument is used to refer to the event procedure in error messages. This value cannot be NULL or a zero-length string. The name string is copied into the PGconn, so what is passed need not be long-lived. The passThrough pointer is passed to the proc whenever an event occurs. This argument can be NULL.

PQsetInstanceData

Sets the connection conn's instanceData for procedure proc to data. This returns non-zero for success and zero for failure. (Failure is only possible if proc has not been properly registered in conn.)

int PQsetInstanceData(PGconn *conn, PGEventProc proc, void *data);

PQinstanceData

Returns the connection conn's instanceData associated with procedure proc, or NULL if there is none.

void *PQinstanceData(const PGconn *conn, PGEventProc proc);

PQresultSetInstanceData

Sets the result's instanceData for proc to data. This returns non-zero for success and zero for failure. (Failure is only possible if proc has not been properly registered in the result.)

int PQresultSetInstanceData(PGresult *res, PGEventProc proc, void *data);

Beware that any storage represented by data will not be accounted for by PQresultMemorySize, unless it is allocated using PQresultAlloc. (Doing so is recommendable because it eliminates the need to free such storage explicitly when the result is destroyed.)

PQresultInstanceData

Returns the result's instanceData associated with proc, or NULL if there is none.

void *PQresultInstanceData(const PGresult *res, PGEventProc proc);

34.14.4. Event Example

Here is a skeleton example of managing private data associated with libpq connections and results.

/* required header for libpq events (note: includes libpq-fe.h) */
#include <libpq-events.h>

/* The instanceData */
typedef struct
{
    int n;
    char *str;
} mydata;

/* PGEventProc */
static int myEventProc(PGEventId evtId, void *evtInfo, void *passThrough);

int
main(void)
{
    mydata *data;
    PGresult *res;
    PGconn *conn =
        PQconnectdb("dbname=postgres options=-csearch_path=");

    if (PQstatus(conn) != CONNECTION_OK)
    {
        /* PQerrorMessage's result includes a trailing newline */
        fprintf(stderr, "%s", PQerrorMessage(conn));
        PQfinish(conn);
        return 1;
    }

    /* called once on any connection that should receive events.
     * Sends a PGEVT_REGISTER to myEventProc.
     */
    if (!PQregisterEventProc(conn, myEventProc, "mydata_proc", NULL))
    {
        fprintf(stderr, "Cannot register PGEventProc\n");
        PQfinish(conn);
        return 1;
    }

    /* conn instanceData is available */
    data = PQinstanceData(conn, myEventProc);

    /* Sends a PGEVT_RESULTCREATE to myEventProc */
    res = PQexec(conn, "SELECT 1 + 1");

    /* result instanceData is available */
    data = PQresultInstanceData(res, myEventProc);

    /* If PG_COPYRES_EVENTS is used, sends a PGEVT_RESULTCOPY to myEventProc */
    res_copy = PQcopyResult(res, PG_COPYRES_TUPLES | PG_COPYRES_EVENTS);

    /* result instanceData is available if PG_COPYRES_EVENTS was
     * used during the PQcopyResult call.
     */
    data = PQresultInstanceData(res_copy, myEventProc);

    /* Both clears send a PGEVT_RESULTDESTROY to myEventProc */
    PQclear(res);
    PQclear(res_copy);

    /* Sends a PGEVT_CONNDESTROY to myEventProc */
    PQfinish(conn);

    return 0;
}

static int
myEventProc(PGEventId evtId, void *evtInfo, void *passThrough)
{
    switch (evtId)
    {
        case PGEVT_REGISTER:
        {
            PGEventRegister *e = (PGEventRegister *)evtInfo;
            mydata *data = get_mydata(e->conn);

            /* associate app specific data with connection */
            PQsetInstanceData(e->conn, myEventProc, data);
            break;
        }

        case PGEVT_CONNRESET:
        {
            PGEventConnReset *e = (PGEventConnReset *)evtInfo;
            mydata *data = PQinstanceData(e->conn, myEventProc);

            if (data)
              memset(data, 0, sizeof(mydata));
            break;
        }

        case PGEVT_CONNDESTROY:
        {
            PGEventConnDestroy *e = (PGEventConnDestroy *)evtInfo;
            mydata *data = PQinstanceData(e->conn, myEventProc);

            /* free instance data because the conn is being destroyed */
            if (data)
              free_mydata(data);
            break;
        }

        case PGEVT_RESULTCREATE:
        {
            PGEventResultCreate *e = (PGEventResultCreate *)evtInfo;
            mydata *conn_data = PQinstanceData(e->conn, myEventProc);
            mydata *res_data = dup_mydata(conn_data);

            /* associate app specific data with result (copy it from conn) */
            PQsetResultInstanceData(e->result, myEventProc, res_data);
            break;
        }

        case PGEVT_RESULTCOPY:
        {
            PGEventResultCopy *e = (PGEventResultCopy *)evtInfo;
            mydata *src_data = PQresultInstanceData(e->src, myEventProc);
            mydata *dest_data = dup_mydata(src_data);

            /* associate app specific data with result (copy it from a result) */
            PQsetResultInstanceData(e->dest, myEventProc, dest_data);
            break;
        }

        case PGEVT_RESULTDESTROY:
        {
            PGEventResultDestroy *e = (PGEventResultDestroy *)evtInfo;
            mydata *data = PQresultInstanceData(e->result, myEventProc);

            /* free instance data because the result is being destroyed */
            if (data)
              free_mydata(data);
            break;
        }

        /* unknown event ID, just return true. */
        default:
            break;
    }

    return true; /* event processing succeeded */
}

Prev | Up | Next
34.13. Notice Processing | Home | 34.15. Environment Variables


34.15. Environment Variables
Prev Up Chapter 34. libpq — C Library Home Next

34.15. Environment Variables

The following environment variables can be used to select default connection parameter values, which will be used by PQconnectdb, PQsetdbLogin and PQsetdb if no value is directly specified by the calling code. These are useful to avoid hard-coding database connection information into simple client applications, for example.

  • PGHOST behaves the same as the host connection parameter.

  • PGHOSTADDR behaves the same as the hostaddr connection parameter. This can be set instead of or in addition to PGHOST to avoid DNS lookup overhead.

  • PGPORT behaves the same as the port connection parameter.

  • PGDATABASE behaves the same as the dbname connection parameter.

  • PGUSER behaves the same as the user connection parameter.

  • PGPASSWORD behaves the same as the password connection parameter. Use of this environment variable is not recommended for security reasons, as some operating systems allow non-root users to see process environment variables via ps; instead consider using a password file (see Section 34.16).

  • PGPASSFILE behaves the same as the passfile connection parameter.

  • PGCHANNELBINDING behaves the same as the channel_binding connection parameter.

  • PGSERVICE behaves the same as the service connection parameter.

  • PGSERVICEFILE specifies the name of the per-user connection service file (see Section 34.17). Defaults to ~/.pg_service.conf, or %APPDATA%\postgresql\.pg_service.conf on Microsoft Windows.

  • PGOPTIONS behaves the same as the options connection parameter.

  • PGAPPNAME behaves the same as the application_name connection parameter.

  • PGSSLMODE behaves the same as the sslmode connection parameter.

  • PGREQUIRESSL behaves the same as the requiressl connection parameter. This environment variable is deprecated in favor of the PGSSLMODE variable; setting both variables suppresses the effect of this one.

  • PGSSLCOMPRESSION behaves the same as the sslcompression connection parameter.

  • PGSSLCERT behaves the same as the sslcert connection parameter.

  • PGSSLKEY behaves the same as the sslkey connection parameter.

  • PGSSLROOTCERT behaves the same as the sslrootcert connection parameter.

  • PGSSLCRL behaves the same as the sslcrl connection parameter.

  • PGSSLCRLDIR behaves the same as the sslcrldir connection parameter.

  • PGSSLSNI behaves the same as the sslsni connection parameter.

  • PGREQUIREPEER behaves the same as the requirepeer connection parameter.

  • PGSSLMINPROTOCOLVERSION behaves the same as the ssl_min_protocol_version connection parameter.

  • PGSSLMAXPROTOCOLVERSION behaves the same as the ssl_max_protocol_version connection parameter.

  • PGGSSENCMODE behaves the same as the gssencmode connection parameter.

  • PGKRBSRVNAME behaves the same as the krbsrvname connection parameter.

  • PGGSSLIB behaves the same as the gsslib connection parameter.

  • PGCONNECT_TIMEOUT behaves the same as the connect_timeout connection parameter.

  • PGCLIENTENCODING behaves the same as the client_encoding connection parameter.

  • PGTARGETSESSIONATTRS behaves the same as the target_session_attrs connection parameter.

The following environment variables can be used to specify default behavior for each PostgreSQL session. (See also the ALTER ROLE and ALTER DATABASE commands for ways to set default behavior on a per-user or per-database basis.)

  • PGDATESTYLE sets the default style of date/time representation. (Equivalent to SET datestyle TO ....)

  • PGTZ sets the default time zone. (Equivalent to SET timezone TO ....)

  • PGGEQO sets the default mode for the genetic query optimizer. (Equivalent to SET geqo TO ....)

Refer to the SQL command SET for information on correct values for these environment variables.

The following environment variables determine internal behavior of libpq; they override compiled-in defaults.

  • PGSYSCONFDIR sets the directory containing the pg_service.conf file and in a future version possibly other system-wide configuration files.

  • PGLOCALEDIR sets the directory containing the locale files for message localization.

Prev | Up | Next
34.14. Event System | Home | 34.16. The Password File


34.16. The Password File
Prev Up Chapter 34. libpq — C Library Home Next

34.16. The Password File

The file .pgpass in a user's home directory can contain passwords to be used if the connection requires a password (and no password has been specified otherwise). On Microsoft Windows the file is named %APPDATA%\postgresql\pgpass.conf (where %APPDATA% refers to the Application Data subdirectory in the user's profile). Alternatively, the password file to use can be specified using the connection parameter passfile or the environment variable PGPASSFILE.

This file should contain lines of the following format:

hostname:port:database:username:password

(You can add a reminder comment to the file by copying the line above and preceding it with #.) Each of the first four fields can be a literal value, or *, which matches anything. The password field from the first line that matches the current connection parameters will be used. (Therefore, put more-specific entries first when you are using wildcards.) If an entry needs to contain : or \, escape this character with \. The host name field is matched to the host connection parameter if that is specified, otherwise to the hostaddr parameter if that is specified; if neither are given then the host name localhost is searched for. The host name localhost is also searched for when the connection is a Unix-domain socket connection and the host parameter matches libpq's default socket directory path. In a standby server, a database field of replication matches streaming replication connections made to the primary server. The database field is of limited usefulness otherwise, because users have the same password for all databases in the same cluster.

On Unix systems, the permissions on a password file must disallow any access to world or group; achieve this by a command such as chmod 0600 ~/.pgpass. If the permissions are less strict than this, the file will be ignored. On Microsoft Windows, it is assumed that the file is stored in a directory that is secure, so no special permissions check is made.

Prev | Up | Next
34.15. Environment Variables | Home | 34.17. The Connection Service File


34.17. The Connection Service File
Prev Up Chapter 34. libpq — C Library Home Next

34.17. The Connection Service File

The connection service file allows libpq connection parameters to be associated with a single service name. That service name can then be specified in a libpq connection string, and the associated settings will be used. This allows connection parameters to be modified without requiring a recompile of the libpq-using application. The service name can also be specified using the PGSERVICE environment variable.

Service names can be defined in either a per-user service file or a system-wide file. If the same service name exists in both the user and the system file, the user file takes precedence. By default, the per-user service file is named ~/.pg_service.conf. On Microsoft Windows, it is named %APPDATA%\postgresql\.pg_service.conf (where %APPDATA% refers to the Application Data subdirectory in the user's profile). A different file name can be specified by setting the environment variable PGSERVICEFILE. The system-wide file is named pg_service.conf. By default it is sought in the etc directory of the PostgreSQL installation (use pg_config --sysconfdir to identify this directory precisely). Another directory, but not a different file name, can be specified by setting the environment variable PGSYSCONFDIR.

Either service file uses an “INI file” format where the section name is the service name and the parameters are connection parameters; see Section 34.1.2 for a list. For example:

# comment
[mydb]
host=somehost
port=5433
user=admin

An example file is provided in the PostgreSQL installation at share/pg_service.conf.sample.

Connection parameters obtained from a service file are combined with parameters obtained from other sources. A service file setting overrides the corresponding environment variable, and in turn can be overridden by a value given directly in the connection string. For example, using the above service file, a connection string service=mydb port=5434 will use host somehost, port 5434, user admin, and other parameters as set by environment variables or built-in defaults.

Prev | Up | Next
34.16. The Password File | Home | 34.18. LDAP Lookup of Connection Parameters


34.18. LDAP Lookup of Connection Parameters
Prev Up Chapter 34. libpq — C Library Home Next

34.18. LDAP Lookup of Connection Parameters

If libpq has been compiled with LDAP support (option --with-ldap for configure) it is possible to retrieve connection options like host or dbname via LDAP from a central server. The advantage is that if the connection parameters for a database change, the connection information doesn't have to be updated on all client machines.

LDAP connection parameter lookup uses the connection service file pg_service.conf (see Section 34.17). A line in a pg_service.conf stanza that starts with ldap:// will be recognized as an LDAP URL and an LDAP query will be performed. The result must be a list of keyword = value pairs which will be used to set connection options. The URL must conform to RFC 1959 and be of the form

ldap://[hostname[:port]]/search_base?attribute?search_scope?filter

where hostname defaults to localhost and port defaults to 389.

Processing of pg_service.conf is terminated after a successful LDAP lookup, but is continued if the LDAP server cannot be contacted. This is to provide a fallback with further LDAP URL lines that point to different LDAP servers, classical keyword = value pairs, or default connection options. If you would rather get an error message in this case, add a syntactically incorrect line after the LDAP URL.

A sample LDAP entry that has been created with the LDIF file

version:1
dn:cn=mydatabase,dc=mycompany,dc=com
changetype:add
objectclass:top
objectclass:device
cn:mydatabase
description:host=dbserver.mycompany.com
description:port=5439
description:dbname=mydb
description:user=mydb_user
description:sslmode=require

might be queried with the following LDAP URL:

ldap://ldap.mycompany.com/dc=mycompany,dc=com?description?one?(cn=mydatabase)

You can also mix regular service file entries with LDAP lookups. A complete example for a stanza in pg_service.conf would be:

# only host and port are stored in LDAP, specify dbname and user explicitly
[customerdb]
dbname=customer
user=appuser
ldap://ldap.acme.com/cn=dbserver,cn=hosts?pgconnectinfo?base?(objectclass=*)

Prev | Up | Next
34.17. The Connection Service File | Home | 34.19. SSL Support


34.19. SSL Support
Prev Up Chapter 34. libpq — C Library Home Next

34.19. SSL Support

34.19.1. Client Verification of Server Certificates
34.19.2. Client Certificates
34.19.3. Protection Provided in Different Modes
34.19.4. SSL Client File Usage
34.19.5. SSL Library Initialization

PostgreSQL has native support for using SSL connections to encrypt client/server communications for increased security. See Section 19.9 for details about the server-side SSL functionality.

libpq reads the system-wide OpenSSL configuration file. By default, this file is named openssl.cnf and is located in the directory reported by openssl version -d. This default can be overridden by setting environment variable OPENSSL_CONF to the name of the desired configuration file.

34.19.1. Client Verification of Server Certificates

By default, PostgreSQL will not perform any verification of the server certificate. This means that it is possible to spoof the server identity (for example by modifying a DNS record or by taking over the server IP address) without the client knowing. In order to prevent spoofing, the client must be able to verify the server's identity via a chain of trust. A chain of trust is established by placing a root (self-signed) certificate authority (CA) certificate on one computer and a leaf certificate signed by the root certificate on another computer. It is also possible to use an “intermediate” certificate which is signed by the root certificate and signs leaf certificates.

To allow the client to verify the identity of the server, place a root certificate on the client and a leaf certificate signed by the root certificate on the server. To allow the server to verify the identity of the client, place a root certificate on the server and a leaf certificate signed by the root certificate on the client. One or more intermediate certificates (usually stored with the leaf certificate) can also be used to link the leaf certificate to the root certificate.

Once a chain of trust has been established, there are two ways for the client to validate the leaf certificate sent by the server. If the parameter sslmode is set to verify-ca, libpq will verify that the server is trustworthy by checking the certificate chain up to the root certificate stored on the client. If sslmode is set to verify-full, libpq will also verify that the server host name matches the name stored in the server certificate. The SSL connection will fail if the server certificate cannot be verified. verify-full is recommended in most security-sensitive environments.

In verify-full mode, the host name is matched against the certificate's Subject Alternative Name attribute(s), or against the Common Name attribute if no Subject Alternative Name of type dNSName is present. If the certificate's name attribute starts with an asterisk (*), the asterisk will be treated as a wildcard, which will match all characters except a dot (.). This means the certificate will not match subdomains. If the connection is made using an IP address instead of a host name, the IP address will be matched (without doing any DNS lookups).

To allow server certificate verification, one or more root certificates must be placed in the file ~/.postgresql/root.crt in the user's home directory. (On Microsoft Windows the file is named %APPDATA%\postgresql\root.crt.) Intermediate certificates should also be added to the file if they are needed to link the certificate chain sent by the server to the root certificates stored on the client.

Certificate Revocation List (CRL) entries are also checked if the file ~/.postgresql/root.crl exists (%APPDATA%\postgresql\root.crl on Microsoft Windows).

The location of the root certificate file and the CRL can be changed by setting the connection parameters sslrootcert and sslcrl or the environment variables PGSSLROOTCERT and PGSSLCRL. sslcrldir or the environment variable PGSSLCRLDIR can also be used to specify a directory containing CRL files.

Note

For backwards compatibility with earlier versions of PostgreSQL, if a root CA file exists, the behavior of sslmode=require will be the same as that of verify-ca, meaning the server certificate is validated against the CA. Relying on this behavior is discouraged, and applications that need certificate validation should always use verify-ca or verify-full.

34.19.2. Client Certificates

If the server attempts to verify the identity of the client by requesting the client's leaf certificate, libpq will send the certificate(s) stored in file ~/.postgresql/postgresql.crt in the user's home directory. The certificates must chain to the root certificate trusted by the server. A matching private key file ~/.postgresql/postgresql.key must also be present. On Microsoft Windows these files are named %APPDATA%\postgresql\postgresql.crt and %APPDATA%\postgresql\postgresql.key. The location of the certificate and key files can be overridden by the connection parameters sslcert and sslkey, or by the environment variables PGSSLCERT and PGSSLKEY.

On Unix systems, the permissions on the private key file must disallow any access to world or group; achieve this by a command such as chmod 0600 ~/.postgresql/postgresql.key. Alternatively, the file can be owned by root and have group read access (that is, 0640 permissions). That setup is intended for installations where certificate and key files are managed by the operating system. The user of libpq should then be made a member of the group that has access to those certificate and key files. (On Microsoft Windows, there is no file permissions check, since the %APPDATA%\postgresql directory is presumed secure.)

The first certificate in postgresql.crt must be the client's certificate because it must match the client's private key. “Intermediate” certificates can be optionally appended to the file — doing so avoids requiring storage of intermediate certificates on the server (ssl_ca_file).

The certificate and key may be in PEM or ASN.1 DER format.

The key may be stored in cleartext or encrypted with a passphrase using any algorithm supported by OpenSSL, like AES-128. If the key is stored encrypted, then the passphrase may be provided in the sslpassword connection option. If an encrypted key is supplied and the sslpassword option is absent or blank, a password will be prompted for interactively by OpenSSL with a Enter PEM pass phrase: prompt if a TTY is available. Applications can override the client certificate prompt and the handling of the sslpassword parameter by supplying their own key password callback; see PQsetSSLKeyPassHook_OpenSSL.

For instructions on creating certificates, see Section 19.9.5.

34.19.3. Protection Provided in Different Modes

The different values for the sslmode parameter provide different levels of protection. SSL can provide protection against three types of attacks:

Eavesdropping

If a third party can examine the network traffic between the client and the server, it can read both connection information (including the user name and password) and the data that is passed. SSL uses encryption to prevent this.

Man-in-the-middle (MITM)

If a third party can modify the data while passing between the client and server, it can pretend to be the server and therefore see and modify data even if it is encrypted. The third party can then forward the connection information and data to the original server, making it impossible to detect this attack. Common vectors to do this include DNS poisoning and address hijacking, whereby the client is directed to a different server than intended. There are also several other attack methods that can accomplish this. SSL uses certificate verification to prevent this, by authenticating the server to the client.

Impersonation

If a third party can pretend to be an authorized client, it can simply access data it should not have access to. Typically this can happen through insecure password management. SSL uses client certificates to prevent this, by making sure that only holders of valid certificates can access the server.

For a connection to be known SSL-secured, SSL usage must be configured on both the client and the server before the connection is made. If it is only configured on the server, the client may end up sending sensitive information (e.g., passwords) before it knows that the server requires high security. In libpq, secure connections can be ensured by setting the sslmode parameter to verify-full or verify-ca, and providing the system with a root certificate to verify against. This is analogous to using an https URL for encrypted web browsing.

Once the server has been authenticated, the client can pass sensitive data. This means that up until this point, the client does not need to know if certificates will be used for authentication, making it safe to specify that only in the server configuration.

All SSL options carry overhead in the form of encryption and key-exchange, so there is a trade-off that has to be made between performance and security. Table 34.1 illustrates the risks the different sslmode values protect against, and what statement they make about security and overhead.

Table 34.1. SSL Mode Descriptions

sslmode | Eavesdropping protection | MITM protection | Statement
disable | No | No | I don't care about security, and I don't want to pay the overhead of encryption.
allow | Maybe | No | I don't care about security, but I will pay the overhead of encryption if the server insists on it.
prefer | Maybe | No | I don't care about encryption, but I wish to pay the overhead of encryption if the server supports it.
require | Yes | No | I want my data to be encrypted, and I accept the overhead. I trust that the network will make sure I always connect to the server I want.
verify-ca | Yes | Depends on CA policy | I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server that I trust.
verify-full | Yes | Yes | I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server I trust, and that it's the one I specify.

The difference between verify-ca and verify-full depends on the policy of the root CA. If a public CA is used, verify-ca allows connections to a server that somebody else may have registered with the CA. In this case, verify-full should always be used. If a local CA is used, or even a self-signed certificate, using verify-ca often provides enough protection.

The default value for sslmode is prefer. As is shown in the table, this makes no sense from a security point of view, and it only promises performance overhead if possible. It is only provided as the default for backward compatibility, and is not recommended in secure deployments.

34.19.4. SSL Client File Usage

Table 34.2 summarizes the files that are relevant to the SSL setup on the client.

Table 34.2. Libpq/Client SSL File Usage

File | Contents | Effect
~/.postgresql/postgresql.crt | client certificate | sent to server
~/.postgresql/postgresql.key | client private key | proves client certificate sent by owner; does not indicate certificate owner is trustworthy
~/.postgresql/root.crt | trusted certificate authorities | checks that server certificate is signed by a trusted certificate authority
~/.postgresql/root.crl | certificates revoked by certificate authorities | server certificate must not be on this list

34.19.5. SSL Library Initialization

If your application initializes libssl and/or libcrypto libraries and libpq is built with SSL support, you should call PQinitOpenSSL to tell libpq that the libssl and/or libcrypto libraries have been initialized by your application, so that libpq will not also initialize those libraries.

PQinitOpenSSL

Allows applications to select which security libraries to initialize.

void PQinitOpenSSL(int do_ssl, int do_crypto);

When do_ssl is non-zero, libpq will initialize the OpenSSL library before first opening a database connection. When do_crypto is non-zero, the libcrypto library will be initialized. By default (if PQinitOpenSSL is not called), both libraries are initialized. When SSL support is not compiled in, this function is present but does nothing.

If your application uses and initializes either OpenSSL or its underlying libcrypto library, you must call this function with zeroes for the appropriate parameter(s) before first opening a database connection. Also be sure that you have done that initialization before opening a database connection.

PQinitSSL

Allows applications to select which security libraries to initialize.

void PQinitSSL(int do_ssl);

This function is equivalent to PQinitOpenSSL(do_ssl, do_ssl). It is sufficient for applications that initialize both or neither of OpenSSL and libcrypto.

PQinitSSL has been present since PostgreSQL 8.0, while PQinitOpenSSL was added in PostgreSQL 8.4, so PQinitSSL might be preferable for applications that need to work with older versions of libpq.

Prev | Up | Next
34.18. LDAP Lookup of Connection Parameters | Home | 34.20. Behavior in Threaded Programs


34.20. Behavior in Threaded Programs
Prev Up Chapter 34. libpq — C Library Home Next

34.20. Behavior in Threaded Programs

libpq is reentrant and thread-safe by default. You might need to use special compiler command-line options when you compile your application code. Refer to your system's documentation for information about how to build thread-enabled applications, or look in src/Makefile.global for PTHREAD_CFLAGS and PTHREAD_LIBS. This function allows the querying of libpq's thread-safe status:

PQisthreadsafe

Returns the thread safety status of the libpq library.

int PQisthreadsafe();

Returns 1 if the libpq is thread-safe and 0 if it is not.

One thread restriction is that no two threads attempt to manipulate the same PGconn object at the same time. In particular, you cannot issue concurrent commands from different threads through the same connection object. (If you need to run concurrent commands, use multiple connections.)

PGresult objects are normally read-only after creation, and so can be passed around freely between threads. However, if you use any of the PGresult-modifying functions described in Section 34.12 or Section 34.14, it's up to you to avoid concurrent operations on the same PGresult, too.

The deprecated functions PQrequestCancel and PQoidStatus are not thread-safe and should not be used in multithread programs. PQrequestCancel can be replaced by PQcancel. PQoidStatus can be replaced by PQoidValue.

If you are using Kerberos inside your application (in addition to inside libpq), you will need to do locking around Kerberos calls because Kerberos functions are not thread-safe. See function PQregisterThreadLock in the libpq source code for a way to do cooperative locking between libpq and your application.

Prev | Up | Next
34.19. SSL Support | Home | 34.21. Building libpq Programs


34.21. Building libpq Programs
Prev Up Chapter 34. libpq — C Library Home Next

34.21. Building libpq Programs

To build (i.e., compile and link) a program using libpq you need to do all of the following things:

  • Include the libpq-fe.h header file:

    #include <libpq-fe.h>

    If you failed to do that then you will normally get error messages from your compiler similar to:

    foo.c: In function `main':
foo.c:34: `PGconn' undeclared (first use in this function)
foo.c:35: `PGresult' undeclared (first use in this function)
foo.c:54: `CONNECTION_BAD' undeclared (first use in this function)
foo.c:68: `PGRES_COMMAND_OK' undeclared (first use in this function)
foo.c:95: `PGRES_TUPLES_OK' undeclared (first use in this function)

  • Point your compiler to the directory where the PostgreSQL header files were installed, by supplying the -Idirectory option to your compiler. (In some cases the compiler will look into the directory in question by default, so you can omit this option.) For instance, your compile command line could look like:

    cc -c -I/usr/local/pgsql/include testprog.c

    If you are using makefiles then add the option to the CPPFLAGS variable:

    CPPFLAGS += -I/usr/local/pgsql/include

    If there is any chance that your program might be compiled by other users then you should not hardcode the directory location like that. Instead, you can run the utility pg_config to find out where the header files are on the local system:

    $ pg_config --includedir
/usr/local/include

    If you have pkg-config installed, you can run instead:

    $ pkg-config --cflags libpq
-I/usr/local/include

    Note that this will already include the -I in front of the path.

    Failure to specify the correct option to the compiler will result in an error message such as:

    testlibpq.c:8:22: libpq-fe.h: No such file or directory

  • When linking the final program, specify the option -lpq so that the libpq library gets pulled in, as well as the option -Ldirectory to point the compiler to the directory where the libpq library resides. (Again, the compiler will search some directories by default.) For maximum portability, put the -L option before the -lpq option. For example:

    cc -o testprog testprog1.o testprog2.o -L/usr/local/pgsql/lib -lpq

    You can find out the library directory using pg_config as well:

    $ pg_config --libdir
/usr/local/pgsql/lib

    Or again use pkg-config:

    $ pkg-config --libs libpq
-L/usr/local/pgsql/lib -lpq

    Note again that this prints the full options, not only the path.

    Error messages that point to problems in this area could look like the following:

    testlibpq.o: In function `main':
testlibpq.o(.text+0x60): undefined reference to `PQsetdbLogin'
testlibpq.o(.text+0x71): undefined reference to `PQstatus'
testlibpq.o(.text+0xa4): undefined reference to `PQerrorMessage'

    This means you forgot -lpq.

    /usr/bin/ld: cannot find -lpq

    This means you forgot the -L option or did not specify the right directory.

Prev | Up | Next
34.20. Behavior in Threaded Programs | Home | 34.22. Example Programs


34.22. Example Programs
Prev Up Chapter 34. libpq — C Library Home Next

34.22. Example Programs

These examples and others can be found in the directory src/test/examples in the source code distribution.

Example 34.1. libpq Example Program 1

/*
 * src/test/examples/testlibpq.c
 *
 *
 * testlibpq.c
 *
 *      Test the C version of libpq, the PostgreSQL frontend library.
 */
#include <stdio.h>
#include <stdlib.h>
#include "libpq-fe.h"

static void
exit_nicely(PGconn *conn)
{
    PQfinish(conn);
    exit(1);
}

int
main(int argc, char **argv)
{
    const char *conninfo;
    PGconn     *conn;
    PGresult   *res;
    int         nFields;
    int         i,
                j;

    /*
     * If the user supplies a parameter on the command line, use it as the
     * conninfo string; otherwise default to setting dbname=postgres and using
     * environment variables or defaults for all other connection parameters.
     */
    if (argc > 1)
        conninfo = argv[1];
    else
        conninfo = "dbname = postgres";

    /* Make a connection to the database */
    conn = PQconnectdb(conninfo);

    /* Check to see that the backend connection was successfully made */
    if (PQstatus(conn) != CONNECTION_OK)
    {
        fprintf(stderr, "%s", PQerrorMessage(conn));
        exit_nicely(conn);
    }

    /* Set always-secure search path, so malicious users can't take control. */
    res = PQexec(conn,
                 "SELECT pg_catalog.set_config('search_path', '', false)");
    if (PQresultStatus(res) != PGRES_TUPLES_OK)
    {
        fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
        PQclear(res);
        exit_nicely(conn);
    }

    /*
     * Should PQclear PGresult whenever it is no longer needed to avoid memory
     * leaks
     */
    PQclear(res);

    /*
     * Our test case here involves using a cursor, for which we must be inside
     * a transaction block.  We could do the whole thing with a single
     * PQexec() of "select * from pg_database", but that's too trivial to make
     * a good example.
     */

    /* Start a transaction block */
    res = PQexec(conn, "BEGIN");
    if (PQresultStatus(res) != PGRES_COMMAND_OK)
    {
        fprintf(stderr, "BEGIN command failed: %s", PQerrorMessage(conn));
        PQclear(res);
        exit_nicely(conn);
    }
    PQclear(res);

    /*
     * Fetch rows from pg_database, the system catalog of databases
     */
    res = PQexec(conn, "DECLARE myportal CURSOR FOR select * from pg_database");
    if (PQresultStatus(res) != PGRES_COMMAND_OK)
    {
        fprintf(stderr, "DECLARE CURSOR failed: %s", PQerrorMessage(conn));
        PQclear(res);
        exit_nicely(conn);
    }
    PQclear(res);

    res = PQexec(conn, "FETCH ALL in myportal");
    if (PQresultStatus(res) != PGRES_TUPLES_OK)
    {
        fprintf(stderr, "FETCH ALL failed: %s", PQerrorMessage(conn));
        PQclear(res);
        exit_nicely(conn);
    }

    /* first, print out the attribute names */
    nFields = PQnfields(res);
    for (i = 0; i < nFields; i++)
        printf("%-15s", PQfname(res, i));
    printf("\n\n");

    /* next, print out the rows */
    for (i = 0; i < PQntuples(res); i++)
    {
        for (j = 0; j < nFields; j++)
            printf("%-15s", PQgetvalue(res, i, j));
        printf("\n");
    }

    PQclear(res);

    /* close the portal ... we don't bother to check for errors ... */
    res = PQexec(conn, "CLOSE myportal");
    PQclear(res);

    /* end the transaction */
    res = PQexec(conn, "END");
    PQclear(res);

    /* close the connection to the database and cleanup */
    PQfinish(conn);

    return 0;
}

Example 34.2. libpq Example Program 2

/*
 * src/test/examples/testlibpq2.c
 *
 *
 * testlibpq2.c
 *      Test of the asynchronous notification interface
 *
 * Start this program, then from psql in another window do
 *   NOTIFY TBL2;
 * Repeat four times to get this program to exit.
 *
 * Or, if you want to get fancy, try this:
 * populate a database with the following commands
 * (provided in src/test/examples/testlibpq2.sql):
 *
 *   CREATE SCHEMA TESTLIBPQ2;
 *   SET search_path = TESTLIBPQ2;
 *   CREATE TABLE TBL1 (i int4);
 *   CREATE TABLE TBL2 (i int4);
 *   CREATE RULE r1 AS ON INSERT TO TBL1 DO
 *     (INSERT INTO TBL2 VALUES (new.i); NOTIFY TBL2);
 *
 * Start this program, then from psql do this four times:
 *
 *   INSERT INTO TESTLIBPQ2.TBL1 VALUES (10);
 */

#ifdef WIN32
#include <windows.h>
#endif
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <sys/time.h>
#include <sys/types.h>
#ifdef HAVE_SYS_SELECT_H
#include <sys/select.h>
#endif

#include "libpq-fe.h"

static void
exit_nicely(PGconn *conn)
{
    PQfinish(conn);
    exit(1);
}

int
main(int argc, char **argv)
{
    const char *conninfo;
    PGconn     *conn;
    PGresult   *res;
    PGnotify   *notify;
    int         nnotifies;

    /*
     * If the user supplies a parameter on the command line, use it as the
     * conninfo string; otherwise default to setting dbname=postgres and using
     * environment variables or defaults for all other connection parameters.
     */
    if (argc > 1)
        conninfo = argv[1];
    else
        conninfo = "dbname = postgres";

    /* Make a connection to the database */
    conn = PQconnectdb(conninfo);

    /* Check to see that the backend connection was successfully made */
    if (PQstatus(conn) != CONNECTION_OK)
    {
        fprintf(stderr, "%s", PQerrorMessage(conn));
        exit_nicely(conn);
    }

    /* Set always-secure search path, so malicious users can't take control. */
    res = PQexec(conn,
                 "SELECT pg_catalog.set_config('search_path', '', false)");
    if (PQresultStatus(res) != PGRES_TUPLES_OK)
    {
        fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
        PQclear(res);
        exit_nicely(conn);
    }

    /*
     * Should PQclear PGresult whenever it is no longer needed to avoid memory
     * leaks
     */
    PQclear(res);

    /*
     * Issue LISTEN command to enable notifications from the rule's NOTIFY.
     */
    res = PQexec(conn, "LISTEN TBL2");
    if (PQresultStatus(res) != PGRES_COMMAND_OK)
    {
        fprintf(stderr, "LISTEN command failed: %s", PQerrorMessage(conn));
        PQclear(res);
        exit_nicely(conn);
    }
    PQclear(res);

    /* Quit after four notifies are received. */
    nnotifies = 0;
    while (nnotifies < 4)
    {
        /*
         * Sleep until something happens on the connection.  We use select(2)
         * to wait for input, but you could also use poll() or similar
         * facilities.
         */
        int         sock;
        fd_set      input_mask;

        sock = PQsocket(conn);

        if (sock < 0)
            break;              /* shouldn't happen */

        FD_ZERO(&input_mask);
        FD_SET(sock, &input_mask);

        if (select(sock + 1, &input_mask, NULL, NULL, NULL) < 0)
        {
            fprintf(stderr, "select() failed: %s\n", strerror(errno));
            exit_nicely(conn);
        }

        /* Now check for input */
        PQconsumeInput(conn);
        while ((notify = PQnotifies(conn)) != NULL)
        {
            fprintf(stderr,
                    "ASYNC NOTIFY of '%s' received from backend PID %d\n",
                    notify->relname, notify->be_pid);
            PQfreemem(notify);
            nnotifies++;
            PQconsumeInput(conn);
        }
    }

    fprintf(stderr, "Done.\n");

    /* close the connection to the database and cleanup */
    PQfinish(conn);

    return 0;
}

Example 34.3. libpq Example Program 3

/*
 * src/test/examples/testlibpq3.c
 *
 *
 * testlibpq3.c
 *      Test out-of-line parameters and binary I/O.
 *
 * Before running this, populate a database with the following commands
 * (provided in src/test/examples/testlibpq3.sql):
 *
 * CREATE SCHEMA testlibpq3;
 * SET search_path = testlibpq3;
 * SET standard_conforming_strings = ON;
 * CREATE TABLE test1 (i int4, t text, b bytea);
 * INSERT INTO test1 values (1, 'joe''s place', '\000\001\002\003\004');
 * INSERT INTO test1 values (2, 'ho there', '\004\003\002\001\000');
 *
 * The expected output is:
 *
 * tuple 0: got
 *  i = (4 bytes) 1
 *  t = (11 bytes) 'joe's place'
 *  b = (5 bytes) \000\001\002\003\004
 *
 * tuple 0: got
 *  i = (4 bytes) 2
 *  t = (8 bytes) 'ho there'
 *  b = (5 bytes) \004\003\002\001\000
 */

#ifdef WIN32
#include <windows.h>
#endif

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <string.h>
#include <sys/types.h>
#include "libpq-fe.h"

/* for ntohl/htonl */
#include <netinet/in.h>
#include <arpa/inet.h>

static void
exit_nicely(PGconn *conn)
{
    PQfinish(conn);
    exit(1);
}

/*
 * This function prints a query result that is a binary-format fetch from
 * a table defined as in the comment above.  We split it out because the
 * main() function uses it twice.
 */
static void
show_binary_results(PGresult *res)
{
    int         i,
                j;
    int         i_fnum,
                t_fnum,
                b_fnum;

    /* Use PQfnumber to avoid assumptions about field order in result */
    i_fnum = PQfnumber(res, "i");
    t_fnum = PQfnumber(res, "t");
    b_fnum = PQfnumber(res, "b");

    for (i = 0; i < PQntuples(res); i++)
    {
        char       *iptr;
        char       *tptr;
        char       *bptr;
        int         blen;
        int         ival;

        /* Get the field values (we ignore possibility they are null!) */
        iptr = PQgetvalue(res, i, i_fnum);
        tptr = PQgetvalue(res, i, t_fnum);
        bptr = PQgetvalue(res, i, b_fnum);

        /*
         * The binary representation of INT4 is in network byte order, which
         * we'd better coerce to the local byte order.
         */
        ival = ntohl(*((uint32_t *) iptr));

        /*
         * The binary representation of TEXT is, well, text, and since libpq
         * was nice enough to append a zero byte to it, it'll work just fine
         * as a C string.
         *
         * The binary representation of BYTEA is a bunch of bytes, which could
         * include embedded nulls so we have to pay attention to field length.
         */
        blen = PQgetlength(res, i, b_fnum);

        printf("tuple %d: got\n", i);
        printf(" i = (%d bytes) %d\n",
               PQgetlength(res, i, i_fnum), ival);
        printf(" t = (%d bytes) '%s'\n",
               PQgetlength(res, i, t_fnum), tptr);
        printf(" b = (%d bytes) ", blen);
        for (j = 0; j < blen; j++)
            printf("\\%03o", bptr[j]);
        printf("\n\n");
    }
}

int
main(int argc, char **argv)
{
    const char *conninfo;
    PGconn     *conn;
    PGresult   *res;
    const char *paramValues[1];
    int         paramLengths[1];
    int         paramFormats[1];
    uint32_t    binaryIntVal;

    /*
     * If the user supplies a parameter on the command line, use it as the
     * conninfo string; otherwise default to setting dbname=postgres and using
     * environment variables or defaults for all other connection parameters.
     */
    if (argc > 1)
        conninfo = argv[1];
    else
        conninfo = "dbname = postgres";

    /* Make a connection to the database */
    conn = PQconnectdb(conninfo);

    /* Check to see that the backend connection was successfully made */
    if (PQstatus(conn) != CONNECTION_OK)
    {
        fprintf(stderr, "%s", PQerrorMessage(conn));
        exit_nicely(conn);
    }

    /* Set always-secure search path, so malicious users can't take control. */
    res = PQexec(conn, "SET search_path = testlibpq3");
    if (PQresultStatus(res) != PGRES_COMMAND_OK)
    {
        fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
        PQclear(res);
        exit_nicely(conn);
    }
    PQclear(res);

    /*
     * The point of this program is to illustrate use of PQexecParams() with
     * out-of-line parameters, as well as binary transmission of data.
     *
     * This first example transmits the parameters as text, but receives the
     * results in binary format.  By using out-of-line parameters we can avoid
     * a lot of tedious mucking about with quoting and escaping, even though
     * the data is text.  Notice how we don't have to do anything special with
     * the quote mark in the parameter value.
     */

    /* Here is our out-of-line parameter value */
    paramValues[0] = "joe's place";

    res = PQexecParams(conn,
                       "SELECT * FROM test1 WHERE t = $1",
                       1,       /* one param */
                       NULL,    /* let the backend deduce param type */
                       paramValues,
                       NULL,    /* don't need param lengths since text */
                       NULL,    /* default to all text params */
                       1);      /* ask for binary results */

    if (PQresultStatus(res) != PGRES_TUPLES_OK)
    {
        fprintf(stderr, "SELECT failed: %s", PQerrorMessage(conn));
        PQclear(res);
        exit_nicely(conn);
    }

    show_binary_results(res);

    PQclear(res);

    /*
     * In this second example we transmit an integer parameter in binary form,
     * and again retrieve the results in binary form.
     *
     * Although we tell PQexecParams we are letting the backend deduce
     * parameter type, we really force the decision by casting the parameter
     * symbol in the query text.  This is a good safety measure when sending
     * binary parameters.
     */

    /* Convert integer value "2" to network byte order */
    binaryIntVal = htonl((uint32_t) 2);

    /* Set up parameter arrays for PQexecParams */
    paramValues[0] = (char *) &binaryIntVal;
    paramLengths[0] = sizeof(binaryIntVal);
    paramFormats[0] = 1;        /* binary */

    res = PQexecParams(conn,
                       "SELECT * FROM test1 WHERE i = $1::int4",
                       1,       /* one param */
                       NULL,    /* let the backend deduce param type */
                       paramValues,
                       paramLengths,
                       paramFormats,
                       1);      /* ask for binary results */

    if (PQresultStatus(res) != PGRES_TUPLES_OK)
    {
        fprintf(stderr, "SELECT failed: %s", PQerrorMessage(conn));
        PQclear(res);
        exit_nicely(conn);
    }

    show_binary_results(res);

    PQclear(res);

    /* close the connection to the database and cleanup */
    PQfinish(conn);

    return 0;
}

Prev | Up | Next
34.21. Building libpq Programs | Home | Chapter 35. Large Objects


Chapter 35. Large Objects
Prev Up Part IV. Client Interfaces Home Next

Chapter 35. Large Objects

Table of Contents

35.1. Introduction
35.2. Implementation Features
35.3. Client Interfaces
35.3.1. Creating a Large Object
35.3.2. Importing a Large Object
35.3.3. Exporting a Large Object
35.3.4. Opening an Existing Large Object
35.3.5. Writing Data to a Large Object
35.3.6. Reading Data from a Large Object
35.3.7. Seeking in a Large Object
35.3.8. Obtaining the Seek Position of a Large Object
35.3.9. Truncating a Large Object
35.3.10. Closing a Large Object Descriptor
35.3.11. Removing a Large Object
35.4. Server-Side Functions
35.5. Example Program

PostgreSQL has a large object facility, which provides stream-style access to user data that is stored in a special large-object structure. Streaming access is useful when working with data values that are too large to manipulate conveniently as a whole.

This chapter describes the implementation and the programming and query language interfaces to PostgreSQL large object data. We use the libpq C library for the examples in this chapter, but most programming interfaces native to PostgreSQL support equivalent functionality. Other interfaces might use the large object interface internally to provide generic support for large values. This is not described here.

Prev | Up | Next
34.22. Example Programs | Home | 35.1. Introduction


35.1. Introduction
Prev Up Chapter 35. Large Objects Home Next

35.1. Introduction

All large objects are stored in a single system table named pg_largeobject. Each large object also has an entry in the system table pg_largeobject_metadata. Large objects can be created, modified, and deleted using a read/write API that is similar to standard operations on files.

PostgreSQL also supports a storage system called “TOAST”, which automatically stores values larger than a single database page into a secondary storage area per table. This makes the large object facility partially obsolete. One remaining advantage of the large object facility is that it allows values up to 4 TB in size, whereas TOASTed fields can be at most 1 GB. Also, reading and updating portions of a large object can be done efficiently, while most operations on a TOASTed field will read or write the whole value as a unit.

Prev | Up | Next
Chapter 35. Large Objects | Home | 35.2. Implementation Features


35.2. Implementation Features
Prev Up Chapter 35. Large Objects Home Next

35.2. Implementation Features

The large object implementation breaks large objects up into “chunks” and stores the chunks in rows in the database. A B-tree index guarantees fast searches for the correct chunk number when doing random access reads and writes.

The chunks stored for a large object do not have to be contiguous. For example, if an application opens a new large object, seeks to offset 1000000, and writes a few bytes there, this does not result in allocation of 1000000 bytes worth of storage; only of chunks covering the range of data bytes actually written. A read operation will, however, read out zeroes for any unallocated locations preceding the last existing chunk. This corresponds to the common behavior of “sparsely allocated” files in Unix file systems.

As of PostgreSQL 9.0, large objects have an owner and a set of access permissions, which can be managed using GRANT and REVOKE. SELECT privileges are required to read a large object, and UPDATE privileges are required to write or truncate it. Only the large object's owner (or a database superuser) can delete, comment on, or change the owner of a large object. To adjust this behavior for compatibility with prior releases, see the lo_compat_privileges run-time parameter.

Prev | Up | Next
35.1. Introduction | Home | 35.3. Client Interfaces


35.3. Client Interfaces
Prev Up Chapter 35. Large Objects Home Next

35.3. Client Interfaces

35.3.1. Creating a Large Object
35.3.2. Importing a Large Object
35.3.3. Exporting a Large Object
35.3.4. Opening an Existing Large Object
35.3.5. Writing Data to a Large Object
35.3.6. Reading Data from a Large Object
35.3.7. Seeking in a Large Object
35.3.8. Obtaining the Seek Position of a Large Object
35.3.9. Truncating a Large Object
35.3.10. Closing a Large Object Descriptor
35.3.11. Removing a Large Object

This section describes the facilities that PostgreSQL's libpq client interface library provides for accessing large objects. The PostgreSQL large object interface is modeled after the Unix file-system interface, with analogues of open, read, write, lseek, etc.

All large object manipulation using these functions must take place within an SQL transaction block, since large object file descriptors are only valid for the duration of a transaction.

If an error occurs while executing any one of these functions, the function will return an otherwise-impossible value, typically 0 or -1. A message describing the error is stored in the connection object and can be retrieved with PQerrorMessage.

Client applications that use these functions should include the header file libpq/libpq-fs.h and link with the libpq library.

Client applications cannot use these functions while a libpq connection is in pipeline mode.

35.3.1. Creating a Large Object

The function

Oid lo_creat(PGconn *conn, int mode);

creates a new large object. The return value is the OID that was assigned to the new large object, or InvalidOid (zero) on failure. mode is unused and ignored as of PostgreSQL 8.1; however, for backward compatibility with earlier releases it is best to set it to INV_READ, INV_WRITE, or INV_READ | INV_WRITE. (These symbolic constants are defined in the header file libpq/libpq-fs.h.)

An example:

inv_oid = lo_creat(conn, INV_READ|INV_WRITE);

The function

Oid lo_create(PGconn *conn, Oid lobjId);

also creates a new large object. The OID to be assigned can be specified by lobjId; if so, failure occurs if that OID is already in use for some large object. If lobjId is InvalidOid (zero) then lo_create assigns an unused OID (this is the same behavior as lo_creat). The return value is the OID that was assigned to the new large object, or InvalidOid (zero) on failure.

lo_create is new as of PostgreSQL 8.1; if this function is run against an older server version, it will fail and return InvalidOid.

An example:

inv_oid = lo_create(conn, desired_oid);

35.3.2. Importing a Large Object

To import an operating system file as a large object, call

Oid lo_import(PGconn *conn, const char *filename);

filename specifies the operating system name of the file to be imported as a large object. The return value is the OID that was assigned to the new large object, or InvalidOid (zero) on failure. Note that the file is read by the client interface library, not by the server; so it must exist in the client file system and be readable by the client application.

The function

Oid lo_import_with_oid(PGconn *conn, const char *filename, Oid lobjId);

also imports a new large object. The OID to be assigned can be specified by lobjId; if so, failure occurs if that OID is already in use for some large object. If lobjId is InvalidOid (zero) then lo_import_with_oid assigns an unused OID (this is the same behavior as lo_import). The return value is the OID that was assigned to the new large object, or InvalidOid (zero) on failure.

lo_import_with_oid is new as of PostgreSQL 8.4 and uses lo_create internally which is new in 8.1; if this function is run against 8.0 or before, it will fail and return InvalidOid.

35.3.3. Exporting a Large Object

To export a large object into an operating system file, call

int lo_export(PGconn *conn, Oid lobjId, const char *filename);

The lobjId argument specifies the OID of the large object to export and the filename argument specifies the operating system name of the file. Note that the file is written by the client interface library, not by the server. Returns 1 on success, -1 on failure.

35.3.4. Opening an Existing Large Object

To open an existing large object for reading or writing, call

int lo_open(PGconn *conn, Oid lobjId, int mode);

The lobjId argument specifies the OID of the large object to open. The mode bits control whether the object is opened for reading (INV_READ), writing (INV_WRITE), or both. (These symbolic constants are defined in the header file libpq/libpq-fs.h.) lo_open returns a (non-negative) large object descriptor for later use in lo_read, lo_write, lo_lseek, lo_lseek64, lo_tell, lo_tell64, lo_truncate, lo_truncate64, and lo_close. The descriptor is only valid for the duration of the current transaction. On failure, -1 is returned.

The server currently does not distinguish between modes INV_WRITE and INV_READ | INV_WRITE: you are allowed to read from the descriptor in either case. However there is a significant difference between these modes and INV_READ alone: with INV_READ you cannot write on the descriptor, and the data read from it will reflect the contents of the large object at the time of the transaction snapshot that was active when lo_open was executed, regardless of later writes by this or other transactions. Reading from a descriptor opened with INV_WRITE returns data that reflects all writes of other committed transactions as well as writes of the current transaction. This is similar to the behavior of REPEATABLE READ versus READ COMMITTED transaction modes for ordinary SQL SELECT commands.

lo_open will fail if SELECT privilege is not available for the large object, or if INV_WRITE is specified and UPDATE privilege is not available. (Prior to PostgreSQL 11, these privilege checks were instead performed at the first actual read or write call using the descriptor.) These privilege checks can be disabled with the lo_compat_privileges run-time parameter.

An example:

inv_fd = lo_open(conn, inv_oid, INV_READ|INV_WRITE);

35.3.5. Writing Data to a Large Object

The function

int lo_write(PGconn *conn, int fd, const char *buf, size_t len);

writes len bytes from buf (which must be of size len) to large object descriptor fd. The fd argument must have been returned by a previous lo_open. The number of bytes actually written is returned (in the current implementation, this will always equal len unless there is an error). In the event of an error, the return value is -1.

Although the len parameter is declared as size_t, this function will reject length values larger than INT_MAX. In practice, it's best to transfer data in chunks of at most a few megabytes anyway.

35.3.6. Reading Data from a Large Object

The function

int lo_read(PGconn *conn, int fd, char *buf, size_t len);

reads up to len bytes from large object descriptor fd into buf (which must be of size len). The fd argument must have been returned by a previous lo_open. The number of bytes actually read is returned; this will be less than len if the end of the large object is reached first. In the event of an error, the return value is -1.

Although the len parameter is declared as size_t, this function will reject length values larger than INT_MAX. In practice, it's best to transfer data in chunks of at most a few megabytes anyway.

35.3.7. Seeking in a Large Object

To change the current read or write location associated with a large object descriptor, call

int lo_lseek(PGconn *conn, int fd, int offset, int whence);

This function moves the current location pointer for the large object descriptor identified by fd to the new location specified by offset. The valid values for whence are SEEK_SET (seek from object start), SEEK_CUR (seek from current position), and SEEK_END (seek from object end). The return value is the new location pointer, or -1 on error.

When dealing with large objects that might exceed 2GB in size, instead use

pg_int64 lo_lseek64(PGconn *conn, int fd, pg_int64 offset, int whence);

This function has the same behavior as lo_lseek, but it can accept an offset larger than 2GB and/or deliver a result larger than 2GB. Note that lo_lseek will fail if the new location pointer would be greater than 2GB.

lo_lseek64 is new as of PostgreSQL 9.3. If this function is run against an older server version, it will fail and return -1.

35.3.8. Obtaining the Seek Position of a Large Object

To obtain the current read or write location of a large object descriptor, call

int lo_tell(PGconn *conn, int fd);

If there is an error, the return value is -1.

When dealing with large objects that might exceed 2GB in size, instead use

pg_int64 lo_tell64(PGconn *conn, int fd);

This function has the same behavior as lo_tell, but it can deliver a result larger than 2GB. Note that lo_tell will fail if the current read/write location is greater than 2GB.

lo_tell64 is new as of PostgreSQL 9.3. If this function is run against an older server version, it will fail and return -1.

35.3.9. Truncating a Large Object

To truncate a large object to a given length, call

int lo_truncate(PGconn *conn, int fd, size_t len);

This function truncates the large object descriptor fd to length len. The fd argument must have been returned by a previous lo_open. If len is greater than the large object's current length, the large object is extended to the specified length with null bytes ('\0'). On success, lo_truncate returns zero. On error, the return value is -1.

The read/write location associated with the descriptor fd is not changed.

Although the len parameter is declared as size_t, lo_truncate will reject length values larger than INT_MAX.

When dealing with large objects that might exceed 2GB in size, instead use

int lo_truncate64(PGconn *conn, int fd, pg_int64 len);

This function has the same behavior as lo_truncate, but it can accept a len value exceeding 2GB.

lo_truncate is new as of PostgreSQL 8.3; if this function is run against an older server version, it will fail and return -1.

lo_truncate64 is new as of PostgreSQL 9.3; if this function is run against an older server version, it will fail and return -1.

35.3.10. Closing a Large Object Descriptor

A large object descriptor can be closed by calling

int lo_close(PGconn *conn, int fd);

where fd is a large object descriptor returned by lo_open. On success, lo_close returns zero. On error, the return value is -1.

Any large object descriptors that remain open at the end of a transaction will be closed automatically.

35.3.11. Removing a Large Object

To remove a large object from the database, call

int lo_unlink(PGconn *conn, Oid lobjId);

The lobjId argument specifies the OID of the large object to remove. Returns 1 if successful, -1 on failure.

Prev | Up | Next
35.2. Implementation Features | Home | 35.4. Server-Side Functions


35.4. Server-Side Functions
Prev Up Chapter 35. Large Objects Home Next

35.4. Server-Side Functions

Server-side functions tailored for manipulating large objects from SQL are listed in Table 35.1.

Table 35.1. SQL-Oriented Large Object Functions

Function

Description

Example(s)

lo_from_bytea ( loid oid, data bytea ) → oid

Creates a large object and stores data in it. If loid is zero then the system will choose a free OID, otherwise that OID is used (with an error if some large object already has that OID). On success, the large object's OID is returned.

lo_from_bytea(0, '\xffffff00') → 24528

lo_put ( loid oid, offset bigint, data bytea ) → void

Writes data starting at the given offset within the large object; the large object is enlarged if necessary.

lo_put(24528, 1, '\xaa') →

lo_get ( loid oid [, offset bigint, length integer ] ) → bytea

Extracts the large object's contents, or a substring thereof.

lo_get(24528, 0, 3) → \xffaaff

There are additional server-side functions corresponding to each of the client-side functions described earlier; indeed, for the most part the client-side functions are simply interfaces to the equivalent server-side functions. The ones just as convenient to call via SQL commands are lo_creat, lo_create, lo_unlink, lo_import, and lo_export. Here are examples of their use:

CREATE TABLE image (
    name            text,
    raster          oid
);

SELECT lo_creat(-1);       -- returns OID of new, empty large object

SELECT lo_create(43213);   -- attempts to create large object with OID 43213

SELECT lo_unlink(173454);  -- deletes large object with OID 173454

INSERT INTO image (name, raster)
    VALUES ('beautiful image', lo_import('/etc/motd'));

INSERT INTO image (name, raster)  -- same as above, but specify OID to use
    VALUES ('beautiful image', lo_import('/etc/motd', 68583));

SELECT lo_export(image.raster, '/tmp/motd') FROM image
    WHERE name = 'beautiful image';

The server-side lo_import and lo_export functions behave considerably differently from their client-side analogs. These two functions read and write files in the server's file system, using the permissions of the database's owning user. Therefore, by default their use is restricted to superusers. In contrast, the client-side import and export functions read and write files in the client's file system, using the permissions of the client program. The client-side functions do not require any database privileges, except the privilege to read or write the large object in question.

Caution

It is possible to GRANT use of the server-side lo_import and lo_export functions to non-superusers, but careful consideration of the security implications is required. A malicious user of such privileges could easily parlay them into becoming superuser (for example by rewriting server configuration files), or could attack the rest of the server's file system without bothering to obtain database superuser privileges as such. Access to roles having such privilege must therefore be guarded just as carefully as access to superuser roles. Nonetheless, if use of server-side lo_import or lo_export is needed for some routine task, it's safer to use a role with such privileges than one with full superuser privileges, as that helps to reduce the risk of damage from accidental errors.

The functionality of lo_read and lo_write is also available via server-side calls, but the names of the server-side functions differ from the client side interfaces in that they do not contain underscores. You must call these functions as loread and lowrite.

Prev | Up | Next
35.3. Client Interfaces | Home | 35.5. Example Program


35.5. Example Program
Prev Up Chapter 35. Large Objects Home Next

35.5. Example Program

Example 35.1 is a sample program which shows how the large object interface in libpq can be used. Parts of the program are commented out but are left in the source for the reader's benefit. This program can also be found in src/test/examples/testlo.c in the source distribution.

Example 35.1. Large Objects with libpq Example Program

/*-----------------------------------------------------------------
 *
 * testlo.c
 *    test using large objects with libpq
 *
 * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
 * Portions Copyright (c) 1994, Regents of the University of California
 *
 *
 * IDENTIFICATION
 *    src/test/examples/testlo.c
 *
 *-----------------------------------------------------------------
 */
#include <stdio.h>
#include <stdlib.h>

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

#include "libpq-fe.h"
#include "libpq/libpq-fs.h"

#define BUFSIZE         1024

/*
 * importFile -
 *    import file "in_filename" into database as large object "lobjOid"
 *
 */
static Oid
importFile(PGconn *conn, char *filename)
{
    Oid         lobjId;
    int         lobj_fd;
    char        buf[BUFSIZE];
    int         nbytes,
                tmp;
    int         fd;

    /*
     * open the file to be read in
     */
    fd = open(filename, O_RDONLY, 0666);
    if (fd < 0)
    {                           /* error */
        fprintf(stderr, "cannot open unix file\"%s\"\n", filename);
    }

    /*
     * create the large object
     */
    lobjId = lo_creat(conn, INV_READ | INV_WRITE);
    if (lobjId == 0)
        fprintf(stderr, "cannot create large object");

    lobj_fd = lo_open(conn, lobjId, INV_WRITE);

    /*
     * read in from the Unix file and write to the inversion file
     */
    while ((nbytes = read(fd, buf, BUFSIZE)) > 0)
    {
        tmp = lo_write(conn, lobj_fd, buf, nbytes);
        if (tmp < nbytes)
            fprintf(stderr, "error while reading \"%s\"", filename);
    }

    close(fd);
    lo_close(conn, lobj_fd);

    return lobjId;
}

static void
pickout(PGconn *conn, Oid lobjId, int start, int len)
{
    int         lobj_fd;
    char       *buf;
    int         nbytes;
    int         nread;

    lobj_fd = lo_open(conn, lobjId, INV_READ);
    if (lobj_fd < 0)
        fprintf(stderr, "cannot open large object %u", lobjId);

    lo_lseek(conn, lobj_fd, start, SEEK_SET);
    buf = malloc(len + 1);

    nread = 0;
    while (len - nread > 0)
    {
        nbytes = lo_read(conn, lobj_fd, buf, len - nread);
        buf[nbytes] = '\0';
        fprintf(stderr, ">>> %s", buf);
        nread += nbytes;
        if (nbytes <= 0)
            break;              /* no more data? */
    }
    free(buf);
    fprintf(stderr, "\n");
    lo_close(conn, lobj_fd);
}

static void
overwrite(PGconn *conn, Oid lobjId, int start, int len)
{
    int         lobj_fd;
    char       *buf;
    int         nbytes;
    int         nwritten;
    int         i;

    lobj_fd = lo_open(conn, lobjId, INV_WRITE);
    if (lobj_fd < 0)
        fprintf(stderr, "cannot open large object %u", lobjId);

    lo_lseek(conn, lobj_fd, start, SEEK_SET);
    buf = malloc(len + 1);

    for (i = 0; i < len; i++)
        buf[i] = 'X';
    buf[i] = '\0';

    nwritten = 0;
    while (len - nwritten > 0)
    {
        nbytes = lo_write(conn, lobj_fd, buf + nwritten, len - nwritten);
        nwritten += nbytes;
        if (nbytes <= 0)
        {
            fprintf(stderr, "\nWRITE FAILED!\n");
            break;
        }
    }
    free(buf);
    fprintf(stderr, "\n");
    lo_close(conn, lobj_fd);
}

/*
 * exportFile -
 *    export large object "lobjOid" to file "out_filename"
 *
 */
static void
exportFile(PGconn *conn, Oid lobjId, char *filename)
{
    int         lobj_fd;
    char        buf[BUFSIZE];
    int         nbytes,
                tmp;
    int         fd;

    /*
     * open the large object
     */
    lobj_fd = lo_open(conn, lobjId, INV_READ);
    if (lobj_fd < 0)
        fprintf(stderr, "cannot open large object %u", lobjId);

    /*
     * open the file to be written to
     */
    fd = open(filename, O_CREAT | O_WRONLY | O_TRUNC, 0666);
    if (fd < 0)
    {                           /* error */
        fprintf(stderr, "cannot open unix file\"%s\"",
                filename);
    }

    /*
     * read in from the inversion file and write to the Unix file
     */
    while ((nbytes = lo_read(conn, lobj_fd, buf, BUFSIZE)) > 0)
    {
        tmp = write(fd, buf, nbytes);
        if (tmp < nbytes)
        {
            fprintf(stderr, "error while writing \"%s\"",
                    filename);
        }
    }

    lo_close(conn, lobj_fd);
    close(fd);
}

static void
exit_nicely(PGconn *conn)
{
    PQfinish(conn);
    exit(1);
}

int
main(int argc, char **argv)
{
    char       *in_filename,
               *out_filename;
    char       *database;
    Oid         lobjOid;
    PGconn     *conn;
    PGresult   *res;

    if (argc != 4)
    {
        fprintf(stderr, "Usage: %s database_name in_filename out_filename\n",
                argv[0]);
        exit(1);
    }

    database = argv[1];
    in_filename = argv[2];
    out_filename = argv[3];

    /*
     * set up the connection
     */
    conn = PQsetdb(NULL, NULL, NULL, NULL, database);

    /* check to see that the backend connection was successfully made */
    if (PQstatus(conn) != CONNECTION_OK)
    {
        fprintf(stderr, "%s", PQerrorMessage(conn));
        exit_nicely(conn);
    }

    /* Set always-secure search path, so malicious users can't take control. */
    res = PQexec(conn,
                 "SELECT pg_catalog.set_config('search_path', '', false)");
    if (PQresultStatus(res) != PGRES_TUPLES_OK)
    {
        fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
        PQclear(res);
        exit_nicely(conn);
    }
    PQclear(res);

    res = PQexec(conn, "begin");
    PQclear(res);
    printf("importing file \"%s\" ...\n", in_filename);
/*  lobjOid = importFile(conn, in_filename); */
    lobjOid = lo_import(conn, in_filename);
    if (lobjOid == 0)
        fprintf(stderr, "%s\n", PQerrorMessage(conn));
    else
    {
        printf("\tas large object %u.\n", lobjOid);

        printf("picking out bytes 1000-2000 of the large object\n");
        pickout(conn, lobjOid, 1000, 1000);

        printf("overwriting bytes 1000-2000 of the large object with X's\n");
        overwrite(conn, lobjOid, 1000, 1000);

        printf("exporting large object to file \"%s\" ...\n", out_filename);
/*      exportFile(conn, lobjOid, out_filename); */
        if (lo_export(conn, lobjOid, out_filename) < 0)
            fprintf(stderr, "%s\n", PQerrorMessage(conn));
    }

    res = PQexec(conn, "end");
    PQclear(res);
    PQfinish(conn);
    return 0;
}

Prev | Up | Next
35.4. Server-Side Functions | Home | Chapter 36. ECPG — Embedded SQL in C


Chapter 36. ECPG — Embedded SQL in C
Prev Up Part IV. Client Interfaces Home Next

Chapter 36. ECPG — Embedded SQL in C

Table of Contents

36.1. The Concept
36.2. Managing Database Connections
36.2.1. Connecting to the Database Server
36.2.2. Choosing a Connection
36.2.3. Closing a Connection
36.3. Running SQL Commands
36.3.1. Executing SQL Statements
36.3.2. Using Cursors
36.3.3. Managing Transactions
36.3.4. Prepared Statements
36.4. Using Host Variables
36.4.1. Overview
36.4.2. Declare Sections
36.4.3. Retrieving Query Results
36.4.4. Type Mapping
36.4.5. Handling Nonprimitive SQL Data Types
36.4.6. Indicators
36.5. Dynamic SQL
36.5.1. Executing Statements without a Result Set
36.5.2. Executing a Statement with Input Parameters
36.5.3. Executing a Statement with a Result Set
36.6. pgtypes Library
36.6.1. Character Strings
36.6.2. The numeric Type
36.6.3. The date Type
36.6.4. The timestamp Type
36.6.5. The interval Type
36.6.6. The decimal Type
36.6.7. errno Values of pgtypeslib
36.6.8. Special Constants of pgtypeslib
36.7. Using Descriptor Areas
36.7.1. Named SQL Descriptor Areas
36.7.2. SQLDA Descriptor Areas
36.8. Error Handling
36.8.1. Setting Callbacks
36.8.2. sqlca
36.8.3. SQLSTATE vs. SQLCODE
36.9. Preprocessor Directives
36.9.1. Including Files
36.9.2. The define and undef Directives
36.9.3. ifdef, ifndef, elif, else, and endif Directives
36.10. Processing Embedded SQL Programs
36.11. Library Functions
36.12. Large Objects
36.13. C++ Applications
36.13.1. Scope for Host Variables
36.13.2. C++ Application Development with External C Module
36.14. Embedded SQL Commands
ALLOCATE DESCRIPTOR — allocate an SQL descriptor area
CONNECT — establish a database connection
DEALLOCATE DESCRIPTOR — deallocate an SQL descriptor area
DECLARE — define a cursor
DECLARE STATEMENT — declare SQL statement identifier
DESCRIBE — obtain information about a prepared statement or result set
DISCONNECT — terminate a database connection
EXECUTE IMMEDIATE — dynamically prepare and execute a statement
GET DESCRIPTOR — get information from an SQL descriptor area
OPEN — open a dynamic cursor
PREPARE — prepare a statement for execution
SET AUTOCOMMIT — set the autocommit behavior of the current session
SET CONNECTION — select a database connection
SET DESCRIPTOR — set information in an SQL descriptor area
TYPE — define a new data type
VAR — define a variable
WHENEVER — specify the action to be taken when an SQL statement causes a specific class condition to be raised
36.15. Informix Compatibility Mode
36.15.1. Additional Types
36.15.2. Additional/Missing Embedded SQL Statements
36.15.3. Informix-compatible SQLDA Descriptor Areas
36.15.4. Additional Functions
36.15.5. Additional Constants
36.16. Internals

This chapter describes the embedded SQL package for PostgreSQL. It was written by Linus Tolke (<linus@epact.se>) and Michael Meskes (<meskes@postgresql.org>). Originally it was written to work with C. It also works with C++, but it does not recognize all C++ constructs yet.

This documentation is quite incomplete. But since this interface is standardized, additional information can be found in many resources about SQL.

Prev | Up | Next
35.5. Example Program | Home | 36.1. The Concept


36.1. The Concept
Prev Up Chapter 36. ECPG — Embedded SQL in C Home Next

36.1. The Concept

An embedded SQL program consists of code written in an ordinary programming language, in this case C, mixed with SQL commands in specially marked sections. To build the program, the source code (*.pgc) is first passed through the embedded SQL preprocessor, which converts it to an ordinary C program (*.c), and afterwards it can be processed by a C compiler. (For details about the compiling and linking see Section 36.10.) Converted ECPG applications call functions in the libpq library through the embedded SQL library (ecpglib), and communicate with the PostgreSQL server using the normal frontend-backend protocol.

Embedded SQL has advantages over other methods for handling SQL commands from C code. First, it takes care of the tedious passing of information to and from variables in your C program. Second, the SQL code in the program is checked at build time for syntactical correctness. Third, embedded SQL in C is specified in the SQL standard and supported by many other SQL database systems. The PostgreSQL implementation is designed to match this standard as much as possible, and it is usually possible to port embedded SQL programs written for other SQL databases to PostgreSQL with relative ease.

As already stated, programs written for the embedded SQL interface are normal C programs with special code inserted to perform database-related actions. This special code always has the form:

EXEC SQL ...;

These statements syntactically take the place of a C statement. Depending on the particular statement, they can appear at the global level or within a function.

Embedded SQL statements follow the case-sensitivity rules of normal SQL code, and not those of C. Also they allow nested C-style comments as per the SQL standard. The C part of the program, however, follows the C standard of not accepting nested comments. Embedded SQL statements likewise use SQL rules, not C rules, for parsing quoted strings and identifiers. (See Section 4.1.2.1 and Section 4.1.1 respectively. Note that ECPG assumes that standard_conforming_strings is on.) Of course, the C part of the program follows C quoting rules.

The following sections explain all the embedded SQL statements.

Prev | Up | Next
Chapter 36. ECPG — Embedded SQL in C | Home | 36.2. Managing Database Connections


36.2. Managing Database Connections
Prev Up Chapter 36. ECPG — Embedded SQL in C Home Next

36.2. Managing Database Connections

36.2.1. Connecting to the Database Server
36.2.2. Choosing a Connection
36.2.3. Closing a Connection

This section describes how to open, close, and switch database connections.

36.2.1. Connecting to the Database Server

One connects to a database using the following statement:

EXEC SQL CONNECT TO target [AS connection-name] [USER user-name];

The target can be specified in the following ways:

  • dbname[@hostname][:port]
  • tcp:postgresql://hostname[:port][/dbname][?options]
  • unix:postgresql://localhost[:port][/dbname][?options]
  • an SQL string literal containing one of the above forms
  • a reference to a character variable containing one of the above forms (see examples)
  • DEFAULT

The connection target DEFAULT initiates a connection to the default database under the default user name. No separate user name or connection name can be specified in that case.

If you specify the connection target directly (that is, not as a string literal or variable reference), then the components of the target are passed through normal SQL parsing; this means that, for example, the hostname must look like one or more SQL identifiers separated by dots, and those identifiers will be case-folded unless double-quoted. Values of any options must be SQL identifiers, integers, or variable references. Of course, you can put nearly anything into an SQL identifier by double-quoting it. In practice, it is probably less error-prone to use a (single-quoted) string literal or a variable reference than to write the connection target directly.

There are also different ways to specify the user name:

  • username
  • username/password
  • username IDENTIFIED BY password
  • username USING password

As above, the parameters username and password can be an SQL identifier, an SQL string literal, or a reference to a character variable.

If the connection target includes any options, those consist of keyword=value specifications separated by ampersands (&). The allowed key words are the same ones recognized by libpq (see Section 34.1.2). Spaces are ignored before any keyword or value, though not within or after one. Note that there is no way to write & within a value.

Notice that when specifying a socket connection (with the unix: prefix), the host name must be exactly localhost. To select a non-default socket directory, write the directory's pathname as the value of a host option in the options part of the target.

The connection-name is used to handle multiple connections in one program. It can be omitted if a program uses only one connection. The most recently opened connection becomes the current connection, which is used by default when an SQL statement is to be executed (see later in this chapter).

Here are some examples of CONNECT statements:

EXEC SQL CONNECT TO mydb@sql.mydomain.com;

EXEC SQL CONNECT TO tcp:postgresql://sql.mydomain.com/mydb AS myconnection USER john;

EXEC SQL BEGIN DECLARE SECTION;
const char *target = "mydb@sql.mydomain.com";
const char *user = "john";
const char *passwd = "secret";
EXEC SQL END DECLARE SECTION;
 ...
EXEC SQL CONNECT TO :target USER :user USING :passwd;
/* or EXEC SQL CONNECT TO :target USER :user/:passwd; */

The last example makes use of the feature referred to above as character variable references. You will see in later sections how C variables can be used in SQL statements when you prefix them with a colon.

Be advised that the format of the connection target is not specified in the SQL standard. So if you want to develop portable applications, you might want to use something based on the last example above to encapsulate the connection target string somewhere.

If untrusted users have access to a database that has not adopted a secure schema usage pattern, begin each session by removing publicly-writable schemas from search_path. For example, add options=-c search_path= to options, or issue EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); after connecting. This consideration is not specific to ECPG; it applies to every interface for executing arbitrary SQL commands.

36.2.2. Choosing a Connection

SQL statements in embedded SQL programs are by default executed on the current connection, that is, the most recently opened one. If an application needs to manage multiple connections, then there are three ways to handle this.

The first option is to explicitly choose a connection for each SQL statement, for example:

EXEC SQL AT connection-name SELECT ...;

This option is particularly suitable if the application needs to use several connections in mixed order.

If your application uses multiple threads of execution, they cannot share a connection concurrently. You must either explicitly control access to the connection (using mutexes) or use a connection for each thread.

The second option is to execute a statement to switch the current connection. That statement is:

EXEC SQL SET CONNECTION connection-name;

This option is particularly convenient if many statements are to be executed on the same connection.

Here is an example program managing multiple database connections:

#include <stdio.h>

EXEC SQL BEGIN DECLARE SECTION;
    char dbname[1024];
EXEC SQL END DECLARE SECTION;

int
main()
{
    EXEC SQL CONNECT TO testdb1 AS con1 USER testuser;
    EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); EXEC SQL COMMIT;
    EXEC SQL CONNECT TO testdb2 AS con2 USER testuser;
    EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); EXEC SQL COMMIT;
    EXEC SQL CONNECT TO testdb3 AS con3 USER testuser;
    EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); EXEC SQL COMMIT;

    /* This query would be executed in the last opened database "testdb3". */
    EXEC SQL SELECT current_database() INTO :dbname;
    printf("current=%s (should be testdb3)\n", dbname);

    /* Using "AT" to run a query in "testdb2" */
    EXEC SQL AT con2 SELECT current_database() INTO :dbname;
    printf("current=%s (should be testdb2)\n", dbname);

    /* Switch the current connection to "testdb1". */
    EXEC SQL SET CONNECTION con1;

    EXEC SQL SELECT current_database() INTO :dbname;
    printf("current=%s (should be testdb1)\n", dbname);

    EXEC SQL DISCONNECT ALL;
    return 0;
}

This example would produce this output:

current=testdb3 (should be testdb3)
current=testdb2 (should be testdb2)
current=testdb1 (should be testdb1)

The third option is to declare an SQL identifier linked to the connection, for example:

EXEC SQL AT connection-name DECLARE statement-name STATEMENT;
EXEC SQL PREPARE statement-name FROM :dyn-string;

Once you link an SQL identifier to a connection, you execute dynamic SQL without an AT clause. Note that this option behaves like preprocessor directives, therefore the link is enabled only in the file.

Here is an example program using this option:

#include <stdio.h>

EXEC SQL BEGIN DECLARE SECTION;
char dbname[128];
char *dyn_sql = "SELECT current_database()";
EXEC SQL END DECLARE SECTION;

int main(){
  EXEC SQL CONNECT TO postgres AS con1;
  EXEC SQL CONNECT TO testdb AS con2;
  EXEC SQL AT con1 DECLARE stmt STATEMENT;
  EXEC SQL PREPARE stmt FROM :dyn_sql;
  EXEC SQL EXECUTE stmt INTO :dbname;
  printf("%s\n", dbname);

  EXEC SQL DISCONNECT ALL;
  return 0;
}

This example would produce this output, even if the default connection is testdb:

postgres

36.2.3. Closing a Connection

To close a connection, use the following statement:

EXEC SQL DISCONNECT [connection];

The connection can be specified in the following ways:

  • connection-name
  • DEFAULT
  • CURRENT
  • ALL

If no connection name is specified, the current connection is closed.

It is good style that an application always explicitly disconnect from every connection it opened.

Prev | Up | Next
36.1. The Concept | Home | 36.3. Running SQL Commands


36.3. Running SQL Commands
Prev Up Chapter 36. ECPG — Embedded SQL in C Home Next

36.3. Running SQL Commands

36.3.1. Executing SQL Statements
36.3.2. Using Cursors
36.3.3. Managing Transactions
36.3.4. Prepared Statements

Any SQL command can be run from within an embedded SQL application. Below are some examples of how to do that.

36.3.1. Executing SQL Statements

Creating a table:

EXEC SQL CREATE TABLE foo (number integer, ascii char(16));
EXEC SQL CREATE UNIQUE INDEX num1 ON foo(number);
EXEC SQL COMMIT;

Inserting rows:

EXEC SQL INSERT INTO foo (number, ascii) VALUES (9999, 'doodad');
EXEC SQL COMMIT;

Deleting rows:

EXEC SQL DELETE FROM foo WHERE number = 9999;
EXEC SQL COMMIT;

Updates:

EXEC SQL UPDATE foo
    SET ascii = 'foobar'
    WHERE number = 9999;
EXEC SQL COMMIT;

SELECT statements that return a single result row can also be executed using EXEC SQL directly. To handle result sets with multiple rows, an application has to use a cursor; see Section 36.3.2 below. (As a special case, an application can fetch multiple rows at once into an array host variable; see Section 36.4.4.3.1.)

Single-row select:

EXEC SQL SELECT foo INTO :FooBar FROM table1 WHERE ascii = 'doodad';

Also, a configuration parameter can be retrieved with the SHOW command:

EXEC SQL SHOW search_path INTO :var;

The tokens of the form :something are host variables, that is, they refer to variables in the C program. They are explained in Section 36.4.

36.3.2. Using Cursors

To retrieve a result set holding multiple rows, an application has to declare a cursor and fetch each row from the cursor. The steps to use a cursor are the following: declare a cursor, open it, fetch a row from the cursor, repeat, and finally close it.

Select using cursors:

EXEC SQL DECLARE foo_bar CURSOR FOR
    SELECT number, ascii FROM foo
    ORDER BY ascii;
EXEC SQL OPEN foo_bar;
EXEC SQL FETCH foo_bar INTO :FooBar, DooDad;
...
EXEC SQL CLOSE foo_bar;
EXEC SQL COMMIT;

For more details about declaring a cursor, see DECLARE; for more details about fetching rows from a cursor, see FETCH.

Note

The ECPG DECLARE command does not actually cause a statement to be sent to the PostgreSQL backend. The cursor is opened in the backend (using the backend's DECLARE command) at the point when the OPEN command is executed.

36.3.3. Managing Transactions

In the default mode, statements are committed only when EXEC SQL COMMIT is issued. The embedded SQL interface also supports autocommit of transactions (similar to psql's default behavior) via the -t command-line option to ecpg (see ecpg) or via the EXEC SQL SET AUTOCOMMIT TO ON statement. In autocommit mode, each command is automatically committed unless it is inside an explicit transaction block. This mode can be explicitly turned off using EXEC SQL SET AUTOCOMMIT TO OFF.

The following transaction management commands are available:

EXEC SQL COMMIT

Commit an in-progress transaction.

EXEC SQL ROLLBACK

Roll back an in-progress transaction.

EXEC SQL PREPARE TRANSACTION transaction_id

Prepare the current transaction for two-phase commit.

EXEC SQL COMMIT PREPARED transaction_id

Commit a transaction that is in prepared state.

EXEC SQL ROLLBACK PREPARED transaction_id

Roll back a transaction that is in prepared state.

EXEC SQL SET AUTOCOMMIT TO ON

Enable autocommit mode.

EXEC SQL SET AUTOCOMMIT TO OFF

Disable autocommit mode. This is the default.

36.3.4. Prepared Statements

When the values to be passed to an SQL statement are not known at compile time, or the same statement is going to be used many times, then prepared statements can be useful.

The statement is prepared using the command PREPARE. For the values that are not known yet, use the placeholder “?”:

EXEC SQL PREPARE stmt1 FROM "SELECT oid, datname FROM pg_database WHERE oid = ?";

If a statement returns a single row, the application can call EXECUTE after PREPARE to execute the statement, supplying the actual values for the placeholders with a USING clause:

EXEC SQL EXECUTE stmt1 INTO :dboid, :dbname USING 1;

If a statement returns multiple rows, the application can use a cursor declared based on the prepared statement. To bind input parameters, the cursor must be opened with a USING clause:

EXEC SQL PREPARE stmt1 FROM "SELECT oid,datname FROM pg_database WHERE oid > ?";
EXEC SQL DECLARE foo_bar CURSOR FOR stmt1;

/* when end of result set reached, break out of while loop */
EXEC SQL WHENEVER NOT FOUND DO BREAK;

EXEC SQL OPEN foo_bar USING 100;
...
while (1)
{
    EXEC SQL FETCH NEXT FROM foo_bar INTO :dboid, :dbname;
    ...
}
EXEC SQL CLOSE foo_bar;

When you don't need the prepared statement anymore, you should deallocate it:

EXEC SQL DEALLOCATE PREPARE name;

For more details about PREPARE, see PREPARE. Also see Section 36.5 for more details about using placeholders and input parameters.

Prev | Up | Next
36.2. Managing Database Connections | Home | 36.4. Using Host Variables


36.4. Using Host Variables
Prev Up Chapter 36. ECPG — Embedded SQL in C Home Next

36.4. Using Host Variables

36.4.1. Overview
36.4.2. Declare Sections
36.4.3. Retrieving Query Results
36.4.4. Type Mapping
36.4.5. Handling Nonprimitive SQL Data Types
36.4.6. Indicators

In Section 36.3 you saw how you can execute SQL statements from an embedded SQL program. Some of those statements only used fixed values and did not provide a way to insert user-supplied values into statements or have the program process the values returned by the query. Those kinds of statements are not really useful in real applications. This section explains in detail how you can pass data between your C program and the embedded SQL statements using a simple mechanism called host variables. In an embedded SQL program we consider the SQL statements to be guests in the C program code which is the host language. Therefore the variables of the C program are called host variables.

Another way to exchange values between PostgreSQL backends and ECPG applications is the use of SQL descriptors, described in Section 36.7.

36.4.1. Overview

Passing data between the C program and the SQL statements is particularly simple in embedded SQL. Instead of having the program paste the data into the statement, which entails various complications, such as properly quoting the value, you can simply write the name of a C variable into the SQL statement, prefixed by a colon. For example:

EXEC SQL INSERT INTO sometable VALUES (:v1, 'foo', :v2);

This statement refers to two C variables named v1 and v2 and also uses a regular SQL string literal, to illustrate that you are not restricted to use one kind of data or the other.

This style of inserting C variables in SQL statements works anywhere a value expression is expected in an SQL statement.

36.4.2. Declare Sections

To pass data from the program to the database, for example as parameters in a query, or to pass data from the database back to the program, the C variables that are intended to contain this data need to be declared in specially marked sections, so the embedded SQL preprocessor is made aware of them.

This section starts with:

EXEC SQL BEGIN DECLARE SECTION;

and ends with:

EXEC SQL END DECLARE SECTION;

Between those lines, there must be normal C variable declarations, such as:

int   x = 4;
char  foo[16], bar[16];

As you can see, you can optionally assign an initial value to the variable. The variable's scope is determined by the location of its declaring section within the program. You can also declare variables with the following syntax which implicitly creates a declare section:

EXEC SQL int i = 4;

You can have as many declare sections in a program as you like.

The declarations are also echoed to the output file as normal C variables, so there's no need to declare them again. Variables that are not intended to be used in SQL commands can be declared normally outside these special sections.

The definition of a structure or union also must be listed inside a DECLARE section. Otherwise the preprocessor cannot handle these types since it does not know the definition.

36.4.3. Retrieving Query Results

Now you should be able to pass data generated by your program into an SQL command. But how do you retrieve the results of a query? For that purpose, embedded SQL provides special variants of the usual commands SELECT and FETCH. These commands have a special INTO clause that specifies which host variables the retrieved values are to be stored in. SELECT is used for a query that returns only single row, and FETCH is used for a query that returns multiple rows, using a cursor.

Here is an example:

/*
 * assume this table:
 * CREATE TABLE test1 (a int, b varchar(50));
 */

EXEC SQL BEGIN DECLARE SECTION;
int v1;
VARCHAR v2;
EXEC SQL END DECLARE SECTION;

 ...

EXEC SQL SELECT a, b INTO :v1, :v2 FROM test;

So the INTO clause appears between the select list and the FROM clause. The number of elements in the select list and the list after INTO (also called the target list) must be equal.

Here is an example using the command FETCH:

EXEC SQL BEGIN DECLARE SECTION;
int v1;
VARCHAR v2;
EXEC SQL END DECLARE SECTION;

 ...

EXEC SQL DECLARE foo CURSOR FOR SELECT a, b FROM test;

 ...

do
{
    ...
    EXEC SQL FETCH NEXT FROM foo INTO :v1, :v2;
    ...
} while (...);

Here the INTO clause appears after all the normal clauses.

36.4.4. Type Mapping

When ECPG applications exchange values between the PostgreSQL server and the C application, such as when retrieving query results from the server or executing SQL statements with input parameters, the values need to be converted between PostgreSQL data types and host language variable types (C language data types, concretely). One of the main points of ECPG is that it takes care of this automatically in most cases.

In this respect, there are two kinds of data types: Some simple PostgreSQL data types, such as integer and text, can be read and written by the application directly. Other PostgreSQL data types, such as timestamp and numeric can only be accessed through special library functions; see Section 36.4.4.2.

Table 36.1 shows which PostgreSQL data types correspond to which C data types. When you wish to send or receive a value of a given PostgreSQL data type, you should declare a C variable of the corresponding C data type in the declare section.

Table 36.1. Mapping Between PostgreSQL Data Types and C Variable Types

PostgreSQL data type | Host variable type
smallint | short
integer | int
bigint | long long int
decimal | decimal [a]
numeric | numeric [a]
real | float
double precision | double
smallserial | short
serial | int
bigserial | long long int
oid | unsigned int
character(n), varchar(n), text | char[n+1], VARCHAR[n+1]
name | char[NAMEDATALEN]
timestamp | timestamp [a]
interval | interval [a]
date | date [a]
boolean | bool [b]
bytea | char *, bytea[n]
[a] This type can only be accessed through special library functions; see Section 36.4.4.2. [b] declared in ecpglib.h if not native

36.4.4.1. Handling Character Strings

To handle SQL character string data types, such as varchar and text, there are two possible ways to declare the host variables.

One way is using char[], an array of char, which is the most common way to handle character data in C.

EXEC SQL BEGIN DECLARE SECTION;
    char str[50];
EXEC SQL END DECLARE SECTION;

Note that you have to take care of the length yourself. If you use this host variable as the target variable of a query which returns a string with more than 49 characters, a buffer overflow occurs.

The other way is using the VARCHAR type, which is a special type provided by ECPG. The definition on an array of type VARCHAR is converted into a named struct for every variable. A declaration like:

VARCHAR var[180];

is converted into:

struct varchar_var { int len; char arr[180]; } var;

The member arr hosts the string including a terminating zero byte. Thus, to store a string in a VARCHAR host variable, the host variable has to be declared with the length including the zero byte terminator. The member len holds the length of the string stored in the arr without the terminating zero byte. When a host variable is used as input for a query, if strlen(arr) and len are different, the shorter one is used.

VARCHAR can be written in upper or lower case, but not in mixed case.

char and VARCHAR host variables can also hold values of other SQL types, which will be stored in their string forms.

36.4.4.2. Accessing Special Data Types

ECPG contains some special types that help you to interact easily with some special data types from the PostgreSQL server. In particular, it has implemented support for the numeric, decimal, date, timestamp, and interval types. These data types cannot usefully be mapped to primitive host variable types (such as int, long long int, or char[]), because they have a complex internal structure. Applications deal with these types by declaring host variables in special types and accessing them using functions in the pgtypes library. The pgtypes library, described in detail in Section 36.6 contains basic functions to deal with those types, such that you do not need to send a query to the SQL server just for adding an interval to a time stamp for example.

The follow subsections describe these special data types. For more details about pgtypes library functions, see Section 36.6.

36.4.4.2.1. timestamp, date

Here is a pattern for handling timestamp variables in the ECPG host application.

First, the program has to include the header file for the timestamp type:

#include <pgtypes_timestamp.h>

Next, declare a host variable as type timestamp in the declare section:

EXEC SQL BEGIN DECLARE SECTION;
timestamp ts;
EXEC SQL END DECLARE SECTION;

And after reading a value into the host variable, process it using pgtypes library functions. In following example, the timestamp value is converted into text (ASCII) form with the PGTYPEStimestamp_to_asc() function:

EXEC SQL SELECT now()::timestamp INTO :ts;

printf("ts = %s\n", PGTYPEStimestamp_to_asc(ts));

This example will show some result like following:

ts = 2010-06-27 18:03:56.949343

In addition, the DATE type can be handled in the same way. The program has to include pgtypes_date.h, declare a host variable as the date type and convert a DATE value into a text form using PGTYPESdate_to_asc() function. For more details about the pgtypes library functions, see Section 36.6.

36.4.4.2.2. interval

The handling of the interval type is also similar to the timestamp and date types. It is required, however, to allocate memory for an interval type value explicitly. In other words, the memory space for the variable has to be allocated in the heap memory, not in the stack memory.

Here is an example program:

#include <stdio.h>
#include <stdlib.h>
#include <pgtypes_interval.h>

int
main(void)
{
EXEC SQL BEGIN DECLARE SECTION;
    interval *in;
EXEC SQL END DECLARE SECTION;

    EXEC SQL CONNECT TO testdb;
    EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); EXEC SQL COMMIT;

    in = PGTYPESinterval_new();
    EXEC SQL SELECT '1 min'::interval INTO :in;
    printf("interval = %s\n", PGTYPESinterval_to_asc(in));
    PGTYPESinterval_free(in);

    EXEC SQL COMMIT;
    EXEC SQL DISCONNECT ALL;
    return 0;
}

36.4.4.2.3. numeric, decimal

The handling of the numeric and decimal types is similar to the interval type: It requires defining a pointer, allocating some memory space on the heap, and accessing the variable using the pgtypes library functions. For more details about the pgtypes library functions, see Section 36.6.

No functions are provided specifically for the decimal type. An application has to convert it to a numeric variable using a pgtypes library function to do further processing.

Here is an example program handling numeric and decimal type variables.

#include <stdio.h>
#include <stdlib.h>
#include <pgtypes_numeric.h>

EXEC SQL WHENEVER SQLERROR STOP;

int
main(void)
{
EXEC SQL BEGIN DECLARE SECTION;
    numeric *num;
    numeric *num2;
    decimal *dec;
EXEC SQL END DECLARE SECTION;

    EXEC SQL CONNECT TO testdb;
    EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); EXEC SQL COMMIT;

    num = PGTYPESnumeric_new();
    dec = PGTYPESdecimal_new();

    EXEC SQL SELECT 12.345::numeric(4,2), 23.456::decimal(4,2) INTO :num, :dec;

    printf("numeric = %s\n", PGTYPESnumeric_to_asc(num, 0));
    printf("numeric = %s\n", PGTYPESnumeric_to_asc(num, 1));
    printf("numeric = %s\n", PGTYPESnumeric_to_asc(num, 2));

    /* Convert decimal to numeric to show a decimal value. */
    num2 = PGTYPESnumeric_new();
    PGTYPESnumeric_from_decimal(dec, num2);

    printf("decimal = %s\n", PGTYPESnumeric_to_asc(num2, 0));
    printf("decimal = %s\n", PGTYPESnumeric_to_asc(num2, 1));
    printf("decimal = %s\n", PGTYPESnumeric_to_asc(num2, 2));

    PGTYPESnumeric_free(num2);
    PGTYPESdecimal_free(dec);
    PGTYPESnumeric_free(num);

    EXEC SQL COMMIT;
    EXEC SQL DISCONNECT ALL;
    return 0;
}

36.4.4.2.4. bytea

The handling of the bytea type is similar to that of VARCHAR. The definition on an array of type bytea is converted into a named struct for every variable. A declaration like:

bytea var[180];

is converted into:

struct bytea_var { int len; char arr[180]; } var;

The member arr hosts binary format data. It can also handle '\0' as part of data, unlike VARCHAR. The data is converted from/to hex format and sent/received by ecpglib.

Note

bytea variable can be used only when bytea_output is set to hex.

36.4.4.3. Host Variables with Nonprimitive Types

As a host variable you can also use arrays, typedefs, structs, and pointers.

36.4.4.3.1. Arrays

There are two use cases for arrays as host variables. The first is a way to store some text string in char[] or VARCHAR[], as explained in Section 36.4.4.1. The second use case is to retrieve multiple rows from a query result without using a cursor. Without an array, to process a query result consisting of multiple rows, it is required to use a cursor and the FETCH command. But with array host variables, multiple rows can be received at once. The length of the array has to be defined to be able to accommodate all rows, otherwise a buffer overflow will likely occur.

Following example scans the pg_database system table and shows all OIDs and names of the available databases:

int
main(void)
{
EXEC SQL BEGIN DECLARE SECTION;
    int dbid[8];
    char dbname[8][16];
    int i;
EXEC SQL END DECLARE SECTION;

    memset(dbname, 0, sizeof(char)* 16 * 8);
    memset(dbid, 0, sizeof(int) * 8);

    EXEC SQL CONNECT TO testdb;
    EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); EXEC SQL COMMIT;

    /* Retrieve multiple rows into arrays at once. */
    EXEC SQL SELECT oid,datname INTO :dbid, :dbname FROM pg_database;

    for (i = 0; i < 8; i++)
        printf("oid=%d, dbname=%s\n", dbid[i], dbname[i]);

    EXEC SQL COMMIT;
    EXEC SQL DISCONNECT ALL;
    return 0;
}

This example shows following result. (The exact values depend on local circumstances.)

oid=1, dbname=template1
oid=11510, dbname=template0
oid=11511, dbname=postgres
oid=313780, dbname=testdb
oid=0, dbname=
oid=0, dbname=
oid=0, dbname=

36.4.4.3.2. Structures

A structure whose member names match the column names of a query result, can be used to retrieve multiple columns at once. The structure enables handling multiple column values in a single host variable.

The following example retrieves OIDs, names, and sizes of the available databases from the pg_database system table and using the pg_database_size() function. In this example, a structure variable dbinfo_t with members whose names match each column in the SELECT result is used to retrieve one result row without putting multiple host variables in the FETCH statement.

EXEC SQL BEGIN DECLARE SECTION;
    typedef struct
    {
       int oid;
       char datname[65];
       long long int size;
    } dbinfo_t;

    dbinfo_t dbval;
EXEC SQL END DECLARE SECTION;

    memset(&dbval, 0, sizeof(dbinfo_t));

    EXEC SQL DECLARE cur1 CURSOR FOR SELECT oid, datname, pg_database_size(oid) AS size FROM pg_database;
    EXEC SQL OPEN cur1;

    /* when end of result set reached, break out of while loop */
    EXEC SQL WHENEVER NOT FOUND DO BREAK;

    while (1)
    {
        /* Fetch multiple columns into one structure. */
        EXEC SQL FETCH FROM cur1 INTO :dbval;

        /* Print members of the structure. */
        printf("oid=%d, datname=%s, size=%lld\n", dbval.oid, dbval.datname, dbval.size);
    }

    EXEC SQL CLOSE cur1;

This example shows following result. (The exact values depend on local circumstances.)

oid=1, datname=template1, size=4324580
oid=11510, datname=template0, size=4243460
oid=11511, datname=postgres, size=4324580
oid=313780, datname=testdb, size=8183012

Structure host variables “absorb” as many columns as the structure as fields. Additional columns can be assigned to other host variables. For example, the above program could also be restructured like this, with the size variable outside the structure:

EXEC SQL BEGIN DECLARE SECTION;
    typedef struct
    {
       int oid;
       char datname[65];
    } dbinfo_t;

    dbinfo_t dbval;
    long long int size;
EXEC SQL END DECLARE SECTION;

    memset(&dbval, 0, sizeof(dbinfo_t));

    EXEC SQL DECLARE cur1 CURSOR FOR SELECT oid, datname, pg_database_size(oid) AS size FROM pg_database;
    EXEC SQL OPEN cur1;

    /* when end of result set reached, break out of while loop */
    EXEC SQL WHENEVER NOT FOUND DO BREAK;

    while (1)
    {
        /* Fetch multiple columns into one structure. */
        EXEC SQL FETCH FROM cur1 INTO :dbval, :size;

        /* Print members of the structure. */
        printf("oid=%d, datname=%s, size=%lld\n", dbval.oid, dbval.datname, size);
    }

    EXEC SQL CLOSE cur1;

36.4.4.3.3. Typedefs

Use the typedef keyword to map new types to already existing types.

EXEC SQL BEGIN DECLARE SECTION;
    typedef char mychartype[40];
    typedef long serial_t;
EXEC SQL END DECLARE SECTION;

Note that you could also use:

EXEC SQL TYPE serial_t IS long;

This declaration does not need to be part of a declare section.

36.4.4.3.4. Pointers

You can declare pointers to the most common types. Note however that you cannot use pointers as target variables of queries without auto-allocation. See Section 36.7 for more information on auto-allocation.

EXEC SQL BEGIN DECLARE SECTION;
    int   *intp;
    char **charp;
EXEC SQL END DECLARE SECTION;

36.4.5. Handling Nonprimitive SQL Data Types

This section contains information on how to handle nonscalar and user-defined SQL-level data types in ECPG applications. Note that this is distinct from the handling of host variables of nonprimitive types, described in the previous section.

36.4.5.1. Arrays

Multi-dimensional SQL-level arrays are not directly supported in ECPG. One-dimensional SQL-level arrays can be mapped into C array host variables and vice-versa. However, when creating a statement ecpg does not know the types of the columns, so that it cannot check if a C array is input into a corresponding SQL-level array. When processing the output of an SQL statement, ecpg has the necessary information and thus checks if both are arrays.

If a query accesses elements of an array separately, then this avoids the use of arrays in ECPG. Then, a host variable with a type that can be mapped to the element type should be used. For example, if a column type is array of integer, a host variable of type int can be used. Also if the element type is varchar or text, a host variable of type char[] or VARCHAR[] can be used.

Here is an example. Assume the following table:

CREATE TABLE t3 (
    ii integer[]
);

testdb=> SELECT * FROM t3;
     ii
-------------
 {1,2,3,4,5}
(1 row)

The following example program retrieves the 4th element of the array and stores it into a host variable of type int:

EXEC SQL BEGIN DECLARE SECTION;
int ii;
EXEC SQL END DECLARE SECTION;

EXEC SQL DECLARE cur1 CURSOR FOR SELECT ii[4] FROM t3;
EXEC SQL OPEN cur1;

EXEC SQL WHENEVER NOT FOUND DO BREAK;

while (1)
{
    EXEC SQL FETCH FROM cur1 INTO :ii ;
    printf("ii=%d\n", ii);
}

EXEC SQL CLOSE cur1;

This example shows the following result:

ii=4

To map multiple array elements to the multiple elements in an array type host variables each element of array column and each element of the host variable array have to be managed separately, for example:

EXEC SQL BEGIN DECLARE SECTION;
int ii_a[8];
EXEC SQL END DECLARE SECTION;

EXEC SQL DECLARE cur1 CURSOR FOR SELECT ii[1], ii[2], ii[3], ii[4] FROM t3;
EXEC SQL OPEN cur1;

EXEC SQL WHENEVER NOT FOUND DO BREAK;

while (1)
{
    EXEC SQL FETCH FROM cur1 INTO :ii_a[0], :ii_a[1], :ii_a[2], :ii_a[3];
    ...
}

Note again that

EXEC SQL BEGIN DECLARE SECTION;
int ii_a[8];
EXEC SQL END DECLARE SECTION;

EXEC SQL DECLARE cur1 CURSOR FOR SELECT ii FROM t3;
EXEC SQL OPEN cur1;

EXEC SQL WHENEVER NOT FOUND DO BREAK;

while (1)
{
    /* WRONG */
    EXEC SQL FETCH FROM cur1 INTO :ii_a;
    ...
}

would not work correctly in this case, because you cannot map an array type column to an array host variable directly.

Another workaround is to store arrays in their external string representation in host variables of type char[] or VARCHAR[]. For more details about this representation, see Section 8.15.2. Note that this means that the array cannot be accessed naturally as an array in the host program (without further processing that parses the text representation).

36.4.5.2. Composite Types

Composite types are not directly supported in ECPG, but an easy workaround is possible. The available workarounds are similar to the ones described for arrays above: Either access each attribute separately or use the external string representation.

For the following examples, assume the following type and table:

CREATE TYPE comp_t AS (intval integer, textval varchar(32));
CREATE TABLE t4 (compval comp_t);
INSERT INTO t4 VALUES ( (256, 'PostgreSQL') );

The most obvious solution is to access each attribute separately. The following program retrieves data from the example table by selecting each attribute of the type comp_t separately:

EXEC SQL BEGIN DECLARE SECTION;
int intval;
varchar textval[33];
EXEC SQL END DECLARE SECTION;

/* Put each element of the composite type column in the SELECT list. */
EXEC SQL DECLARE cur1 CURSOR FOR SELECT (compval).intval, (compval).textval FROM t4;
EXEC SQL OPEN cur1;

EXEC SQL WHENEVER NOT FOUND DO BREAK;

while (1)
{
    /* Fetch each element of the composite type column into host variables. */
    EXEC SQL FETCH FROM cur1 INTO :intval, :textval;

    printf("intval=%d, textval=%s\n", intval, textval.arr);
}

EXEC SQL CLOSE cur1;

To enhance this example, the host variables to store values in the FETCH command can be gathered into one structure. For more details about the host variable in the structure form, see Section 36.4.4.3.2. To switch to the structure, the example can be modified as below. The two host variables, intval and textval, become members of the comp_t structure, and the structure is specified on the FETCH command.

EXEC SQL BEGIN DECLARE SECTION;
typedef struct
{
    int intval;
    varchar textval[33];
} comp_t;

comp_t compval;
EXEC SQL END DECLARE SECTION;

/* Put each element of the composite type column in the SELECT list. */
EXEC SQL DECLARE cur1 CURSOR FOR SELECT (compval).intval, (compval).textval FROM t4;
EXEC SQL OPEN cur1;

EXEC SQL WHENEVER NOT FOUND DO BREAK;

while (1)
{
    /* Put all values in the SELECT list into one structure. */
    EXEC SQL FETCH FROM cur1 INTO :compval;

    printf("intval=%d, textval=%s\n", compval.intval, compval.textval.arr);
}

EXEC SQL CLOSE cur1;

Although a structure is used in the FETCH command, the attribute names in the SELECT clause are specified one by one. This can be enhanced by using a * to ask for all attributes of the composite type value.

...
EXEC SQL DECLARE cur1 CURSOR FOR SELECT (compval).* FROM t4;
EXEC SQL OPEN cur1;

EXEC SQL WHENEVER NOT FOUND DO BREAK;

while (1)
{
    /* Put all values in the SELECT list into one structure. */
    EXEC SQL FETCH FROM cur1 INTO :compval;

    printf("intval=%d, textval=%s\n", compval.intval, compval.textval.arr);
}
...

This way, composite types can be mapped into structures almost seamlessly, even though ECPG does not understand the composite type itself.

Finally, it is also possible to store composite type values in their external string representation in host variables of type char[] or VARCHAR[]. But that way, it is not easily possible to access the fields of the value from the host program.

36.4.5.3. User-Defined Base Types

New user-defined base types are not directly supported by ECPG. You can use the external string representation and host variables of type char[] or VARCHAR[], and this solution is indeed appropriate and sufficient for many types.

Here is an example using the data type complex from the example in Section 38.13. The external string representation of that type is (%f,%f), which is defined in the functions complex_in() and complex_out() functions in Section 38.13. The following example inserts the complex type values (1,1) and (3,3) into the columns a and b, and select them from the table after that.

EXEC SQL BEGIN DECLARE SECTION;
    varchar a[64];
    varchar b[64];
EXEC SQL END DECLARE SECTION;

    EXEC SQL INSERT INTO test_complex VALUES ('(1,1)', '(3,3)');

    EXEC SQL DECLARE cur1 CURSOR FOR SELECT a, b FROM test_complex;
    EXEC SQL OPEN cur1;

    EXEC SQL WHENEVER NOT FOUND DO BREAK;

    while (1)
    {
        EXEC SQL FETCH FROM cur1 INTO :a, :b;
        printf("a=%s, b=%s\n", a.arr, b.arr);
    }

    EXEC SQL CLOSE cur1;

This example shows following result:

a=(1,1), b=(3,3)

Another workaround is avoiding the direct use of the user-defined types in ECPG and instead create a function or cast that converts between the user-defined type and a primitive type that ECPG can handle. Note, however, that type casts, especially implicit ones, should be introduced into the type system very carefully.

For example,

CREATE FUNCTION create_complex(r double, i double) RETURNS complex
LANGUAGE SQL
IMMUTABLE
AS $$ SELECT $1 * complex '(1,0')' + $2 * complex '(0,1)' $$;

After this definition, the following

EXEC SQL BEGIN DECLARE SECTION;
double a, b, c, d;
EXEC SQL END DECLARE SECTION;

a = 1;
b = 2;
c = 3;
d = 4;

EXEC SQL INSERT INTO test_complex VALUES (create_complex(:a, :b), create_complex(:c, :d));

has the same effect as

EXEC SQL INSERT INTO test_complex VALUES ('(1,2)', '(3,4)');

36.4.6. Indicators

The examples above do not handle null values. In fact, the retrieval examples will raise an error if they fetch a null value from the database. To be able to pass null values to the database or retrieve null values from the database, you need to append a second host variable specification to each host variable that contains data. This second host variable is called the indicator and contains a flag that tells whether the datum is null, in which case the value of the real host variable is ignored. Here is an example that handles the retrieval of null values correctly:

EXEC SQL BEGIN DECLARE SECTION;
VARCHAR val;
int val_ind;
EXEC SQL END DECLARE SECTION:

 ...

EXEC SQL SELECT b INTO :val :val_ind FROM test1;

The indicator variable val_ind will be zero if the value was not null, and it will be negative if the value was null.

The indicator has another function: if the indicator value is positive, it means that the value is not null, but it was truncated when it was stored in the host variable.

If the argument -r no_indicator is passed to the preprocessor ecpg, it works in “no-indicator” mode. In no-indicator mode, if no indicator variable is specified, null values are signaled (on input and output) for character string types as empty string and for integer types as the lowest possible value for type (for example, INT_MIN for int).

Prev | Up | Next
36.3. Running SQL Commands | Home | 36.5. Dynamic SQL


36.5. Dynamic SQL
Prev Up Chapter 36. ECPG — Embedded SQL in C Home Next

36.5. Dynamic SQL

36.5.1. Executing Statements without a Result Set
36.5.2. Executing a Statement with Input Parameters
36.5.3. Executing a Statement with a Result Set

In many cases, the particular SQL statements that an application has to execute are known at the time the application is written. In some cases, however, the SQL statements are composed at run time or provided by an external source. In these cases you cannot embed the SQL statements directly into the C source code, but there is a facility that allows you to call arbitrary SQL statements that you provide in a string variable.

36.5.1. Executing Statements without a Result Set

The simplest way to execute an arbitrary SQL statement is to use the command EXECUTE IMMEDIATE. For example:

EXEC SQL BEGIN DECLARE SECTION;
const char *stmt = "CREATE TABLE test1 (...);";
EXEC SQL END DECLARE SECTION;

EXEC SQL EXECUTE IMMEDIATE :stmt;

EXECUTE IMMEDIATE can be used for SQL statements that do not return a result set (e.g., DDL, INSERT, UPDATE, DELETE). You cannot execute statements that retrieve data (e.g., SELECT) this way. The next section describes how to do that.

36.5.2. Executing a Statement with Input Parameters

A more powerful way to execute arbitrary SQL statements is to prepare them once and execute the prepared statement as often as you like. It is also possible to prepare a generalized version of a statement and then execute specific versions of it by substituting parameters. When preparing the statement, write question marks where you want to substitute parameters later. For example:

EXEC SQL BEGIN DECLARE SECTION;
const char *stmt = "INSERT INTO test1 VALUES(?, ?);";
EXEC SQL END DECLARE SECTION;

EXEC SQL PREPARE mystmt FROM :stmt;
 ...
EXEC SQL EXECUTE mystmt USING 42, 'foobar';

When you don't need the prepared statement anymore, you should deallocate it:

EXEC SQL DEALLOCATE PREPARE name;

36.5.3. Executing a Statement with a Result Set

To execute an SQL statement with a single result row, EXECUTE can be used. To save the result, add an INTO clause.

EXEC SQL BEGIN DECLARE SECTION;
const char *stmt = "SELECT a, b, c FROM test1 WHERE a > ?";
int v1, v2;
VARCHAR v3[50];
EXEC SQL END DECLARE SECTION;

EXEC SQL PREPARE mystmt FROM :stmt;
 ...
EXEC SQL EXECUTE mystmt INTO :v1, :v2, :v3 USING 37;

An EXECUTE command can have an INTO clause, a USING clause, both, or neither.

If a query is expected to return more than one result row, a cursor should be used, as in the following example. (See Section 36.3.2 for more details about the cursor.)

EXEC SQL BEGIN DECLARE SECTION;
char dbaname[128];
char datname[128];
char *stmt = "SELECT u.usename as dbaname, d.datname "
             "  FROM pg_database d, pg_user u "
             "  WHERE d.datdba = u.usesysid";
EXEC SQL END DECLARE SECTION;

EXEC SQL CONNECT TO testdb AS con1 USER testuser;
EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); EXEC SQL COMMIT;

EXEC SQL PREPARE stmt1 FROM :stmt;

EXEC SQL DECLARE cursor1 CURSOR FOR stmt1;
EXEC SQL OPEN cursor1;

EXEC SQL WHENEVER NOT FOUND DO BREAK;

while (1)
{
    EXEC SQL FETCH cursor1 INTO :dbaname,:datname;
    printf("dbaname=%s, datname=%s\n", dbaname, datname);
}

EXEC SQL CLOSE cursor1;

EXEC SQL COMMIT;
EXEC SQL DISCONNECT ALL;

Prev | Up | Next
36.4. Using Host Variables | Home | 36.6. pgtypes Library


36.6. pgtypes Library
Prev Up Chapter 36. ECPG — Embedded SQL in C Home Next

36.6. pgtypes Library

36.6.1. Character Strings
36.6.2. The numeric Type
36.6.3. The date Type
36.6.4. The timestamp Type
36.6.5. The interval Type
36.6.6. The decimal Type
36.6.7. errno Values of pgtypeslib
36.6.8. Special Constants of pgtypeslib

The pgtypes library maps PostgreSQL database types to C equivalents that can be used in C programs. It also offers functions to do basic calculations with those types within C, i.e., without the help of the PostgreSQL server. See the following example:

EXEC SQL BEGIN DECLARE SECTION;
   date date1;
   timestamp ts1, tsout;
   interval iv1;
   char *out;
EXEC SQL END DECLARE SECTION;

PGTYPESdate_today(&date1);
EXEC SQL SELECT started, duration INTO :ts1, :iv1 FROM datetbl WHERE d=:date1;
PGTYPEStimestamp_add_interval(&ts1, &iv1, &tsout);
out = PGTYPEStimestamp_to_asc(&tsout);
printf("Started + duration: %s\n", out);
PGTYPESchar_free(out);

36.6.1. Character Strings

Some functions such as PGTYPESnumeric_to_asc return a pointer to a freshly allocated character string. These results should be freed with PGTYPESchar_free instead of free. (This is important only on Windows, where memory allocation and release sometimes need to be done by the same library.)

36.6.2. The numeric Type

The numeric type offers to do calculations with arbitrary precision. See Section 8.1 for the equivalent type in the PostgreSQL server. Because of the arbitrary precision this variable needs to be able to expand and shrink dynamically. That's why you can only create numeric variables on the heap, by means of the PGTYPESnumeric_new and PGTYPESnumeric_free functions. The decimal type, which is similar but limited in precision, can be created on the stack as well as on the heap.

The following functions can be used to work with the numeric type:

PGTYPESnumeric_new

Request a pointer to a newly allocated numeric variable.

numeric *PGTYPESnumeric_new(void);

PGTYPESnumeric_free

Free a numeric type, release all of its memory.

void PGTYPESnumeric_free(numeric *var);

PGTYPESnumeric_from_asc

Parse a numeric type from its string notation.

numeric *PGTYPESnumeric_from_asc(char *str, char **endptr);

Valid formats are for example: -2, .794, +3.44, 592.49E07 or -32.84e-4. If the value could be parsed successfully, a valid pointer is returned, else the NULL pointer. At the moment ECPG always parses the complete string and so it currently does not support to store the address of the first invalid character in *endptr. You can safely set endptr to NULL.

PGTYPESnumeric_to_asc

Returns a pointer to a string allocated by malloc that contains the string representation of the numeric type num.

char *PGTYPESnumeric_to_asc(numeric *num, int dscale);

The numeric value will be printed with dscale decimal digits, with rounding applied if necessary. The result must be freed with PGTYPESchar_free().

PGTYPESnumeric_add

Add two numeric variables into a third one.

int PGTYPESnumeric_add(numeric *var1, numeric *var2, numeric *result);

The function adds the variables var1 and var2 into the result variable result. The function returns 0 on success and -1 in case of error.

PGTYPESnumeric_sub

Subtract two numeric variables and return the result in a third one.

int PGTYPESnumeric_sub(numeric *var1, numeric *var2, numeric *result);

The function subtracts the variable var2 from the variable var1. The result of the operation is stored in the variable result. The function returns 0 on success and -1 in case of error.

PGTYPESnumeric_mul

Multiply two numeric variables and return the result in a third one.

int PGTYPESnumeric_mul(numeric *var1, numeric *var2, numeric *result);

The function multiplies the variables var1 and var2. The result of the operation is stored in the variable result. The function returns 0 on success and -1 in case of error.

PGTYPESnumeric_div

Divide two numeric variables and return the result in a third one.

int PGTYPESnumeric_div(numeric *var1, numeric *var2, numeric *result);

The function divides the variables var1 by var2. The result of the operation is stored in the variable result. The function returns 0 on success and -1 in case of error.

PGTYPESnumeric_cmp

Compare two numeric variables.

int PGTYPESnumeric_cmp(numeric *var1, numeric *var2)

This function compares two numeric variables. In case of error, INT_MAX is returned. On success, the function returns one of three possible results:

  • 1, if var1 is bigger than var2

  • -1, if var1 is smaller than var2

  • 0, if var1 and var2 are equal

PGTYPESnumeric_from_int

Convert an int variable to a numeric variable.

int PGTYPESnumeric_from_int(signed int int_val, numeric *var);

This function accepts a variable of type signed int and stores it in the numeric variable var. Upon success, 0 is returned and -1 in case of a failure.

PGTYPESnumeric_from_long

Convert a long int variable to a numeric variable.

int PGTYPESnumeric_from_long(signed long int long_val, numeric *var);

This function accepts a variable of type signed long int and stores it in the numeric variable var. Upon success, 0 is returned and -1 in case of a failure.

PGTYPESnumeric_copy

Copy over one numeric variable into another one.

int PGTYPESnumeric_copy(numeric *src, numeric *dst);

This function copies over the value of the variable that src points to into the variable that dst points to. It returns 0 on success and -1 if an error occurs.

PGTYPESnumeric_from_double

Convert a variable of type double to a numeric.

int  PGTYPESnumeric_from_double(double d, numeric *dst);

This function accepts a variable of type double and stores the result in the variable that dst points to. It returns 0 on success and -1 if an error occurs.

PGTYPESnumeric_to_double

Convert a variable of type numeric to double.

int PGTYPESnumeric_to_double(numeric *nv, double *dp)

The function converts the numeric value from the variable that nv points to into the double variable that dp points to. It returns 0 on success and -1 if an error occurs, including overflow. On overflow, the global variable errno will be set to PGTYPES_NUM_OVERFLOW additionally.

PGTYPESnumeric_to_int

Convert a variable of type numeric to int.

int PGTYPESnumeric_to_int(numeric *nv, int *ip);

The function converts the numeric value from the variable that nv points to into the integer variable that ip points to. It returns 0 on success and -1 if an error occurs, including overflow. On overflow, the global variable errno will be set to PGTYPES_NUM_OVERFLOW additionally.

PGTYPESnumeric_to_long

Convert a variable of type numeric to long.

int PGTYPESnumeric_to_long(numeric *nv, long *lp);

The function converts the numeric value from the variable that nv points to into the long integer variable that lp points to. It returns 0 on success and -1 if an error occurs, including overflow. On overflow, the global variable errno will be set to PGTYPES_NUM_OVERFLOW additionally.

PGTYPESnumeric_to_decimal

Convert a variable of type numeric to decimal.

int PGTYPESnumeric_to_decimal(numeric *src, decimal *dst);

The function converts the numeric value from the variable that src points to into the decimal variable that dst points to. It returns 0 on success and -1 if an error occurs, including overflow. On overflow, the global variable errno will be set to PGTYPES_NUM_OVERFLOW additionally.

PGTYPESnumeric_from_decimal

Convert a variable of type decimal to numeric.

int PGTYPESnumeric_from_decimal(decimal *src, numeric *dst);

The function converts the decimal value from the variable that src points to into the numeric variable that dst points to. It returns 0 on success and -1 if an error occurs. Since the decimal type is implemented as a limited version of the numeric type, overflow cannot occur with this conversion.

36.6.3. The date Type

The date type in C enables your programs to deal with data of the SQL type date. See Section 8.5 for the equivalent type in the PostgreSQL server.

The following functions can be used to work with the date type:

PGTYPESdate_from_timestamp

Extract the date part from a timestamp.

date PGTYPESdate_from_timestamp(timestamp dt);

The function receives a timestamp as its only argument and returns the extracted date part from this timestamp.

PGTYPESdate_from_asc

Parse a date from its textual representation.

date PGTYPESdate_from_asc(char *str, char **endptr);

The function receives a C char* string str and a pointer to a C char* string endptr. At the moment ECPG always parses the complete string and so it currently does not support to store the address of the first invalid character in *endptr. You can safely set endptr to NULL.

Note that the function always assumes MDY-formatted dates and there is currently no variable to change that within ECPG.

Table 36.2 shows the allowed input formats.

Table 36.2. Valid Input Formats for PGTYPESdate_from_asc

Input | Result
January 8, 1999 | January 8, 1999
1999-01-08 | January 8, 1999
1/8/1999 | January 8, 1999
1/18/1999 | January 18, 1999
01/02/03 | February 1, 2003
1999-Jan-08 | January 8, 1999
Jan-08-1999 | January 8, 1999
08-Jan-1999 | January 8, 1999
99-Jan-08 | January 8, 1999
08-Jan-99 | January 8, 1999
08-Jan-06 | January 8, 2006
Jan-08-99 | January 8, 1999
19990108 | ISO 8601; January 8, 1999
990108 | ISO 8601; January 8, 1999
1999.008 | year and day of year
J2451187 | Julian day
January 8, 99 BC | year 99 before the Common Era

PGTYPESdate_to_asc

Return the textual representation of a date variable.

char *PGTYPESdate_to_asc(date dDate);

The function receives the date dDate as its only parameter. It will output the date in the form 1999-01-18, i.e., in the YYYY-MM-DD format. The result must be freed with PGTYPESchar_free().

PGTYPESdate_julmdy

Extract the values for the day, the month and the year from a variable of type date.

void PGTYPESdate_julmdy(date d, int *mdy);

The function receives the date d and a pointer to an array of 3 integer values mdy. The variable name indicates the sequential order: mdy[0] will be set to contain the number of the month, mdy[1] will be set to the value of the day and mdy[2] will contain the year.

PGTYPESdate_mdyjul

Create a date value from an array of 3 integers that specify the day, the month and the year of the date.

void PGTYPESdate_mdyjul(int *mdy, date *jdate);

The function receives the array of the 3 integers (mdy) as its first argument and as its second argument a pointer to a variable of type date that should hold the result of the operation.

PGTYPESdate_dayofweek

Return a number representing the day of the week for a date value.

int PGTYPESdate_dayofweek(date d);

The function receives the date variable d as its only argument and returns an integer that indicates the day of the week for this date.

  • 0 - Sunday

  • 1 - Monday

  • 2 - Tuesday

  • 3 - Wednesday

  • 4 - Thursday

  • 5 - Friday

  • 6 - Saturday

PGTYPESdate_today

Get the current date.

void PGTYPESdate_today(date *d);

The function receives a pointer to a date variable (d) that it sets to the current date.

PGTYPESdate_fmt_asc

Convert a variable of type date to its textual representation using a format mask.

int PGTYPESdate_fmt_asc(date dDate, char *fmtstring, char *outbuf);

The function receives the date to convert (dDate), the format mask (fmtstring) and the string that will hold the textual representation of the date (outbuf).

On success, 0 is returned and a negative value if an error occurred.

The following literals are the field specifiers you can use:

  • dd - The number of the day of the month.

  • mm - The number of the month of the year.

  • yy - The number of the year as a two digit number.

  • yyyy - The number of the year as a four digit number.

  • ddd - The name of the day (abbreviated).

  • mmm - The name of the month (abbreviated).

All other characters are copied 1:1 to the output string.

Table 36.3 indicates a few possible formats. This will give you an idea of how to use this function. All output lines are based on the same date: November 23, 1959.

Table 36.3. Valid Input Formats for PGTYPESdate_fmt_asc

Format | Result
mmddyy | 112359
ddmmyy | 231159
yymmdd | 591123
yy/mm/dd | 59/11/23
yy mm dd | 59 11 23
yy.mm.dd | 59.11.23
.mm.yyyy.dd. | .11.1959.23.
mmm. dd, yyyy | Nov. 23, 1959
mmm dd yyyy | Nov 23 1959
yyyy dd mm | 1959 23 11
ddd, mmm. dd, yyyy | Mon, Nov. 23, 1959
(ddd) mmm. dd, yyyy | (Mon) Nov. 23, 1959

PGTYPESdate_defmt_asc

Use a format mask to convert a C char* string to a value of type date.

int PGTYPESdate_defmt_asc(date *d, char *fmt, char *str);

The function receives a pointer to the date value that should hold the result of the operation (d), the format mask to use for parsing the date (fmt) and the C char* string containing the textual representation of the date (str). The textual representation is expected to match the format mask. However you do not need to have a 1:1 mapping of the string to the format mask. The function only analyzes the sequential order and looks for the literals yy or yyyy that indicate the position of the year, mm to indicate the position of the month and dd to indicate the position of the day.

Table 36.4 indicates a few possible formats. This will give you an idea of how to use this function.

Table 36.4. Valid Input Formats for rdefmtdate

Format | String | Result
ddmmyy | 21-2-54 | 1954-02-21
ddmmyy | 2-12-54 | 1954-12-02
ddmmyy | 20111954 | 1954-11-20
ddmmyy | 130464 | 1964-04-13
mmm.dd.yyyy | MAR-12-1967 | 1967-03-12
yy/mm/dd | 1954, February 3rd | 1954-02-03
mmm.dd.yyyy | 041269 | 1969-04-12
yy/mm/dd | In the year 2525, in the month of July, mankind will be alive on the 28th day | 2525-07-28
dd-mm-yy | I said on the 28th of July in the year 2525 | 2525-07-28
mmm.dd.yyyy | 9/14/58 | 1958-09-14
yy/mm/dd | 47/03/29 | 1947-03-29
mmm.dd.yyyy | oct 28 1975 | 1975-10-28
mmddyy | Nov 14th, 1985 | 1985-11-14

36.6.4. The timestamp Type

The timestamp type in C enables your programs to deal with data of the SQL type timestamp. See Section 8.5 for the equivalent type in the PostgreSQL server.

The following functions can be used to work with the timestamp type:

PGTYPEStimestamp_from_asc

Parse a timestamp from its textual representation into a timestamp variable.

timestamp PGTYPEStimestamp_from_asc(char *str, char **endptr);

The function receives the string to parse (str) and a pointer to a C char* (endptr). At the moment ECPG always parses the complete string and so it currently does not support to store the address of the first invalid character in *endptr. You can safely set endptr to NULL.

The function returns the parsed timestamp on success. On error, PGTYPESInvalidTimestamp is returned and errno is set to PGTYPES_TS_BAD_TIMESTAMP. See PGTYPESInvalidTimestamp for important notes on this value.

In general, the input string can contain any combination of an allowed date specification, a whitespace character and an allowed time specification. Note that time zones are not supported by ECPG. It can parse them but does not apply any calculation as the PostgreSQL server does for example. Timezone specifiers are silently discarded.

Table 36.5 contains a few examples for input strings.

Table 36.5. Valid Input Formats for PGTYPEStimestamp_from_asc

Input | Result
1999-01-08 04:05:06 | 1999-01-08 04:05:06
January 8 04:05:06 1999 PST | 1999-01-08 04:05:06
1999-Jan-08 04:05:06.789-8 | 1999-01-08 04:05:06.789 (time zone specifier ignored)
J2451187 04:05-08:00 | 1999-01-08 04:05:00 (time zone specifier ignored)

PGTYPEStimestamp_to_asc

Converts a date to a C char* string.

char *PGTYPEStimestamp_to_asc(timestamp tstamp);

The function receives the timestamp tstamp as its only argument and returns an allocated string that contains the textual representation of the timestamp. The result must be freed with PGTYPESchar_free().

PGTYPEStimestamp_current

Retrieve the current timestamp.

void PGTYPEStimestamp_current(timestamp *ts);

The function retrieves the current timestamp and saves it into the timestamp variable that ts points to.

PGTYPEStimestamp_fmt_asc

Convert a timestamp variable to a C char* using a format mask.

int PGTYPEStimestamp_fmt_asc(timestamp *ts, char *output, int str_len, char *fmtstr);

The function receives a pointer to the timestamp to convert as its first argument (ts), a pointer to the output buffer (output), the maximal length that has been allocated for the output buffer (str_len) and the format mask to use for the conversion (fmtstr).

Upon success, the function returns 0 and a negative value if an error occurred.

You can use the following format specifiers for the format mask. The format specifiers are the same ones that are used in the strftime function in libc. Any non-format specifier will be copied into the output buffer.

  • %A - is replaced by national representation of the full weekday name.

  • %a - is replaced by national representation of the abbreviated weekday name.

  • %B - is replaced by national representation of the full month name.

  • %b - is replaced by national representation of the abbreviated month name.

  • %C - is replaced by (year / 100) as decimal number; single digits are preceded by a zero.

  • %c - is replaced by national representation of time and date.

  • %D - is equivalent to %m/%d/%y.

  • %d - is replaced by the day of the month as a decimal number (01–31).

  • %E* %O* - POSIX locale extensions. The sequences %Ec %EC %Ex %EX %Ey %EY %Od %Oe %OH %OI %Om %OM %OS %Ou %OU %OV %Ow %OW %Oy are supposed to provide alternative representations.

    Additionally %OB implemented to represent alternative months names (used standalone, without day mentioned).

  • %e - is replaced by the day of month as a decimal number (1–31); single digits are preceded by a blank.

  • %F - is equivalent to %Y-%m-%d.

  • %G - is replaced by a year as a decimal number with century. This year is the one that contains the greater part of the week (Monday as the first day of the week).

  • %g - is replaced by the same year as in %G, but as a decimal number without century (00–99).

  • %H - is replaced by the hour (24-hour clock) as a decimal number (00–23).

  • %h - the same as %b.

  • %I - is replaced by the hour (12-hour clock) as a decimal number (01–12).

  • %j - is replaced by the day of the year as a decimal number (001–366).

  • %k - is replaced by the hour (24-hour clock) as a decimal number (0–23); single digits are preceded by a blank.

  • %l - is replaced by the hour (12-hour clock) as a decimal number (1–12); single digits are preceded by a blank.

  • %M - is replaced by the minute as a decimal number (00–59).

  • %m - is replaced by the month as a decimal number (01–12).

  • %n - is replaced by a newline.

  • %O* - the same as %E*.

  • %p - is replaced by national representation of either “ante meridiem” or “post meridiem” as appropriate.

  • %R - is equivalent to %H:%M.

  • %r - is equivalent to %I:%M:%S %p.

  • %S - is replaced by the second as a decimal number (00–60).

  • %s - is replaced by the number of seconds since the Epoch, UTC.

  • %T - is equivalent to %H:%M:%S

  • %t - is replaced by a tab.

  • %U - is replaced by the week number of the year (Sunday as the first day of the week) as a decimal number (00–53).

  • %u - is replaced by the weekday (Monday as the first day of the week) as a decimal number (1–7).

  • %V - is replaced by the week number of the year (Monday as the first day of the week) as a decimal number (01–53). If the week containing January 1 has four or more days in the new year, then it is week 1; otherwise it is the last week of the previous year, and the next week is week 1.

  • %v - is equivalent to %e-%b-%Y.

  • %W - is replaced by the week number of the year (Monday as the first day of the week) as a decimal number (00–53).

  • %w - is replaced by the weekday (Sunday as the first day of the week) as a decimal number (0–6).

  • %X - is replaced by national representation of the time.

  • %x - is replaced by national representation of the date.

  • %Y - is replaced by the year with century as a decimal number.

  • %y - is replaced by the year without century as a decimal number (00–99).

  • %Z - is replaced by the time zone name.

  • %z - is replaced by the time zone offset from UTC; a leading plus sign stands for east of UTC, a minus sign for west of UTC, hours and minutes follow with two digits each and no delimiter between them (common form for RFC 822 date headers).

  • %+ - is replaced by national representation of the date and time.

  • %-* - GNU libc extension. Do not do any padding when performing numerical outputs.

  • $_* - GNU libc extension. Explicitly specify space for padding.

  • %0* - GNU libc extension. Explicitly specify zero for padding.

  • %% - is replaced by %.

PGTYPEStimestamp_sub

Subtract one timestamp from another one and save the result in a variable of type interval.

int PGTYPEStimestamp_sub(timestamp *ts1, timestamp *ts2, interval *iv);

The function will subtract the timestamp variable that ts2 points to from the timestamp variable that ts1 points to and will store the result in the interval variable that iv points to.

Upon success, the function returns 0 and a negative value if an error occurred.

PGTYPEStimestamp_defmt_asc

Parse a timestamp value from its textual representation using a formatting mask.

int PGTYPEStimestamp_defmt_asc(char *str, char *fmt, timestamp *d);

The function receives the textual representation of a timestamp in the variable str as well as the formatting mask to use in the variable fmt. The result will be stored in the variable that d points to.

If the formatting mask fmt is NULL, the function will fall back to the default formatting mask which is %Y-%m-%d %H:%M:%S.

This is the reverse function to PGTYPEStimestamp_fmt_asc. See the documentation there in order to find out about the possible formatting mask entries.

PGTYPEStimestamp_add_interval

Add an interval variable to a timestamp variable.

int PGTYPEStimestamp_add_interval(timestamp *tin, interval *span, timestamp *tout);

The function receives a pointer to a timestamp variable tin and a pointer to an interval variable span. It adds the interval to the timestamp and saves the resulting timestamp in the variable that tout points to.

Upon success, the function returns 0 and a negative value if an error occurred.

PGTYPEStimestamp_sub_interval

Subtract an interval variable from a timestamp variable.

int PGTYPEStimestamp_sub_interval(timestamp *tin, interval *span, timestamp *tout);

The function subtracts the interval variable that span points to from the timestamp variable that tin points to and saves the result into the variable that tout points to.

Upon success, the function returns 0 and a negative value if an error occurred.

36.6.5. The interval Type

The interval type in C enables your programs to deal with data of the SQL type interval. See Section 8.5 for the equivalent type in the PostgreSQL server.

The following functions can be used to work with the interval type:

PGTYPESinterval_new

Return a pointer to a newly allocated interval variable.

interval *PGTYPESinterval_new(void);

PGTYPESinterval_free

Release the memory of a previously allocated interval variable.

void PGTYPESinterval_free(interval *intvl);

PGTYPESinterval_from_asc

Parse an interval from its textual representation.

interval *PGTYPESinterval_from_asc(char *str, char **endptr);

The function parses the input string str and returns a pointer to an allocated interval variable. At the moment ECPG always parses the complete string and so it currently does not support to store the address of the first invalid character in *endptr. You can safely set endptr to NULL.

PGTYPESinterval_to_asc

Convert a variable of type interval to its textual representation.

char *PGTYPESinterval_to_asc(interval *span);

The function converts the interval variable that span points to into a C char*. The output looks like this example: @ 1 day 12 hours 59 mins 10 secs. The result must be freed with PGTYPESchar_free().

PGTYPESinterval_copy

Copy a variable of type interval.

int PGTYPESinterval_copy(interval *intvlsrc, interval *intvldest);

The function copies the interval variable that intvlsrc points to into the variable that intvldest points to. Note that you need to allocate the memory for the destination variable before.

36.6.6. The decimal Type

The decimal type is similar to the numeric type. However it is limited to a maximum precision of 30 significant digits. In contrast to the numeric type which can be created on the heap only, the decimal type can be created either on the stack or on the heap (by means of the functions PGTYPESdecimal_new and PGTYPESdecimal_free). There are a lot of other functions that deal with the decimal type in the Informix compatibility mode described in Section 36.15.

The following functions can be used to work with the decimal type and are not only contained in the libcompat library.

PGTYPESdecimal_new

Request a pointer to a newly allocated decimal variable.

decimal *PGTYPESdecimal_new(void);

PGTYPESdecimal_free

Free a decimal type, release all of its memory.

void PGTYPESdecimal_free(decimal *var);

36.6.7. errno Values of pgtypeslib

PGTYPES_NUM_BAD_NUMERIC

An argument should contain a numeric variable (or point to a numeric variable) but in fact its in-memory representation was invalid.

PGTYPES_NUM_OVERFLOW

An overflow occurred. Since the numeric type can deal with almost arbitrary precision, converting a numeric variable into other types might cause overflow.

PGTYPES_NUM_UNDERFLOW

An underflow occurred. Since the numeric type can deal with almost arbitrary precision, converting a numeric variable into other types might cause underflow.

PGTYPES_NUM_DIVIDE_ZERO

A division by zero has been attempted.

PGTYPES_DATE_BAD_DATE

An invalid date string was passed to the PGTYPESdate_from_asc function.

PGTYPES_DATE_ERR_EARGS

Invalid arguments were passed to the PGTYPESdate_defmt_asc function.

PGTYPES_DATE_ERR_ENOSHORTDATE

An invalid token in the input string was found by the PGTYPESdate_defmt_asc function.

PGTYPES_INTVL_BAD_INTERVAL

An invalid interval string was passed to the PGTYPESinterval_from_asc function, or an invalid interval value was passed to the PGTYPESinterval_to_asc function.

PGTYPES_DATE_ERR_ENOTDMY

There was a mismatch in the day/month/year assignment in the PGTYPESdate_defmt_asc function.

PGTYPES_DATE_BAD_DAY

An invalid day of the month value was found by the PGTYPESdate_defmt_asc function.

PGTYPES_DATE_BAD_MONTH

An invalid month value was found by the PGTYPESdate_defmt_asc function.

PGTYPES_TS_BAD_TIMESTAMP

An invalid timestamp string pass passed to the PGTYPEStimestamp_from_asc function, or an invalid timestamp value was passed to the PGTYPEStimestamp_to_asc function.

PGTYPES_TS_ERR_EINFTIME

An infinite timestamp value was encountered in a context that cannot handle it.

36.6.8. Special Constants of pgtypeslib

PGTYPESInvalidTimestamp

A value of type timestamp representing an invalid time stamp. This is returned by the function PGTYPEStimestamp_from_asc on parse error. Note that due to the internal representation of the timestamp data type, PGTYPESInvalidTimestamp is also a valid timestamp at the same time. It is set to 1899-12-31 23:59:59. In order to detect errors, make sure that your application does not only test for PGTYPESInvalidTimestamp but also for errno != 0 after each call to PGTYPEStimestamp_from_asc.

Prev | Up | Next
36.5. Dynamic SQL | Home | 36.7. Using Descriptor Areas


36.7. Using Descriptor Areas
Prev Up Chapter 36. ECPG — Embedded SQL in C Home Next

36.7. Using Descriptor Areas

36.7.1. Named SQL Descriptor Areas
36.7.2. SQLDA Descriptor Areas

An SQL descriptor area is a more sophisticated method for processing the result of a SELECT, FETCH or a DESCRIBE statement. An SQL descriptor area groups the data of one row of data together with metadata items into one data structure. The metadata is particularly useful when executing dynamic SQL statements, where the nature of the result columns might not be known ahead of time. PostgreSQL provides two ways to use Descriptor Areas: the named SQL Descriptor Areas and the C-structure SQLDAs.

36.7.1. Named SQL Descriptor Areas

A named SQL descriptor area consists of a header, which contains information concerning the entire descriptor, and one or more item descriptor areas, which basically each describe one column in the result row.

Before you can use an SQL descriptor area, you need to allocate one:

EXEC SQL ALLOCATE DESCRIPTOR identifier;

The identifier serves as the “variable name” of the descriptor area. When you don't need the descriptor anymore, you should deallocate it:

EXEC SQL DEALLOCATE DESCRIPTOR identifier;

To use a descriptor area, specify it as the storage target in an INTO clause, instead of listing host variables:

EXEC SQL FETCH NEXT FROM mycursor INTO SQL DESCRIPTOR mydesc;

If the result set is empty, the Descriptor Area will still contain the metadata from the query, i.e., the field names.

For not yet executed prepared queries, the DESCRIBE statement can be used to get the metadata of the result set:

EXEC SQL BEGIN DECLARE SECTION;
char *sql_stmt = "SELECT * FROM table1";
EXEC SQL END DECLARE SECTION;

EXEC SQL PREPARE stmt1 FROM :sql_stmt;
EXEC SQL DESCRIBE stmt1 INTO SQL DESCRIPTOR mydesc;

Before PostgreSQL 9.0, the SQL keyword was optional, so using DESCRIPTOR and SQL DESCRIPTOR produced named SQL Descriptor Areas. Now it is mandatory, omitting the SQL keyword produces SQLDA Descriptor Areas, see Section 36.7.2.

In DESCRIBE and FETCH statements, the INTO and USING keywords can be used to similarly: they produce the result set and the metadata in a Descriptor Area.

Now how do you get the data out of the descriptor area? You can think of the descriptor area as a structure with named fields. To retrieve the value of a field from the header and store it into a host variable, use the following command:

EXEC SQL GET DESCRIPTOR name :hostvar = field;

Currently, there is only one header field defined: COUNT, which tells how many item descriptor areas exist (that is, how many columns are contained in the result). The host variable needs to be of an integer type. To get a field from the item descriptor area, use the following command:

EXEC SQL GET DESCRIPTOR name VALUE num :hostvar = field;

num can be a literal integer or a host variable containing an integer. Possible fields are:

CARDINALITY (integer)

number of rows in the result set

DATA

actual data item (therefore, the data type of this field depends on the query)

DATETIME_INTERVAL_CODE (integer)

When TYPE is 9, DATETIME_INTERVAL_CODE will have a value of 1 for DATE, 2 for TIME, 3 for TIMESTAMP, 4 for TIME WITH TIME ZONE, or 5 for TIMESTAMP WITH TIME ZONE.

DATETIME_INTERVAL_PRECISION (integer)

not implemented

INDICATOR (integer)

the indicator (indicating a null value or a value truncation)

KEY_MEMBER (integer)

not implemented

LENGTH (integer)

length of the datum in characters

NAME (string)

name of the column

NULLABLE (integer)

not implemented

OCTET_LENGTH (integer)

length of the character representation of the datum in bytes

PRECISION (integer)

precision (for type numeric)

RETURNED_LENGTH (integer)

length of the datum in characters

RETURNED_OCTET_LENGTH (integer)

length of the character representation of the datum in bytes

SCALE (integer)

scale (for type numeric)

TYPE (integer)

numeric code of the data type of the column

In EXECUTE, DECLARE and OPEN statements, the effect of the INTO and USING keywords are different. A Descriptor Area can also be manually built to provide the input parameters for a query or a cursor and USING SQL DESCRIPTOR name is the way to pass the input parameters into a parameterized query. The statement to build a named SQL Descriptor Area is below:

EXEC SQL SET DESCRIPTOR name VALUE num field = :hostvar;

PostgreSQL supports retrieving more that one record in one FETCH statement and storing the data in host variables in this case assumes that the variable is an array. E.g.:

EXEC SQL BEGIN DECLARE SECTION;
int id[5];
EXEC SQL END DECLARE SECTION;

EXEC SQL FETCH 5 FROM mycursor INTO SQL DESCRIPTOR mydesc;

EXEC SQL GET DESCRIPTOR mydesc VALUE 1 :id = DATA;

36.7.2. SQLDA Descriptor Areas

An SQLDA Descriptor Area is a C language structure which can be also used to get the result set and the metadata of a query. One structure stores one record from the result set.

EXEC SQL include sqlda.h;
sqlda_t         *mysqlda;

EXEC SQL FETCH 3 FROM mycursor INTO DESCRIPTOR mysqlda;

Note that the SQL keyword is omitted. The paragraphs about the use cases of the INTO and USING keywords in Section 36.7.1 also apply here with an addition. In a DESCRIBE statement the DESCRIPTOR keyword can be completely omitted if the INTO keyword is used:

EXEC SQL DESCRIBE prepared_statement INTO mysqlda;

The general flow of a program that uses SQLDA is:

  1. Prepare a query, and declare a cursor for it.

  2. Declare an SQLDA for the result rows.

  3. Declare an SQLDA for the input parameters, and initialize them (memory allocation, parameter settings).

  4. Open a cursor with the input SQLDA.

  5. Fetch rows from the cursor, and store them into an output SQLDA.

  6. Read values from the output SQLDA into the host variables (with conversion if necessary).

  7. Close the cursor.

  8. Free the memory area allocated for the input SQLDA.

36.7.2.1. SQLDA Data Structure

SQLDA uses three data structure types: sqlda_t, sqlvar_t, and struct sqlname.

Tip

PostgreSQL's SQLDA has a similar data structure to the one in IBM DB2 Universal Database, so some technical information on DB2's SQLDA could help understanding PostgreSQL's one better.

36.7.2.1.1. sqlda_t Structure

The structure type sqlda_t is the type of the actual SQLDA. It holds one record. And two or more sqlda_t structures can be connected in a linked list with the pointer in the desc_next field, thus representing an ordered collection of rows. So, when two or more rows are fetched, the application can read them by following the desc_next pointer in each sqlda_t node.

The definition of sqlda_t is:

struct sqlda_struct
{
    char            sqldaid[8];
    long            sqldabc;
    short           sqln;
    short           sqld;
    struct sqlda_struct *desc_next;
    struct sqlvar_struct sqlvar[1];
};

typedef struct sqlda_struct sqlda_t;

The meaning of the fields is:

sqldaid

It contains the literal string "SQLDA ".

sqldabc

It contains the size of the allocated space in bytes.

sqln

It contains the number of input parameters for a parameterized query in case it's passed into OPEN, DECLARE or EXECUTE statements using the USING keyword. In case it's used as output of SELECT, EXECUTE or FETCH statements, its value is the same as sqld statement

sqld

It contains the number of fields in a result set.

desc_next

If the query returns more than one record, multiple linked SQLDA structures are returned, and desc_next holds a pointer to the next entry in the list.

sqlvar

This is the array of the columns in the result set.

36.7.2.1.2. sqlvar_t Structure

The structure type sqlvar_t holds a column value and metadata such as type and length. The definition of the type is:

struct sqlvar_struct
{
    short          sqltype;
    short          sqllen;
    char          *sqldata;
    short         *sqlind;
    struct sqlname sqlname;
};

typedef struct sqlvar_struct sqlvar_t;

The meaning of the fields is:

sqltype

Contains the type identifier of the field. For values, see enum ECPGttype in ecpgtype.h.

sqllen

Contains the binary length of the field. e.g., 4 bytes for ECPGt_int.

sqldata

Points to the data. The format of the data is described in Section 36.4.4.

sqlind

Points to the null indicator. 0 means not null, -1 means null.

sqlname

The name of the field.

36.7.2.1.3. struct sqlname Structure

A struct sqlname structure holds a column name. It is used as a member of the sqlvar_t structure. The definition of the structure is:

#define NAMEDATALEN 64

struct sqlname
{
        short           length;
        char            data[NAMEDATALEN];
};

The meaning of the fields is:

length

Contains the length of the field name.

data

Contains the actual field name.

36.7.2.2. Retrieving a Result Set Using an SQLDA

The general steps to retrieve a query result set through an SQLDA are:

  1. Declare an sqlda_t structure to receive the result set.

  2. Execute FETCH/EXECUTE/DESCRIBE commands to process a query specifying the declared SQLDA.

  3. Check the number of records in the result set by looking at sqln, a member of the sqlda_t structure.

  4. Get the values of each column from sqlvar[0], sqlvar[1], etc., members of the sqlda_t structure.

  5. Go to next row (sqlda_t structure) by following the desc_next pointer, a member of the sqlda_t structure.

  6. Repeat above as you need.

Here is an example retrieving a result set through an SQLDA.

First, declare a sqlda_t structure to receive the result set.

sqlda_t *sqlda1;

Next, specify the SQLDA in a command. This is a FETCH command example.

EXEC SQL FETCH NEXT FROM cur1 INTO DESCRIPTOR sqlda1;

Run a loop following the linked list to retrieve the rows.

sqlda_t *cur_sqlda;

for (cur_sqlda = sqlda1;
     cur_sqlda != NULL;
     cur_sqlda = cur_sqlda->desc_next)
{
    ...
}

Inside the loop, run another loop to retrieve each column data (sqlvar_t structure) of the row.

for (i = 0; i < cur_sqlda->sqld; i++)
{
    sqlvar_t v = cur_sqlda->sqlvar[i];
    char *sqldata = v.sqldata;
    short sqllen  = v.sqllen;
    ...
}

To get a column value, check the sqltype value, a member of the sqlvar_t structure. Then, switch to an appropriate way, depending on the column type, to copy data from the sqlvar field to a host variable.

char var_buf[1024];

switch (v.sqltype)
{
    case ECPGt_char:
        memset(&var_buf, 0, sizeof(var_buf));
        memcpy(&var_buf, sqldata, (sizeof(var_buf) <= sqllen ? sizeof(var_buf) - 1 : sqllen));
        break;

    case ECPGt_int: /* integer */
        memcpy(&intval, sqldata, sqllen);
        snprintf(var_buf, sizeof(var_buf), "%d", intval);
        break;

    ...
}

36.7.2.3. Passing Query Parameters Using an SQLDA

The general steps to use an SQLDA to pass input parameters to a prepared query are:

  1. Create a prepared query (prepared statement)

  2. Declare an sqlda_t structure as an input SQLDA.

  3. Allocate memory area (as sqlda_t structure) for the input SQLDA.

  4. Set (copy) input values in the allocated memory.

  5. Open a cursor with specifying the input SQLDA.

Here is an example.

First, create a prepared statement.

EXEC SQL BEGIN DECLARE SECTION;
char query[1024] = "SELECT d.oid, * FROM pg_database d, pg_stat_database s WHERE d.oid = s.datid AND (d.datname = ? OR d.oid = ?)";
EXEC SQL END DECLARE SECTION;

EXEC SQL PREPARE stmt1 FROM :query;

Next, allocate memory for an SQLDA, and set the number of input parameters in sqln, a member variable of the sqlda_t structure. When two or more input parameters are required for the prepared query, the application has to allocate additional memory space which is calculated by (nr. of params - 1) * sizeof(sqlvar_t). The example shown here allocates memory space for two input parameters.

sqlda_t *sqlda2;

sqlda2 = (sqlda_t *) malloc(sizeof(sqlda_t) + sizeof(sqlvar_t));
memset(sqlda2, 0, sizeof(sqlda_t) + sizeof(sqlvar_t));

sqlda2->sqln = 2; /* number of input variables */

After memory allocation, store the parameter values into the sqlvar[] array. (This is same array used for retrieving column values when the SQLDA is receiving a result set.) In this example, the input parameters are "postgres", having a string type, and 1, having an integer type.

sqlda2->sqlvar[0].sqltype = ECPGt_char;
sqlda2->sqlvar[0].sqldata = "postgres";
sqlda2->sqlvar[0].sqllen  = 8;

int intval = 1;
sqlda2->sqlvar[1].sqltype = ECPGt_int;
sqlda2->sqlvar[1].sqldata = (char *) &intval;
sqlda2->sqlvar[1].sqllen  = sizeof(intval);

By opening a cursor and specifying the SQLDA that was set up beforehand, the input parameters are passed to the prepared statement.

EXEC SQL OPEN cur1 USING DESCRIPTOR sqlda2;

Finally, after using input SQLDAs, the allocated memory space must be freed explicitly, unlike SQLDAs used for receiving query results.

free(sqlda2);

36.7.2.4. A Sample Application Using SQLDA

Here is an example program, which describes how to fetch access statistics of the databases, specified by the input parameters, from the system catalogs.

This application joins two system tables, pg_database and pg_stat_database on the database OID, and also fetches and shows the database statistics which are retrieved by two input parameters (a database postgres, and OID 1).

First, declare an SQLDA for input and an SQLDA for output.

EXEC SQL include sqlda.h;

sqlda_t *sqlda1; /* an output descriptor */
sqlda_t *sqlda2; /* an input descriptor  */

Next, connect to the database, prepare a statement, and declare a cursor for the prepared statement.

int
main(void)
{
    EXEC SQL BEGIN DECLARE SECTION;
    char query[1024] = "SELECT d.oid,* FROM pg_database d, pg_stat_database s WHERE d.oid=s.datid AND ( d.datname=? OR d.oid=? )";
    EXEC SQL END DECLARE SECTION;

    EXEC SQL CONNECT TO testdb AS con1 USER testuser;
    EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); EXEC SQL COMMIT;

    EXEC SQL PREPARE stmt1 FROM :query;
    EXEC SQL DECLARE cur1 CURSOR FOR stmt1;

Next, put some values in the input SQLDA for the input parameters. Allocate memory for the input SQLDA, and set the number of input parameters to sqln. Store type, value, and value length into sqltype, sqldata, and sqllen in the sqlvar structure.

    /* Create SQLDA structure for input parameters. */
    sqlda2 = (sqlda_t *) malloc(sizeof(sqlda_t) + sizeof(sqlvar_t));
    memset(sqlda2, 0, sizeof(sqlda_t) + sizeof(sqlvar_t));
    sqlda2->sqln = 2; /* number of input variables */

    sqlda2->sqlvar[0].sqltype = ECPGt_char;
    sqlda2->sqlvar[0].sqldata = "postgres";
    sqlda2->sqlvar[0].sqllen  = 8;

    intval = 1;
    sqlda2->sqlvar[1].sqltype = ECPGt_int;
    sqlda2->sqlvar[1].sqldata = (char *)&intval;
    sqlda2->sqlvar[1].sqllen  = sizeof(intval);

After setting up the input SQLDA, open a cursor with the input SQLDA.

    /* Open a cursor with input parameters. */
    EXEC SQL OPEN cur1 USING DESCRIPTOR sqlda2;

Fetch rows into the output SQLDA from the opened cursor. (Generally, you have to call FETCH repeatedly in the loop, to fetch all rows in the result set.)

    while (1)
    {
        sqlda_t *cur_sqlda;

        /* Assign descriptor to the cursor  */
        EXEC SQL FETCH NEXT FROM cur1 INTO DESCRIPTOR sqlda1;

Next, retrieve the fetched records from the SQLDA, by following the linked list of the sqlda_t structure.

    for (cur_sqlda = sqlda1 ;
         cur_sqlda != NULL ;
         cur_sqlda = cur_sqlda->desc_next)
    {
        ...

Read each columns in the first record. The number of columns is stored in sqld, the actual data of the first column is stored in sqlvar[0], both members of the sqlda_t structure.

        /* Print every column in a row. */
        for (i = 0; i < sqlda1->sqld; i++)
        {
            sqlvar_t v = sqlda1->sqlvar[i];
            char *sqldata = v.sqldata;
            short sqllen  = v.sqllen;

            strncpy(name_buf, v.sqlname.data, v.sqlname.length);
            name_buf[v.sqlname.length] = '\0';

Now, the column data is stored in the variable v. Copy every datum into host variables, looking at v.sqltype for the type of the column.

            switch (v.sqltype) {
                int intval;
                double doubleval;
                unsigned long long int longlongval;

                case ECPGt_char:
                    memset(&var_buf, 0, sizeof(var_buf));
                    memcpy(&var_buf, sqldata, (sizeof(var_buf) <= sqllen ? sizeof(var_buf)-1 : sqllen));
                    break;

                case ECPGt_int: /* integer */
                    memcpy(&intval, sqldata, sqllen);
                    snprintf(var_buf, sizeof(var_buf), "%d", intval);
                    break;

                ...

                default:
                    ...
            }

            printf("%s = %s (type: %d)\n", name_buf, var_buf, v.sqltype);
        }

Close the cursor after processing all of records, and disconnect from the database.

    EXEC SQL CLOSE cur1;
    EXEC SQL COMMIT;

    EXEC SQL DISCONNECT ALL;

The whole program is shown in Example 36.1.

Example 36.1. Example SQLDA Program

#include <stdlib.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>

EXEC SQL include sqlda.h;

sqlda_t *sqlda1; /* descriptor for output */
sqlda_t *sqlda2; /* descriptor for input */

EXEC SQL WHENEVER NOT FOUND DO BREAK;
EXEC SQL WHENEVER SQLERROR STOP;

int
main(void)
{
    EXEC SQL BEGIN DECLARE SECTION;
    char query[1024] = "SELECT d.oid,* FROM pg_database d, pg_stat_database s WHERE d.oid=s.datid AND ( d.datname=? OR d.oid=? )";

    int intval;
    unsigned long long int longlongval;
    EXEC SQL END DECLARE SECTION;

    EXEC SQL CONNECT TO uptimedb AS con1 USER uptime;
    EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); EXEC SQL COMMIT;

    EXEC SQL PREPARE stmt1 FROM :query;
    EXEC SQL DECLARE cur1 CURSOR FOR stmt1;

    /* Create an SQLDA structure for an input parameter */
    sqlda2 = (sqlda_t *)malloc(sizeof(sqlda_t) + sizeof(sqlvar_t));
    memset(sqlda2, 0, sizeof(sqlda_t) + sizeof(sqlvar_t));
    sqlda2->sqln = 2; /* a number of input variables */

    sqlda2->sqlvar[0].sqltype = ECPGt_char;
    sqlda2->sqlvar[0].sqldata = "postgres";
    sqlda2->sqlvar[0].sqllen  = 8;

    intval = 1;
    sqlda2->sqlvar[1].sqltype = ECPGt_int;
    sqlda2->sqlvar[1].sqldata = (char *) &intval;
    sqlda2->sqlvar[1].sqllen  = sizeof(intval);

    /* Open a cursor with input parameters. */
    EXEC SQL OPEN cur1 USING DESCRIPTOR sqlda2;

    while (1)
    {
        sqlda_t *cur_sqlda;

        /* Assign descriptor to the cursor  */
        EXEC SQL FETCH NEXT FROM cur1 INTO DESCRIPTOR sqlda1;

        for (cur_sqlda = sqlda1 ;
             cur_sqlda != NULL ;
             cur_sqlda = cur_sqlda->desc_next)
        {
            int i;
            char name_buf[1024];
            char var_buf[1024];

            /* Print every column in a row. */
            for (i=0 ; i<cur_sqlda->sqld ; i++)
            {
                sqlvar_t v = cur_sqlda->sqlvar[i];
                char *sqldata = v.sqldata;
                short sqllen  = v.sqllen;

                strncpy(name_buf, v.sqlname.data, v.sqlname.length);
                name_buf[v.sqlname.length] = '\0';

                switch (v.sqltype)
                {
                    case ECPGt_char:
                        memset(&var_buf, 0, sizeof(var_buf));
                        memcpy(&var_buf, sqldata, (sizeof(var_buf)<=sqllen ? sizeof(var_buf)-1 : sqllen) );
                        break;

                    case ECPGt_int: /* integer */
                        memcpy(&intval, sqldata, sqllen);
                        snprintf(var_buf, sizeof(var_buf), "%d", intval);
                        break;

                    case ECPGt_long_long: /* bigint */
                        memcpy(&longlongval, sqldata, sqllen);
                        snprintf(var_buf, sizeof(var_buf), "%lld", longlongval);
                        break;

                    default:
                    {
                        int i;
                        memset(var_buf, 0, sizeof(var_buf));
                        for (i = 0; i < sqllen; i++)
                        {
                            char tmpbuf[16];
                            snprintf(tmpbuf, sizeof(tmpbuf), "%02x ", (unsigned char) sqldata[i]);
                            strncat(var_buf, tmpbuf, sizeof(var_buf));
                        }
                    }
                        break;
                }

                printf("%s = %s (type: %d)\n", name_buf, var_buf, v.sqltype);
            }

            printf("\n");
        }
    }

    EXEC SQL CLOSE cur1;
    EXEC SQL COMMIT;

    EXEC SQL DISCONNECT ALL;

    return 0;
}

The output of this example should look something like the following (some numbers will vary).

oid = 1 (type: 1)
datname = template1 (type: 1)
datdba = 10 (type: 1)
encoding = 0 (type: 5)
datistemplate = t (type: 1)
datallowconn = t (type: 1)
datconnlimit = -1 (type: 5)
datlastsysoid = 11510 (type: 1)
datfrozenxid = 379 (type: 1)
dattablespace = 1663 (type: 1)
datconfig =  (type: 1)
datacl = {=c/uptime,uptime=CTc/uptime} (type: 1)
datid = 1 (type: 1)
datname = template1 (type: 1)
numbackends = 0 (type: 5)
xact_commit = 113606 (type: 9)
xact_rollback = 0 (type: 9)
blks_read = 130 (type: 9)
blks_hit = 7341714 (type: 9)
tup_returned = 38262679 (type: 9)
tup_fetched = 1836281 (type: 9)
tup_inserted = 0 (type: 9)
tup_updated = 0 (type: 9)
tup_deleted = 0 (type: 9)

oid = 11511 (type: 1)
datname = postgres (type: 1)
datdba = 10 (type: 1)
encoding = 0 (type: 5)
datistemplate = f (type: 1)
datallowconn = t (type: 1)
datconnlimit = -1 (type: 5)
datlastsysoid = 11510 (type: 1)
datfrozenxid = 379 (type: 1)
dattablespace = 1663 (type: 1)
datconfig =  (type: 1)
datacl =  (type: 1)
datid = 11511 (type: 1)
datname = postgres (type: 1)
numbackends = 0 (type: 5)
xact_commit = 221069 (type: 9)
xact_rollback = 18 (type: 9)
blks_read = 1176 (type: 9)
blks_hit = 13943750 (type: 9)
tup_returned = 77410091 (type: 9)
tup_fetched = 3253694 (type: 9)
tup_inserted = 0 (type: 9)
tup_updated = 0 (type: 9)
tup_deleted = 0 (type: 9)

Prev | Up | Next
36.6. pgtypes Library | Home | 36.8. Error Handling


36.8. Error Handling
Prev Up Chapter 36. ECPG — Embedded SQL in C Home Next

36.8. Error Handling

36.8.1. Setting Callbacks
36.8.2. sqlca
36.8.3. SQLSTATE vs. SQLCODE

This section describes how you can handle exceptional conditions and warnings in an embedded SQL program. There are two nonexclusive facilities for this.

  • Callbacks can be configured to handle warning and error conditions using the WHENEVER command.
  • Detailed information about the error or warning can be obtained from the sqlca variable.

36.8.1. Setting Callbacks

One simple method to catch errors and warnings is to set a specific action to be executed whenever a particular condition occurs. In general:

EXEC SQL WHENEVER condition action;

condition can be one of the following:

SQLERROR

The specified action is called whenever an error occurs during the execution of an SQL statement.

SQLWARNING

The specified action is called whenever a warning occurs during the execution of an SQL statement.

NOT FOUND

The specified action is called whenever an SQL statement retrieves or affects zero rows. (This condition is not an error, but you might be interested in handling it specially.)

action can be one of the following:

CONTINUE

This effectively means that the condition is ignored. This is the default.

GOTO label
GO TO label

Jump to the specified label (using a C goto statement).

SQLPRINT

Print a message to standard error. This is useful for simple programs or during prototyping. The details of the message cannot be configured.

STOP

Call exit(1), which will terminate the program.

DO BREAK

Execute the C statement break. This should only be used in loops or switch statements.

DO CONTINUE

Execute the C statement continue. This should only be used in loops statements. if executed, will cause the flow of control to return to the top of the loop.

CALL name (args)
DO name (args)

Call the specified C functions with the specified arguments. (This use is different from the meaning of CALL and DO in the normal PostgreSQL grammar.)

The SQL standard only provides for the actions CONTINUE and GOTO (and GO TO).

Here is an example that you might want to use in a simple program. It prints a simple message when a warning occurs and aborts the program when an error happens:

EXEC SQL WHENEVER SQLWARNING SQLPRINT;
EXEC SQL WHENEVER SQLERROR STOP;

The statement EXEC SQL WHENEVER is a directive of the SQL preprocessor, not a C statement. The error or warning actions that it sets apply to all embedded SQL statements that appear below the point where the handler is set, unless a different action was set for the same condition between the first EXEC SQL WHENEVER and the SQL statement causing the condition, regardless of the flow of control in the C program. So neither of the two following C program excerpts will have the desired effect:

/*
 * WRONG
 */
int main(int argc, char *argv[])
{
    ...
    if (verbose) {
        EXEC SQL WHENEVER SQLWARNING SQLPRINT;
    }
    ...
    EXEC SQL SELECT ...;
    ...
}

/*
 * WRONG
 */
int main(int argc, char *argv[])
{
    ...
    set_error_handler();
    ...
    EXEC SQL SELECT ...;
    ...
}

static void set_error_handler(void)
{
    EXEC SQL WHENEVER SQLERROR STOP;
}

36.8.2. sqlca

For more powerful error handling, the embedded SQL interface provides a global variable with the name sqlca (SQL communication area) that has the following structure:

struct
{
    char sqlcaid[8];
    long sqlabc;
    long sqlcode;
    struct
    {
        int sqlerrml;
        char sqlerrmc[SQLERRMC_LEN];
    } sqlerrm;
    char sqlerrp[8];
    long sqlerrd[6];
    char sqlwarn[8];
    char sqlstate[5];
} sqlca;

(In a multithreaded program, every thread automatically gets its own copy of sqlca. This works similarly to the handling of the standard C global variable errno.)

sqlca covers both warnings and errors. If multiple warnings or errors occur during the execution of a statement, then sqlca will only contain information about the last one.

If no error occurred in the last SQL statement, sqlca.sqlcode will be 0 and sqlca.sqlstate will be "00000". If a warning or error occurred, then sqlca.sqlcode will be negative and sqlca.sqlstate will be different from "00000". A positive sqlca.sqlcode indicates a harmless condition, such as that the last query returned zero rows. sqlcode and sqlstate are two different error code schemes; details appear below.

If the last SQL statement was successful, then sqlca.sqlerrd[1] contains the OID of the processed row, if applicable, and sqlca.sqlerrd[2] contains the number of processed or returned rows, if applicable to the command.

In case of an error or warning, sqlca.sqlerrm.sqlerrmc will contain a string that describes the error. The field sqlca.sqlerrm.sqlerrml contains the length of the error message that is stored in sqlca.sqlerrm.sqlerrmc (the result of strlen(), not really interesting for a C programmer). Note that some messages are too long to fit in the fixed-size sqlerrmc array; they will be truncated.

In case of a warning, sqlca.sqlwarn[2] is set to W. (In all other cases, it is set to something different from W.) If sqlca.sqlwarn[1] is set to W, then a value was truncated when it was stored in a host variable. sqlca.sqlwarn[0] is set to W if any of the other elements are set to indicate a warning.

The fields sqlcaid, sqlabc, sqlerrp, and the remaining elements of sqlerrd and sqlwarn currently contain no useful information.

The structure sqlca is not defined in the SQL standard, but is implemented in several other SQL database systems. The definitions are similar at the core, but if you want to write portable applications, then you should investigate the different implementations carefully.

Here is one example that combines the use of WHENEVER and sqlca, printing out the contents of sqlca when an error occurs. This is perhaps useful for debugging or prototyping applications, before installing a more “user-friendly” error handler.

EXEC SQL WHENEVER SQLERROR CALL print_sqlca();

void
print_sqlca()
{
    fprintf(stderr, "==== sqlca ====\n");
    fprintf(stderr, "sqlcode: %ld\n", sqlca.sqlcode);
    fprintf(stderr, "sqlerrm.sqlerrml: %d\n", sqlca.sqlerrm.sqlerrml);
    fprintf(stderr, "sqlerrm.sqlerrmc: %s\n", sqlca.sqlerrm.sqlerrmc);
    fprintf(stderr, "sqlerrd: %ld %ld %ld %ld %ld %ld\n", sqlca.sqlerrd[0],sqlca.sqlerrd[1],sqlca.sqlerrd[2],
                                                          sqlca.sqlerrd[3],sqlca.sqlerrd[4],sqlca.sqlerrd[5]);
    fprintf(stderr, "sqlwarn: %d %d %d %d %d %d %d %d\n", sqlca.sqlwarn[0], sqlca.sqlwarn[1], sqlca.sqlwarn[2],
                                                          sqlca.sqlwarn[3], sqlca.sqlwarn[4], sqlca.sqlwarn[5],
                                                          sqlca.sqlwarn[6], sqlca.sqlwarn[7]);
    fprintf(stderr, "sqlstate: %5s\n", sqlca.sqlstate);
    fprintf(stderr, "===============\n");
}

The result could look as follows (here an error due to a misspelled table name):

==== sqlca ====
sqlcode: -400
sqlerrm.sqlerrml: 49
sqlerrm.sqlerrmc: relation "pg_databasep" does not exist on line 38
sqlerrd: 0 0 0 0 0 0
sqlwarn: 0 0 0 0 0 0 0 0
sqlstate: 42P01
===============

36.8.3. SQLSTATE vs. SQLCODE

The fields sqlca.sqlstate and sqlca.sqlcode are two different schemes that provide error codes. Both are derived from the SQL standard, but SQLCODE has been marked deprecated in the SQL-92 edition of the standard and has been dropped in later editions. Therefore, new applications are strongly encouraged to use SQLSTATE.

SQLSTATE is a five-character array. The five characters contain digits or upper-case letters that represent codes of various error and warning conditions. SQLSTATE has a hierarchical scheme: the first two characters indicate the general class of the condition, the last three characters indicate a subclass of the general condition. A successful state is indicated by the code 00000. The SQLSTATE codes are for the most part defined in the SQL standard. The PostgreSQL server natively supports SQLSTATE error codes; therefore a high degree of consistency can be achieved by using this error code scheme throughout all applications. For further information see Appendix A.

SQLCODE, the deprecated error code scheme, is a simple integer. A value of 0 indicates success, a positive value indicates success with additional information, a negative value indicates an error. The SQL standard only defines the positive value +100, which indicates that the last command returned or affected zero rows, and no specific negative values. Therefore, this scheme can only achieve poor portability and does not have a hierarchical code assignment. Historically, the embedded SQL processor for PostgreSQL has assigned some specific SQLCODE values for its use, which are listed below with their numeric value and their symbolic name. Remember that these are not portable to other SQL implementations. To simplify the porting of applications to the SQLSTATE scheme, the corresponding SQLSTATE is also listed. There is, however, no one-to-one or one-to-many mapping between the two schemes (indeed it is many-to-many), so you should consult the global SQLSTATE listing in Appendix A in each case.

These are the assigned SQLCODE values:

0 (ECPG_NO_ERROR)

Indicates no error. (SQLSTATE 00000)

100 (ECPG_NOT_FOUND)

This is a harmless condition indicating that the last command retrieved or processed zero rows, or that you are at the end of the cursor. (SQLSTATE 02000)

When processing a cursor in a loop, you could use this code as a way to detect when to abort the loop, like this:

while (1)
{
    EXEC SQL FETCH ... ;
    if (sqlca.sqlcode == ECPG_NOT_FOUND)
        break;
}

But WHENEVER NOT FOUND DO BREAK effectively does this internally, so there is usually no advantage in writing this out explicitly.

-12 (ECPG_OUT_OF_MEMORY)

Indicates that your virtual memory is exhausted. The numeric value is defined as -ENOMEM. (SQLSTATE YE001)

-200 (ECPG_UNSUPPORTED)

Indicates the preprocessor has generated something that the library does not know about. Perhaps you are running incompatible versions of the preprocessor and the library. (SQLSTATE YE002)

-201 (ECPG_TOO_MANY_ARGUMENTS)

This means that the command specified more host variables than the command expected. (SQLSTATE 07001 or 07002)

-202 (ECPG_TOO_FEW_ARGUMENTS)

This means that the command specified fewer host variables than the command expected. (SQLSTATE 07001 or 07002)

-203 (ECPG_TOO_MANY_MATCHES)

This means a query has returned multiple rows but the statement was only prepared to store one result row (for example, because the specified variables are not arrays). (SQLSTATE 21000)

-204 (ECPG_INT_FORMAT)

The host variable is of type int and the datum in the database is of a different type and contains a value that cannot be interpreted as an int. The library uses strtol() for this conversion. (SQLSTATE 42804)

-205 (ECPG_UINT_FORMAT)

The host variable is of type unsigned int and the datum in the database is of a different type and contains a value that cannot be interpreted as an unsigned int. The library uses strtoul() for this conversion. (SQLSTATE 42804)

-206 (ECPG_FLOAT_FORMAT)

The host variable is of type float and the datum in the database is of another type and contains a value that cannot be interpreted as a float. The library uses strtod() for this conversion. (SQLSTATE 42804)

-207 (ECPG_NUMERIC_FORMAT)

The host variable is of type numeric and the datum in the database is of another type and contains a value that cannot be interpreted as a numeric value. (SQLSTATE 42804)

-208 (ECPG_INTERVAL_FORMAT)

The host variable is of type interval and the datum in the database is of another type and contains a value that cannot be interpreted as an interval value. (SQLSTATE 42804)

-209 (ECPG_DATE_FORMAT)

The host variable is of type date and the datum in the database is of another type and contains a value that cannot be interpreted as a date value. (SQLSTATE 42804)

-210 (ECPG_TIMESTAMP_FORMAT)

The host variable is of type timestamp and the datum in the database is of another type and contains a value that cannot be interpreted as a timestamp value. (SQLSTATE 42804)

-211 (ECPG_CONVERT_BOOL)

This means the host variable is of type bool and the datum in the database is neither 't' nor 'f'. (SQLSTATE 42804)

-212 (ECPG_EMPTY)

The statement sent to the PostgreSQL server was empty. (This cannot normally happen in an embedded SQL program, so it might point to an internal error.) (SQLSTATE YE002)

-213 (ECPG_MISSING_INDICATOR)

A null value was returned and no null indicator variable was supplied. (SQLSTATE 22002)

-214 (ECPG_NO_ARRAY)

An ordinary variable was used in a place that requires an array. (SQLSTATE 42804)

-215 (ECPG_DATA_NOT_ARRAY)

The database returned an ordinary variable in a place that requires array value. (SQLSTATE 42804)

-216 (ECPG_ARRAY_INSERT)

The value could not be inserted into the array. (SQLSTATE 42804)

-220 (ECPG_NO_CONN)

The program tried to access a connection that does not exist. (SQLSTATE 08003)

-221 (ECPG_NOT_CONN)

The program tried to access a connection that does exist but is not open. (This is an internal error.) (SQLSTATE YE002)

-230 (ECPG_INVALID_STMT)

The statement you are trying to use has not been prepared. (SQLSTATE 26000)

-239 (ECPG_INFORMIX_DUPLICATE_KEY)

Duplicate key error, violation of unique constraint (Informix compatibility mode). (SQLSTATE 23505)

-240 (ECPG_UNKNOWN_DESCRIPTOR)

The descriptor specified was not found. The statement you are trying to use has not been prepared. (SQLSTATE 33000)

-241 (ECPG_INVALID_DESCRIPTOR_INDEX)

The descriptor index specified was out of range. (SQLSTATE 07009)

-242 (ECPG_UNKNOWN_DESCRIPTOR_ITEM)

An invalid descriptor item was requested. (This is an internal error.) (SQLSTATE YE002)

-243 (ECPG_VAR_NOT_NUMERIC)

During the execution of a dynamic statement, the database returned a numeric value and the host variable was not numeric. (SQLSTATE 07006)

-244 (ECPG_VAR_NOT_CHAR)

During the execution of a dynamic statement, the database returned a non-numeric value and the host variable was numeric. (SQLSTATE 07006)

-284 (ECPG_INFORMIX_SUBSELECT_NOT_ONE)

A result of the subquery is not single row (Informix compatibility mode). (SQLSTATE 21000)

-400 (ECPG_PGSQL)

Some error caused by the PostgreSQL server. The message contains the error message from the PostgreSQL server.

-401 (ECPG_TRANS)

The PostgreSQL server signaled that we cannot start, commit, or rollback the transaction. (SQLSTATE 08007)

-402 (ECPG_CONNECT)

The connection attempt to the database did not succeed. (SQLSTATE 08001)

-403 (ECPG_DUPLICATE_KEY)

Duplicate key error, violation of unique constraint. (SQLSTATE 23505)

-404 (ECPG_SUBSELECT_NOT_ONE)

A result for the subquery is not single row. (SQLSTATE 21000)

-602 (ECPG_WARNING_UNKNOWN_PORTAL)

An invalid cursor name was specified. (SQLSTATE 34000)

-603 (ECPG_WARNING_IN_TRANSACTION)

Transaction is in progress. (SQLSTATE 25001)

-604 (ECPG_WARNING_NO_TRANSACTION)

There is no active (in-progress) transaction. (SQLSTATE 25P01)

-605 (ECPG_WARNING_PORTAL_EXISTS)

An existing cursor name was specified. (SQLSTATE 42P03)

Prev | Up | Next
36.7. Using Descriptor Areas | Home | 36.9. Preprocessor Directives


36.9. Preprocessor Directives
Prev Up Chapter 36. ECPG — Embedded SQL in C Home Next

36.9. Preprocessor Directives

36.9.1. Including Files
36.9.2. The define and undef Directives
36.9.3. ifdef, ifndef, elif, else, and endif Directives

Several preprocessor directives are available that modify how the ecpg preprocessor parses and processes a file.

36.9.1. Including Files

To include an external file into your embedded SQL program, use:

EXEC SQL INCLUDE filename;
EXEC SQL INCLUDE <filename>;
EXEC SQL INCLUDE "filename";

The embedded SQL preprocessor will look for a file named filename.h, preprocess it, and include it in the resulting C output. Thus, embedded SQL statements in the included file are handled correctly.

The ecpg preprocessor will search a file at several directories in following order:

  • current directory
  • /usr/local/include
  • PostgreSQL include directory, defined at build time (e.g., /usr/local/pgsql/include)
  • /usr/include

But when EXEC SQL INCLUDE "filename" is used, only the current directory is searched.

In each directory, the preprocessor will first look for the file name as given, and if not found will append .h to the file name and try again (unless the specified file name already has that suffix).

Note that EXEC SQL INCLUDE is not the same as:

#include <filename.h>

because this file would not be subject to SQL command preprocessing. Naturally, you can continue to use the C #include directive to include other header files.

Note

The include file name is case-sensitive, even though the rest of the EXEC SQL INCLUDE command follows the normal SQL case-sensitivity rules.

36.9.2. The define and undef Directives

Similar to the directive #define that is known from C, embedded SQL has a similar concept:

EXEC SQL DEFINE name;
EXEC SQL DEFINE name value;

So you can define a name:

EXEC SQL DEFINE HAVE_FEATURE;

And you can also define constants:

EXEC SQL DEFINE MYNUMBER 12;
EXEC SQL DEFINE MYSTRING 'abc';

Use undef to remove a previous definition:

EXEC SQL UNDEF MYNUMBER;

Of course you can continue to use the C versions #define and #undef in your embedded SQL program. The difference is where your defined values get evaluated. If you use EXEC SQL DEFINE then the ecpg preprocessor evaluates the defines and substitutes the values. For example if you write:

EXEC SQL DEFINE MYNUMBER 12;
...
EXEC SQL UPDATE Tbl SET col = MYNUMBER;

then ecpg will already do the substitution and your C compiler will never see any name or identifier MYNUMBER. Note that you cannot use #define for a constant that you are going to use in an embedded SQL query because in this case the embedded SQL precompiler is not able to see this declaration.

36.9.3. ifdef, ifndef, elif, else, and endif Directives

You can use the following directives to compile code sections conditionally:

EXEC SQL ifdef name;

Checks a name and processes subsequent lines if name has been defined via EXEC SQL define name.

EXEC SQL ifndef name;

Checks a name and processes subsequent lines if name has not been defined via EXEC SQL define name.

EXEC SQL elif name;

Begins an optional alternative section after an EXEC SQL ifdef name or EXEC SQL ifndef name directive. Any number of elif sections can appear. Lines following an elif will be processed if name has been defined and no previous section of the same ifdef/ifndef...endif construct has been processed.

EXEC SQL else;

Begins an optional, final alternative section after an EXEC SQL ifdef name or EXEC SQL ifndef name directive. Subsequent lines will be processed if no previous section of the same ifdef/ifndef...endif construct has been processed.

EXEC SQL endif;

Ends an ifdef/ifndef...endif construct. Subsequent lines are processed normally.

ifdef/ifndef...endif constructs can be nested, up to 127 levels deep.

This example will compile exactly one of the three SET TIMEZONE commands:

EXEC SQL ifdef TZVAR;
EXEC SQL SET TIMEZONE TO TZVAR;
EXEC SQL elif TZNAME;
EXEC SQL SET TIMEZONE TO TZNAME;
EXEC SQL else;
EXEC SQL SET TIMEZONE TO 'GMT';
EXEC SQL endif;

Prev | Up | Next
36.8. Error Handling | Home | 36.10. Processing Embedded SQL Programs


36.10. Processing Embedded SQL Programs
Prev Up Chapter 36. ECPG — Embedded SQL in C Home Next

36.10. Processing Embedded SQL Programs

Now that you have an idea how to form embedded SQL C programs, you probably want to know how to compile them. Before compiling you run the file through the embedded SQL C preprocessor, which converts the SQL statements you used to special function calls. After compiling, you must link with a special library that contains the needed functions. These functions fetch information from the arguments, perform the SQL command using the libpq interface, and put the result in the arguments specified for output.

The preprocessor program is called ecpg and is included in a normal PostgreSQL installation. Embedded SQL programs are typically named with an extension .pgc. If you have a program file called prog1.pgc, you can preprocess it by simply calling:

ecpg prog1.pgc

This will create a file called prog1.c. If your input files do not follow the suggested naming pattern, you can specify the output file explicitly using the -o option.

The preprocessed file can be compiled normally, for example:

cc -c prog1.c

The generated C source files include header files from the PostgreSQL installation, so if you installed PostgreSQL in a location that is not searched by default, you have to add an option such as -I/usr/local/pgsql/include to the compilation command line.

To link an embedded SQL program, you need to include the libecpg library, like so:

cc -o myprog prog1.o prog2.o ... -lecpg

Again, you might have to add an option like -L/usr/local/pgsql/lib to that command line.

You can use pg_config or pkg-config with package name libecpg to get the paths for your installation.

If you manage the build process of a larger project using make, it might be convenient to include the following implicit rule to your makefiles:

ECPG = ecpg

%.c: %.pgc
        $(ECPG) $<

The complete syntax of the ecpg command is detailed in ecpg.

The ecpg library is thread-safe by default. However, you might need to use some threading command-line options to compile your client code.

Prev | Up | Next
36.9. Preprocessor Directives | Home | 36.11. Library Functions


36.11. Library Functions
Prev Up Chapter 36. ECPG — Embedded SQL in C Home Next

36.11. Library Functions

The libecpg library primarily contains “hidden” functions that are used to implement the functionality expressed by the embedded SQL commands. But there are some functions that can usefully be called directly. Note that this makes your code unportable.

  • ECPGdebug(int on, FILE *stream) turns on debug logging if called with the first argument non-zero. Debug logging is done on stream. The log contains all SQL statements with all the input variables inserted, and the results from the PostgreSQL server. This can be very useful when searching for errors in your SQL statements.

    Note

    On Windows, if the ecpg libraries and an application are compiled with different flags, this function call will crash the application because the internal representation of the FILE pointers differ. Specifically, multithreaded/single-threaded, release/debug, and static/dynamic flags should be the same for the library and all applications using that library.

  • ECPGget_PGconn(const char *connection_name) returns the library database connection handle identified by the given name. If connection_name is set to NULL, the current connection handle is returned. If no connection handle can be identified, the function returns NULL. The returned connection handle can be used to call any other functions from libpq, if necessary.

    Note

    It is a bad idea to manipulate database connection handles made from ecpg directly with libpq routines.

  • ECPGtransactionStatus(const char *connection_name) returns the current transaction status of the given connection identified by connection_name. See Section 34.2 and libpq's PQtransactionStatus for details about the returned status codes.

  • ECPGstatus(int lineno, const char* connection_name) returns true if you are connected to a database and false if not. connection_name can be NULL if a single connection is being used.

Prev | Up | Next
36.10. Processing Embedded SQL Programs | Home | 36.12. Large Objects


36.12. Large Objects
Prev Up Chapter 36. ECPG — Embedded SQL in C Home Next

36.12. Large Objects

Large objects are not directly supported by ECPG, but ECPG application can manipulate large objects through the libpq large object functions, obtaining the necessary PGconn object by calling the ECPGget_PGconn() function. (However, use of the ECPGget_PGconn() function and touching PGconn objects directly should be done very carefully and ideally not mixed with other ECPG database access calls.)

For more details about the ECPGget_PGconn(), see Section 36.11. For information about the large object function interface, see Chapter 35.

Large object functions have to be called in a transaction block, so when autocommit is off, BEGIN commands have to be issued explicitly.

Example 36.2 shows an example program that illustrates how to create, write, and read a large object in an ECPG application.

Example 36.2. ECPG Program Accessing Large Objects

#include <stdio.h>
#include <stdlib.h>
#include <libpq-fe.h>
#include <libpq/libpq-fs.h>

EXEC SQL WHENEVER SQLERROR STOP;

int
main(void)
{
    PGconn     *conn;
    Oid         loid;
    int         fd;
    char        buf[256];
    int         buflen = 256;
    char        buf2[256];
    int         rc;

    memset(buf, 1, buflen);

    EXEC SQL CONNECT TO testdb AS con1;
    EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); EXEC SQL COMMIT;

    conn = ECPGget_PGconn("con1");
    printf("conn = %p\n", conn);

    /* create */
    loid = lo_create(conn, 0);
    if (loid &lt; 0)
        printf("lo_create() failed: %s", PQerrorMessage(conn));

    printf("loid = %d\n", loid);

    /* write test */
    fd = lo_open(conn, loid, INV_READ|INV_WRITE);
    if (fd &lt; 0)
        printf("lo_open() failed: %s", PQerrorMessage(conn));

    printf("fd = %d\n", fd);

    rc = lo_write(conn, fd, buf, buflen);
    if (rc &lt; 0)
        printf("lo_write() failed\n");

    rc = lo_close(conn, fd);
    if (rc &lt; 0)
        printf("lo_close() failed: %s", PQerrorMessage(conn));

    /* read test */
    fd = lo_open(conn, loid, INV_READ);
    if (fd &lt; 0)
        printf("lo_open() failed: %s", PQerrorMessage(conn));

    printf("fd = %d\n", fd);

    rc = lo_read(conn, fd, buf2, buflen);
    if (rc &lt; 0)
        printf("lo_read() failed\n");

    rc = lo_close(conn, fd);
    if (rc &lt; 0)
        printf("lo_close() failed: %s", PQerrorMessage(conn));

    /* check */
    rc = memcmp(buf, buf2, buflen);
    printf("memcmp() = %d\n", rc);

    /* cleanup */
    rc = lo_unlink(conn, loid);
    if (rc &lt; 0)
        printf("lo_unlink() failed: %s", PQerrorMessage(conn));

    EXEC SQL COMMIT;
    EXEC SQL DISCONNECT ALL;
    return 0;
}

Prev | Up | Next
36.11. Library Functions | Home | 36.13. C++ Applications


36.13. C++ Applications
Prev Up Chapter 36. ECPG — Embedded SQL in C Home Next

36.13. C++ Applications

36.13.1. Scope for Host Variables
36.13.2. C++ Application Development with External C Module

ECPG has some limited support for C++ applications. This section describes some caveats.

The ecpg preprocessor takes an input file written in C (or something like C) and embedded SQL commands, converts the embedded SQL commands into C language chunks, and finally generates a .c file. The header file declarations of the library functions used by the C language chunks that ecpg generates are wrapped in extern "C" { ... } blocks when used under C++, so they should work seamlessly in C++.

In general, however, the ecpg preprocessor only understands C; it does not handle the special syntax and reserved words of the C++ language. So, some embedded SQL code written in C++ application code that uses complicated features specific to C++ might fail to be preprocessed correctly or might not work as expected.

A safe way to use the embedded SQL code in a C++ application is hiding the ECPG calls in a C module, which the C++ application code calls into to access the database, and linking that together with the rest of the C++ code. See Section 36.13.2 about that.

36.13.1. Scope for Host Variables

The ecpg preprocessor understands the scope of variables in C. In the C language, this is rather simple because the scopes of variables is based on their code blocks. In C++, however, the class member variables are referenced in a different code block from the declared position, so the ecpg preprocessor will not understand the scope of the class member variables.

For example, in the following case, the ecpg preprocessor cannot find any declaration for the variable dbname in the test method, so an error will occur.

class TestCpp
{
    EXEC SQL BEGIN DECLARE SECTION;
    char dbname[1024];
    EXEC SQL END DECLARE SECTION;

  public:
    TestCpp();
    void test();
    ~TestCpp();
};

TestCpp::TestCpp()
{
    EXEC SQL CONNECT TO testdb1;
    EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); EXEC SQL COMMIT;
}

void Test::test()
{
    EXEC SQL SELECT current_database() INTO :dbname;
    printf("current_database = %s\n", dbname);
}

TestCpp::~TestCpp()
{
    EXEC SQL DISCONNECT ALL;
}

This code will result in an error like this:

ecpg test_cpp.pgc
test_cpp.pgc:28: ERROR: variable "dbname" is not declared

To avoid this scope issue, the test method could be modified to use a local variable as intermediate storage. But this approach is only a poor workaround, because it uglifies the code and reduces performance.

void TestCpp::test()
{
    EXEC SQL BEGIN DECLARE SECTION;
    char tmp[1024];
    EXEC SQL END DECLARE SECTION;

    EXEC SQL SELECT current_database() INTO :tmp;
    strlcpy(dbname, tmp, sizeof(tmp));

    printf("current_database = %s\n", dbname);
}

36.13.2. C++ Application Development with External C Module

If you understand these technical limitations of the ecpg preprocessor in C++, you might come to the conclusion that linking C objects and C++ objects at the link stage to enable C++ applications to use ECPG features could be better than writing some embedded SQL commands in C++ code directly. This section describes a way to separate some embedded SQL commands from C++ application code with a simple example. In this example, the application is implemented in C++, while C and ECPG is used to connect to the PostgreSQL server.

Three kinds of files have to be created: a C file (*.pgc), a header file, and a C++ file:

test_mod.pgc

A sub-routine module to execute SQL commands embedded in C. It is going to be converted into test_mod.c by the preprocessor.

#include "test_mod.h"
#include <stdio.h>

void
db_connect()
{
    EXEC SQL CONNECT TO testdb1;
    EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); EXEC SQL COMMIT;
}

void
db_test()
{
    EXEC SQL BEGIN DECLARE SECTION;
    char dbname[1024];
    EXEC SQL END DECLARE SECTION;

    EXEC SQL SELECT current_database() INTO :dbname;
    printf("current_database = %s\n", dbname);
}

void
db_disconnect()
{
    EXEC SQL DISCONNECT ALL;
}

test_mod.h

A header file with declarations of the functions in the C module (test_mod.pgc). It is included by test_cpp.cpp. This file has to have an extern "C" block around the declarations, because it will be linked from the C++ module.

#ifdef __cplusplus
extern "C" {
#endif

void db_connect();
void db_test();
void db_disconnect();

#ifdef __cplusplus
}
#endif

test_cpp.cpp

The main code for the application, including the main routine, and in this example a C++ class.

#include "test_mod.h"

class TestCpp
{
  public:
    TestCpp();
    void test();
    ~TestCpp();
};

TestCpp::TestCpp()
{
    db_connect();
}

void
TestCpp::test()
{
    db_test();
}

TestCpp::~TestCpp()
{
    db_disconnect();
}

int
main(void)
{
    TestCpp *t = new TestCpp();

    t->test();
    return 0;
}

To build the application, proceed as follows. Convert test_mod.pgc into test_mod.c by running ecpg, and generate test_mod.o by compiling test_mod.c with the C compiler:

ecpg -o test_mod.c test_mod.pgc
cc -c test_mod.c -o test_mod.o

Next, generate test_cpp.o by compiling test_cpp.cpp with the C++ compiler:

c++ -c test_cpp.cpp -o test_cpp.o

Finally, link these object files, test_cpp.o and test_mod.o, into one executable, using the C++ compiler driver:

c++ test_cpp.o test_mod.o -lecpg -o test_cpp

Prev | Up | Next
36.12. Large Objects | Home | 36.14. Embedded SQL Commands


36.14. Embedded SQL Commands
Prev Up Chapter 36. ECPG — Embedded SQL in C Home Next

36.14. Embedded SQL Commands

ALLOCATE DESCRIPTOR — allocate an SQL descriptor area
CONNECT — establish a database connection
DEALLOCATE DESCRIPTOR — deallocate an SQL descriptor area
DECLARE — define a cursor
DECLARE STATEMENT — declare SQL statement identifier
DESCRIBE — obtain information about a prepared statement or result set
DISCONNECT — terminate a database connection
EXECUTE IMMEDIATE — dynamically prepare and execute a statement
GET DESCRIPTOR — get information from an SQL descriptor area
OPEN — open a dynamic cursor
PREPARE — prepare a statement for execution
SET AUTOCOMMIT — set the autocommit behavior of the current session
SET CONNECTION — select a database connection
SET DESCRIPTOR — set information in an SQL descriptor area
TYPE — define a new data type
VAR — define a variable
WHENEVER — specify the action to be taken when an SQL statement causes a specific class condition to be raised

This section describes all SQL commands that are specific to embedded SQL. Also refer to the SQL commands listed in SQL Commands, which can also be used in embedded SQL, unless stated otherwise.

Prev | Up | Next
36.13. C++ Applications | Home | ALLOCATE DESCRIPTOR


ALLOCATE DESCRIPTOR
Prev Up 36.14. Embedded SQL Commands Home Next

ALLOCATE DESCRIPTOR

ALLOCATE DESCRIPTOR — allocate an SQL descriptor area

Synopsis

ALLOCATE DESCRIPTOR name

Description

ALLOCATE DESCRIPTOR allocates a new named SQL descriptor area, which can be used to exchange data between the PostgreSQL server and the host program.

Descriptor areas should be freed after use using the DEALLOCATE DESCRIPTOR command.

Parameters

name

A name of SQL descriptor, case sensitive. This can be an SQL identifier or a host variable.

Examples

EXEC SQL ALLOCATE DESCRIPTOR mydesc;

Compatibility

ALLOCATE DESCRIPTOR is specified in the SQL standard.

See Also

DEALLOCATE DESCRIPTOR, GET DESCRIPTOR, SET DESCRIPTOR
Prev | Up | Next
36.14. Embedded SQL Commands | Home | CONNECT


CONNECT
Prev Up 36.14. Embedded SQL Commands Home Next

CONNECT

CONNECT — establish a database connection

Synopsis

CONNECT TO connection_target [ AS connection_name ] [ USER connection_user ]
CONNECT TO DEFAULT
CONNECT connection_user
DATABASE connection_target

Description

The CONNECT command establishes a connection between the client and the PostgreSQL server.

Parameters

connection_target

connection_target specifies the target server of the connection on one of several forms.

[ database_name ] [ @host ] [ :port ]

Connect over TCP/IP

unix:postgresql://host [ :port ] / [ database_name ] [ ?connection_option ]

Connect over Unix-domain sockets

tcp:postgresql://host [ :port ] / [ database_name ] [ ?connection_option ]

Connect over TCP/IP

SQL string constant

containing a value in one of the above forms

host variable

host variable of type char[] or VARCHAR[] containing a value in one of the above forms

connection_name

An optional identifier for the connection, so that it can be referred to in other commands. This can be an SQL identifier or a host variable.

connection_user

The user name for the database connection.

This parameter can also specify user name and password, using one the forms user_name/password, user_name IDENTIFIED BY password, or user_name USING password.

User name and password can be SQL identifiers, string constants, or host variables.

DEFAULT

Use all default connection parameters, as defined by libpq.

Examples

Here a several variants for specifying connection parameters:

EXEC SQL CONNECT TO "connectdb" AS main;
EXEC SQL CONNECT TO "connectdb" AS second;
EXEC SQL CONNECT TO "unix:postgresql://200.46.204.71/connectdb" AS main USER connectuser;
EXEC SQL CONNECT TO "unix:postgresql://localhost/connectdb" AS main USER connectuser;
EXEC SQL CONNECT TO 'connectdb' AS main;
EXEC SQL CONNECT TO 'unix:postgresql://localhost/connectdb' AS main USER :user;
EXEC SQL CONNECT TO :db AS :id;
EXEC SQL CONNECT TO :db USER connectuser USING :pw;
EXEC SQL CONNECT TO @localhost AS main USER connectdb;
EXEC SQL CONNECT TO REGRESSDB1 as main;
EXEC SQL CONNECT TO AS main USER connectdb;
EXEC SQL CONNECT TO connectdb AS :id;
EXEC SQL CONNECT TO connectdb AS main USER connectuser/connectdb;
EXEC SQL CONNECT TO connectdb AS main;
EXEC SQL CONNECT TO connectdb@localhost AS main;
EXEC SQL CONNECT TO tcp:postgresql://localhost/ USER connectdb;
EXEC SQL CONNECT TO tcp:postgresql://localhost/connectdb USER connectuser IDENTIFIED BY connectpw;
EXEC SQL CONNECT TO tcp:postgresql://localhost:20/connectdb USER connectuser IDENTIFIED BY connectpw;
EXEC SQL CONNECT TO unix:postgresql://localhost/ AS main USER connectdb;
EXEC SQL CONNECT TO unix:postgresql://localhost/connectdb AS main USER connectuser;
EXEC SQL CONNECT TO unix:postgresql://localhost/connectdb USER connectuser IDENTIFIED BY "connectpw";
EXEC SQL CONNECT TO unix:postgresql://localhost/connectdb USER connectuser USING "connectpw";
EXEC SQL CONNECT TO unix:postgresql://localhost/connectdb?connect_timeout=14 USER connectuser;

Here is an example program that illustrates the use of host variables to specify connection parameters:

int
main(void)
{
EXEC SQL BEGIN DECLARE SECTION;
    char *dbname     = "testdb";    /* database name */
    char *user       = "testuser";  /* connection user name */
    char *connection = "tcp:postgresql://localhost:5432/testdb";
                                    /* connection string */
    char ver[256];                  /* buffer to store the version string */
EXEC SQL END DECLARE SECTION;

    ECPGdebug(1, stderr);

    EXEC SQL CONNECT TO :dbname USER :user;
    EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); EXEC SQL COMMIT;
    EXEC SQL SELECT version() INTO :ver;
    EXEC SQL DISCONNECT;

    printf("version: %s\n", ver);

    EXEC SQL CONNECT TO :connection USER :user;
    EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); EXEC SQL COMMIT;
    EXEC SQL SELECT version() INTO :ver;
    EXEC SQL DISCONNECT;

    printf("version: %s\n", ver);

    return 0;
}

Compatibility

CONNECT is specified in the SQL standard, but the format of the connection parameters is implementation-specific.

See Also

DISCONNECT, SET CONNECTION
Prev | Up | Next
ALLOCATE DESCRIPTOR | Home | DEALLOCATE DESCRIPTOR


DEALLOCATE DESCRIPTOR
Prev Up 36.14. Embedded SQL Commands Home Next

DEALLOCATE DESCRIPTOR

DEALLOCATE DESCRIPTOR — deallocate an SQL descriptor area

Synopsis

DEALLOCATE DESCRIPTOR name

Description

DEALLOCATE DESCRIPTOR deallocates a named SQL descriptor area.

Parameters

name

The name of the descriptor which is going to be deallocated. It is case sensitive. This can be an SQL identifier or a host variable.

Examples

EXEC SQL DEALLOCATE DESCRIPTOR mydesc;

Compatibility

DEALLOCATE DESCRIPTOR is specified in the SQL standard.

See Also

ALLOCATE DESCRIPTOR, GET DESCRIPTOR, SET DESCRIPTOR
Prev | Up | Next
CONNECT | Home | DECLARE


DECLARE
Prev Up 36.14. Embedded SQL Commands Home Next

DECLARE

DECLARE — define a cursor

Synopsis

DECLARE cursor_name [ BINARY ] [ ASENSITIVE | INSENSITIVE ] [ [ NO ] SCROLL ] CURSOR [ { WITH | WITHOUT } HOLD ] FOR prepared_name
DECLARE cursor_name [ BINARY ] [ ASENSITIVE | INSENSITIVE ] [ [ NO ] SCROLL ] CURSOR [ { WITH | WITHOUT } HOLD ] FOR query

Description

DECLARE declares a cursor for iterating over the result set of a prepared statement. This command has slightly different semantics from the direct SQL command DECLARE: Whereas the latter executes a query and prepares the result set for retrieval, this embedded SQL command merely declares a name as a “loop variable” for iterating over the result set of a query; the actual execution happens when the cursor is opened with the OPEN command.

Parameters

cursor_name

A cursor name, case sensitive. This can be an SQL identifier or a host variable.

prepared_name

The name of a prepared query, either as an SQL identifier or a host variable.

query

A SELECT or VALUES command which will provide the rows to be returned by the cursor.

For the meaning of the cursor options, see DECLARE.

Examples

Examples declaring a cursor for a query:

EXEC SQL DECLARE C CURSOR FOR SELECT * FROM My_Table;
EXEC SQL DECLARE C CURSOR FOR SELECT Item1 FROM T;
EXEC SQL DECLARE cur1 CURSOR FOR SELECT version();

An example declaring a cursor for a prepared statement:

EXEC SQL PREPARE stmt1 AS SELECT version();
EXEC SQL DECLARE cur1 CURSOR FOR stmt1;

Compatibility

DECLARE is specified in the SQL standard.

See Also

OPEN, CLOSE, DECLARE
Prev | Up | Next
DEALLOCATE DESCRIPTOR | Home | DECLARE STATEMENT


DECLARE STATEMENT
Prev Up 36.14. Embedded SQL Commands Home Next

DECLARE STATEMENT

DECLARE STATEMENT — declare SQL statement identifier

Synopsis

EXEC SQL [ AT connection_name ] DECLARE statement_name STATEMENT

Description

DECLARE STATEMENT declares an SQL statement identifier. SQL statement identifier can be associated with the connection. When the identifier is used by dynamic SQL statements, the statements are executed using the associated connection. The namespace of the declaration is the precompile unit, and multiple declarations to the same SQL statement identifier are not allowed. Note that if the precompiler runs in Informix compatibility mode and some SQL statement is declared, "database" can not be used as a cursor name.

Parameters

connection_name

A database connection name established by the CONNECT command.

AT clause can be omitted, but such statement has no meaning.

statement_name

The name of an SQL statement identifier, either as an SQL identifier or a host variable.

Notes

This association is valid only if the declaration is physically placed on top of a dynamic statement.

Examples

EXEC SQL CONNECT TO postgres AS con1;
EXEC SQL AT con1 DECLARE sql_stmt STATEMENT;
EXEC SQL DECLARE cursor_name CURSOR FOR sql_stmt;
EXEC SQL PREPARE sql_stmt FROM :dyn_string;
EXEC SQL OPEN cursor_name;
EXEC SQL FETCH cursor_name INTO :column1;
EXEC SQL CLOSE cursor_name;

Compatibility

DECLARE STATEMENT is an extension of the SQL standard, but can be used in famous DBMSs.

See Also

CONNECT, DECLARE, OPEN
Prev | Up | Next
DECLARE | Home | DESCRIBE


DESCRIBE
Prev Up 36.14. Embedded SQL Commands Home Next

DESCRIBE

DESCRIBE — obtain information about a prepared statement or result set

Synopsis

DESCRIBE [ OUTPUT ] prepared_name USING [ SQL ] DESCRIPTOR descriptor_name
DESCRIBE [ OUTPUT ] prepared_name INTO [ SQL ] DESCRIPTOR descriptor_name
DESCRIBE [ OUTPUT ] prepared_name INTO sqlda_name

Description

DESCRIBE retrieves metadata information about the result columns contained in a prepared statement, without actually fetching a row.

Parameters

prepared_name

The name of a prepared statement. This can be an SQL identifier or a host variable.

descriptor_name

A descriptor name. It is case sensitive. It can be an SQL identifier or a host variable.

sqlda_name

The name of an SQLDA variable.

Examples

EXEC SQL ALLOCATE DESCRIPTOR mydesc;
EXEC SQL PREPARE stmt1 FROM :sql_stmt;
EXEC SQL DESCRIBE stmt1 INTO SQL DESCRIPTOR mydesc;
EXEC SQL GET DESCRIPTOR mydesc VALUE 1 :charvar = NAME;
EXEC SQL DEALLOCATE DESCRIPTOR mydesc;

Compatibility

DESCRIBE is specified in the SQL standard.

See Also

ALLOCATE DESCRIPTOR, GET DESCRIPTOR
Prev | Up | Next
DECLARE STATEMENT | Home | DISCONNECT


DISCONNECT
Prev Up 36.14. Embedded SQL Commands Home Next

DISCONNECT

DISCONNECT — terminate a database connection

Synopsis

DISCONNECT connection_name
DISCONNECT [ CURRENT ]
DISCONNECT DEFAULT
DISCONNECT ALL

Description

DISCONNECT closes a connection (or all connections) to the database.

Parameters

connection_name

A database connection name established by the CONNECT command.

CURRENT

Close the “current” connection, which is either the most recently opened connection, or the connection set by the SET CONNECTION command. This is also the default if no argument is given to the DISCONNECT command.

DEFAULT

Close the default connection.

ALL

Close all open connections.

Examples

int
main(void)
{
    EXEC SQL CONNECT TO testdb AS DEFAULT USER testuser;
    EXEC SQL CONNECT TO testdb AS con1 USER testuser;
    EXEC SQL CONNECT TO testdb AS con2 USER testuser;
    EXEC SQL CONNECT TO testdb AS con3 USER testuser;

    EXEC SQL DISCONNECT CURRENT;  /* close con3          */
    EXEC SQL DISCONNECT DEFAULT;  /* close DEFAULT       */
    EXEC SQL DISCONNECT ALL;      /* close con2 and con1 */

    return 0;
}

Compatibility

DISCONNECT is specified in the SQL standard.

See Also

CONNECT, SET CONNECTION
Prev | Up | Next
DESCRIBE | Home | EXECUTE IMMEDIATE


EXECUTE IMMEDIATE
Prev Up 36.14. Embedded SQL Commands Home Next

EXECUTE IMMEDIATE

EXECUTE IMMEDIATE — dynamically prepare and execute a statement

Synopsis

EXECUTE IMMEDIATE string

Description

EXECUTE IMMEDIATE immediately prepares and executes a dynamically specified SQL statement, without retrieving result rows.

Parameters

string

A literal string or a host variable containing the SQL statement to be executed.

Notes

In typical usage, the string is a host variable reference to a string containing a dynamically-constructed SQL statement. The case of a literal string is not very useful; you might as well just write the SQL statement directly, without the extra typing of EXECUTE IMMEDIATE.

If you do use a literal string, keep in mind that any double quotes you might wish to include in the SQL statement must be written as octal escapes (\042) not the usual C idiom \". This is because the string is inside an EXEC SQL section, so the ECPG lexer parses it according to SQL rules not C rules. Any embedded backslashes will later be handled according to C rules; but \" causes an immediate syntax error because it is seen as ending the literal.

Examples

Here is an example that executes an INSERT statement using EXECUTE IMMEDIATE and a host variable named command:

sprintf(command, "INSERT INTO test (name, amount, letter) VALUES ('db: ''r1''', 1, 'f')");
EXEC SQL EXECUTE IMMEDIATE :command;

Compatibility

EXECUTE IMMEDIATE is specified in the SQL standard.

Prev | Up | Next
DISCONNECT | Home | GET DESCRIPTOR


GET DESCRIPTOR
Prev Up 36.14. Embedded SQL Commands Home Next

GET DESCRIPTOR

GET DESCRIPTOR — get information from an SQL descriptor area

Synopsis

GET DESCRIPTOR descriptor_name :cvariable = descriptor_header_item [, ... ]
GET DESCRIPTOR descriptor_name VALUE column_number :cvariable = descriptor_item [, ... ]

Description

GET DESCRIPTOR retrieves information about a query result set from an SQL descriptor area and stores it into host variables. A descriptor area is typically populated using FETCH or SELECT before using this command to transfer the information into host language variables.

This command has two forms: The first form retrieves descriptor “header” items, which apply to the result set in its entirety. One example is the row count. The second form, which requires the column number as additional parameter, retrieves information about a particular column. Examples are the column name and the actual column value.

Parameters

descriptor_name

A descriptor name.

descriptor_header_item

A token identifying which header information item to retrieve. Only COUNT, to get the number of columns in the result set, is currently supported.

column_number

The number of the column about which information is to be retrieved. The count starts at 1.

descriptor_item

A token identifying which item of information about a column to retrieve. See Section 36.7.1 for a list of supported items.

cvariable

A host variable that will receive the data retrieved from the descriptor area.

Examples

An example to retrieve the number of columns in a result set:

EXEC SQL GET DESCRIPTOR d :d_count = COUNT;

An example to retrieve a data length in the first column:

EXEC SQL GET DESCRIPTOR d VALUE 1 :d_returned_octet_length = RETURNED_OCTET_LENGTH;

An example to retrieve the data body of the second column as a string:

EXEC SQL GET DESCRIPTOR d VALUE 2 :d_data = DATA;

Here is an example for a whole procedure of executing SELECT current_database(); and showing the number of columns, the column data length, and the column data:

int
main(void)
{
EXEC SQL BEGIN DECLARE SECTION;
    int  d_count;
    char d_data[1024];
    int  d_returned_octet_length;
EXEC SQL END DECLARE SECTION;

    EXEC SQL CONNECT TO testdb AS con1 USER testuser;
    EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); EXEC SQL COMMIT;
    EXEC SQL ALLOCATE DESCRIPTOR d;

    /* Declare, open a cursor, and assign a descriptor to the cursor  */
    EXEC SQL DECLARE cur CURSOR FOR SELECT current_database();
    EXEC SQL OPEN cur;
    EXEC SQL FETCH NEXT FROM cur INTO SQL DESCRIPTOR d;

    /* Get a number of total columns */
    EXEC SQL GET DESCRIPTOR d :d_count = COUNT;
    printf("d_count                 = %d\n", d_count);

    /* Get length of a returned column */
    EXEC SQL GET DESCRIPTOR d VALUE 1 :d_returned_octet_length = RETURNED_OCTET_LENGTH;
    printf("d_returned_octet_length = %d\n", d_returned_octet_length);

    /* Fetch the returned column as a string */
    EXEC SQL GET DESCRIPTOR d VALUE 1 :d_data = DATA;
    printf("d_data                  = %s\n", d_data);

    /* Closing */
    EXEC SQL CLOSE cur;
    EXEC SQL COMMIT;

    EXEC SQL DEALLOCATE DESCRIPTOR d;
    EXEC SQL DISCONNECT ALL;

    return 0;
}

When the example is executed, the result will look like this:

d_count                 = 1
d_returned_octet_length = 6
d_data                  = testdb

Compatibility

GET DESCRIPTOR is specified in the SQL standard.

See Also

ALLOCATE DESCRIPTOR, SET DESCRIPTOR
Prev | Up | Next
EXECUTE IMMEDIATE | Home | OPEN


OPEN
Prev Up 36.14. Embedded SQL Commands Home Next

OPEN

OPEN — open a dynamic cursor

Synopsis

OPEN cursor_name
OPEN cursor_name USING value [, ... ]
OPEN cursor_name USING SQL DESCRIPTOR descriptor_name

Description

OPEN opens a cursor and optionally binds actual values to the placeholders in the cursor's declaration. The cursor must previously have been declared with the DECLARE command. The execution of OPEN causes the query to start executing on the server.

Parameters

cursor_name

The name of the cursor to be opened. This can be an SQL identifier or a host variable.

value

A value to be bound to a placeholder in the cursor. This can be an SQL constant, a host variable, or a host variable with indicator.

descriptor_name

The name of a descriptor containing values to be bound to the placeholders in the cursor. This can be an SQL identifier or a host variable.

Examples

EXEC SQL OPEN a;
EXEC SQL OPEN d USING 1, 'test';
EXEC SQL OPEN c1 USING SQL DESCRIPTOR mydesc;
EXEC SQL OPEN :curname1;

Compatibility

OPEN is specified in the SQL standard.

See Also

DECLARE, CLOSE
Prev | Up | Next
GET DESCRIPTOR | Home | PREPARE


PREPARE
Prev Up 36.14. Embedded SQL Commands Home Next

PREPARE

PREPARE — prepare a statement for execution

Synopsis

PREPARE prepared_name FROM string

Description

PREPARE prepares a statement dynamically specified as a string for execution. This is different from the direct SQL statement PREPARE, which can also be used in embedded programs. The EXECUTE command is used to execute either kind of prepared statement.

Parameters

prepared_name

An identifier for the prepared query.

string

A literal string or a host variable containing a preparable SQL statement, one of SELECT, INSERT, UPDATE, or DELETE. Use question marks (?) for parameter values to be supplied at execution.

Notes

In typical usage, the string is a host variable reference to a string containing a dynamically-constructed SQL statement. The case of a literal string is not very useful; you might as well just write a direct SQL PREPARE statement.

If you do use a literal string, keep in mind that any double quotes you might wish to include in the SQL statement must be written as octal escapes (\042) not the usual C idiom \". This is because the string is inside an EXEC SQL section, so the ECPG lexer parses it according to SQL rules not C rules. Any embedded backslashes will later be handled according to C rules; but \" causes an immediate syntax error because it is seen as ending the literal.

Examples

char *stmt = "SELECT * FROM test1 WHERE a = ? AND b = ?";

EXEC SQL ALLOCATE DESCRIPTOR outdesc;
EXEC SQL PREPARE foo FROM :stmt;

EXEC SQL EXECUTE foo USING SQL DESCRIPTOR indesc INTO SQL DESCRIPTOR outdesc;

Compatibility

PREPARE is specified in the SQL standard.

See Also

EXECUTE
Prev | Up | Next
OPEN | Home | SET AUTOCOMMIT


SET AUTOCOMMIT
Prev Up 36.14. Embedded SQL Commands Home Next

SET AUTOCOMMIT

SET AUTOCOMMIT — set the autocommit behavior of the current session

Synopsis

SET AUTOCOMMIT { = | TO } { ON | OFF }

Description

SET AUTOCOMMIT sets the autocommit behavior of the current database session. By default, embedded SQL programs are not in autocommit mode, so COMMIT needs to be issued explicitly when desired. This command can change the session to autocommit mode, where each individual statement is committed implicitly.

Compatibility

SET AUTOCOMMIT is an extension of PostgreSQL ECPG.

Prev | Up | Next
PREPARE | Home | SET CONNECTION


SET CONNECTION
Prev Up 36.14. Embedded SQL Commands Home Next

SET CONNECTION

SET CONNECTION — select a database connection

Synopsis

SET CONNECTION [ TO | = ] connection_name

Description

SET CONNECTION sets the “current” database connection, which is the one that all commands use unless overridden.

Parameters

connection_name

A database connection name established by the CONNECT command.

DEFAULT

Set the connection to the default connection.

Examples

EXEC SQL SET CONNECTION TO con2;
EXEC SQL SET CONNECTION = con1;

Compatibility

SET CONNECTION is specified in the SQL standard.

See Also

CONNECT, DISCONNECT
Prev | Up | Next
SET AUTOCOMMIT | Home | SET DESCRIPTOR


SET DESCRIPTOR
Prev Up 36.14. Embedded SQL Commands Home Next

SET DESCRIPTOR

SET DESCRIPTOR — set information in an SQL descriptor area

Synopsis

SET DESCRIPTOR descriptor_name descriptor_header_item = value [, ... ]
SET DESCRIPTOR descriptor_name VALUE number descriptor_item = value [, ...]

Description

SET DESCRIPTOR populates an SQL descriptor area with values. The descriptor area is then typically used to bind parameters in a prepared query execution.

This command has two forms: The first form applies to the descriptor “header”, which is independent of a particular datum. The second form assigns values to particular datums, identified by number.

Parameters

descriptor_name

A descriptor name.

descriptor_header_item

A token identifying which header information item to set. Only COUNT, to set the number of descriptor items, is currently supported.

number

The number of the descriptor item to set. The count starts at 1.

descriptor_item

A token identifying which item of information to set in the descriptor. See Section 36.7.1 for a list of supported items.

value

A value to store into the descriptor item. This can be an SQL constant or a host variable.

Examples

EXEC SQL SET DESCRIPTOR indesc COUNT = 1;
EXEC SQL SET DESCRIPTOR indesc VALUE 1 DATA = 2;
EXEC SQL SET DESCRIPTOR indesc VALUE 1 DATA = :val1;
EXEC SQL SET DESCRIPTOR indesc VALUE 2 INDICATOR = :val1, DATA = 'some string';
EXEC SQL SET DESCRIPTOR indesc VALUE 2 INDICATOR = :val2null, DATA = :val2;

Compatibility

SET DESCRIPTOR is specified in the SQL standard.

See Also

ALLOCATE DESCRIPTOR, GET DESCRIPTOR
Prev | Up | Next
SET CONNECTION | Home | TYPE


TYPE
Prev Up 36.14. Embedded SQL Commands Home Next

TYPE

TYPE — define a new data type

Synopsis

TYPE type_name IS ctype

Description

The TYPE command defines a new C type. It is equivalent to putting a typedef into a declare section.

This command is only recognized when ecpg is run with the -c option.

Parameters

type_name

The name for the new type. It must be a valid C type name.

ctype

A C type specification.

Examples

EXEC SQL TYPE customer IS
    struct
    {
        varchar name[50];
        int     phone;
    };

EXEC SQL TYPE cust_ind IS
    struct ind
    {
        short   name_ind;
        short   phone_ind;
    };

EXEC SQL TYPE c IS char reference;
EXEC SQL TYPE ind IS union { int integer; short smallint; };
EXEC SQL TYPE intarray IS int[AMOUNT];
EXEC SQL TYPE str IS varchar[BUFFERSIZ];
EXEC SQL TYPE string IS char[11];

Here is an example program that uses EXEC SQL TYPE:

EXEC SQL WHENEVER SQLERROR SQLPRINT;

EXEC SQL TYPE tt IS
    struct
    {
        varchar v[256];
        int     i;
    };

EXEC SQL TYPE tt_ind IS
    struct ind {
        short   v_ind;
        short   i_ind;
    };

int
main(void)
{
EXEC SQL BEGIN DECLARE SECTION;
    tt t;
    tt_ind t_ind;
EXEC SQL END DECLARE SECTION;

    EXEC SQL CONNECT TO testdb AS con1;
    EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); EXEC SQL COMMIT;

    EXEC SQL SELECT current_database(), 256 INTO :t:t_ind LIMIT 1;

    printf("t.v = %s\n", t.v.arr);
    printf("t.i = %d\n", t.i);

    printf("t_ind.v_ind = %d\n", t_ind.v_ind);
    printf("t_ind.i_ind = %d\n", t_ind.i_ind);

    EXEC SQL DISCONNECT con1;

    return 0;
}

The output from this program looks like this:

t.v = testdb
t.i = 256
t_ind.v_ind = 0
t_ind.i_ind = 0

Compatibility

The TYPE command is a PostgreSQL extension.

Prev | Up | Next
SET DESCRIPTOR | Home | VAR


VAR
Prev Up 36.14. Embedded SQL Commands Home Next

VAR

VAR — define a variable

Synopsis

VAR varname IS ctype

Description

The VAR command assigns a new C data type to a host variable. The host variable must be previously declared in a declare section.

Parameters

varname

A C variable name.

ctype

A C type specification.

Examples

Exec sql begin declare section;
short a;
exec sql end declare section;
EXEC SQL VAR a IS int;

Compatibility

The VAR command is a PostgreSQL extension.

Prev | Up | Next
TYPE | Home | WHENEVER


WHENEVER
Prev Up 36.14. Embedded SQL Commands Home Next

WHENEVER

WHENEVER — specify the action to be taken when an SQL statement causes a specific class condition to be raised

Synopsis

WHENEVER { NOT FOUND | SQLERROR | SQLWARNING } action

Description

Define a behavior which is called on the special cases (Rows not found, SQL warnings or errors) in the result of SQL execution.

Parameters

See Section 36.8.1 for a description of the parameters.

Examples

EXEC SQL WHENEVER NOT FOUND CONTINUE;
EXEC SQL WHENEVER NOT FOUND DO BREAK;
EXEC SQL WHENEVER NOT FOUND DO CONTINUE;
EXEC SQL WHENEVER SQLWARNING SQLPRINT;
EXEC SQL WHENEVER SQLWARNING DO warn();
EXEC SQL WHENEVER SQLERROR sqlprint;
EXEC SQL WHENEVER SQLERROR CALL print2();
EXEC SQL WHENEVER SQLERROR DO handle_error("select");
EXEC SQL WHENEVER SQLERROR DO sqlnotice(NULL, NONO);
EXEC SQL WHENEVER SQLERROR DO sqlprint();
EXEC SQL WHENEVER SQLERROR GOTO error_label;
EXEC SQL WHENEVER SQLERROR STOP;

A typical application is the use of WHENEVER NOT FOUND BREAK to handle looping through result sets:

int
main(void)
{
    EXEC SQL CONNECT TO testdb AS con1;
    EXEC SQL SELECT pg_catalog.set_config('search_path', '', false); EXEC SQL COMMIT;
    EXEC SQL ALLOCATE DESCRIPTOR d;
    EXEC SQL DECLARE cur CURSOR FOR SELECT current_database(), 'hoge', 256;
    EXEC SQL OPEN cur;

    /* when end of result set reached, break out of while loop */
    EXEC SQL WHENEVER NOT FOUND DO BREAK;

    while (1)
    {
        EXEC SQL FETCH NEXT FROM cur INTO SQL DESCRIPTOR d;
        ...
    }

    EXEC SQL CLOSE cur;
    EXEC SQL COMMIT;

    EXEC SQL DEALLOCATE DESCRIPTOR d;
    EXEC SQL DISCONNECT ALL;

    return 0;
}

Compatibility

WHENEVER is specified in the SQL standard, but most of the actions are PostgreSQL extensions.

Prev | Up | Next
VAR | Home | 36.15. Informix Compatibility Mode


36.15. Informix Compatibility Mode
Prev Up Chapter 36. ECPG — Embedded SQL in C Home Next

36.15. Informix Compatibility Mode

36.15.1. Additional Types
36.15.2. Additional/Missing Embedded SQL Statements
36.15.3. Informix-compatible SQLDA Descriptor Areas
36.15.4. Additional Functions
36.15.5. Additional Constants

ecpg can be run in a so-called Informix compatibility mode. If this mode is active, it tries to behave as if it were the Informix precompiler for Informix E/SQL. Generally spoken this will allow you to use the dollar sign instead of the EXEC SQL primitive to introduce embedded SQL commands:

$int j = 3;
$CONNECT TO :dbname;
$CREATE TABLE test(i INT PRIMARY KEY, j INT);
$INSERT INTO test(i, j) VALUES (7, :j);
$COMMIT;

Note

There must not be any white space between the $ and a following preprocessor directive, that is, include, define, ifdef, etc. Otherwise, the preprocessor will parse the token as a host variable.

There are two compatibility modes: INFORMIX, INFORMIX_SE

When linking programs that use this compatibility mode, remember to link against libcompat that is shipped with ECPG.

Besides the previously explained syntactic sugar, the Informix compatibility mode ports some functions for input, output and transformation of data as well as embedded SQL statements known from E/SQL to ECPG.

Informix compatibility mode is closely connected to the pgtypeslib library of ECPG. pgtypeslib maps SQL data types to data types within the C host program and most of the additional functions of the Informix compatibility mode allow you to operate on those C host program types. Note however that the extent of the compatibility is limited. It does not try to copy Informix behavior; it allows you to do more or less the same operations and gives you functions that have the same name and the same basic behavior but it is no drop-in replacement if you are using Informix at the moment. Moreover, some of the data types are different. For example, PostgreSQL's datetime and interval types do not know about ranges like for example YEAR TO MINUTE so you won't find support in ECPG for that either.

36.15.1. Additional Types

The Informix-special "string" pseudo-type for storing right-trimmed character string data is now supported in Informix-mode without using typedef. In fact, in Informix-mode, ECPG refuses to process source files that contain typedef sometype string;

EXEC SQL BEGIN DECLARE SECTION;
string userid; /* this variable will contain trimmed data */
EXEC SQL END DECLARE SECTION;

EXEC SQL FETCH MYCUR INTO :userid;

36.15.2. Additional/Missing Embedded SQL Statements

CLOSE DATABASE

This statement closes the current connection. In fact, this is a synonym for ECPG's DISCONNECT CURRENT:

$CLOSE DATABASE;                /* close the current connection */
EXEC SQL CLOSE DATABASE;

FREE cursor_name

Due to the differences how ECPG works compared to Informix's ESQL/C (i.e., which steps are purely grammar transformations and which steps rely on the underlying run-time library) there is no FREE cursor_name statement in ECPG. This is because in ECPG, DECLARE CURSOR doesn't translate to a function call into the run-time library that uses to the cursor name. This means that there's no run-time bookkeeping of SQL cursors in the ECPG run-time library, only in the PostgreSQL server.

FREE statement_name

FREE statement_name is a synonym for DEALLOCATE PREPARE statement_name.

36.15.3. Informix-compatible SQLDA Descriptor Areas

Informix-compatible mode supports a different structure than the one described in Section 36.7.2. See below:

struct sqlvar_compat
{
    short   sqltype;
    int     sqllen;
    char   *sqldata;
    short  *sqlind;
    char   *sqlname;
    char   *sqlformat;
    short   sqlitype;
    short   sqlilen;
    char   *sqlidata;
    int     sqlxid;
    char   *sqltypename;
    short   sqltypelen;
    short   sqlownerlen;
    short   sqlsourcetype;
    char   *sqlownername;
    int     sqlsourceid;
    char   *sqlilongdata;
    int     sqlflags;
    void   *sqlreserved;
};

struct sqlda_compat
{
    short  sqld;
    struct sqlvar_compat *sqlvar;
    char   desc_name[19];
    short  desc_occ;
    struct sqlda_compat *desc_next;
    void  *reserved;
};

typedef struct sqlvar_compat    sqlvar_t;
typedef struct sqlda_compat     sqlda_t;

The global properties are:

sqld

The number of fields in the SQLDA descriptor.

sqlvar

Pointer to the per-field properties.

desc_name

Unused, filled with zero-bytes.

desc_occ

Size of the allocated structure.

desc_next

Pointer to the next SQLDA structure if the result set contains more than one record.

reserved

Unused pointer, contains NULL. Kept for Informix-compatibility.

The per-field properties are below, they are stored in the sqlvar array:

sqltype

Type of the field. Constants are in sqltypes.h

sqllen

Length of the field data.

sqldata

Pointer to the field data. The pointer is of char * type, the data pointed by it is in a binary format. Example:

int intval;

switch (sqldata->sqlvar[i].sqltype)
{
    case SQLINTEGER:
        intval = *(int *)sqldata->sqlvar[i].sqldata;
        break;
  ...
}

sqlind

Pointer to the NULL indicator. If returned by DESCRIBE or FETCH then it's always a valid pointer. If used as input for EXECUTE ... USING sqlda; then NULL-pointer value means that the value for this field is non-NULL. Otherwise a valid pointer and sqlitype has to be properly set. Example:

if (*(int2 *)sqldata->sqlvar[i].sqlind != 0)
    printf("value is NULL\n");

sqlname

Name of the field. 0-terminated string.

sqlformat

Reserved in Informix, value of PQfformat for the field.

sqlitype

Type of the NULL indicator data. It's always SQLSMINT when returning data from the server. When the SQLDA is used for a parameterized query, the data is treated according to the set type.

sqlilen

Length of the NULL indicator data.

sqlxid

Extended type of the field, result of PQftype.

sqltypename
sqltypelen
sqlownerlen
sqlsourcetype
sqlownername
sqlsourceid
sqlflags
sqlreserved

Unused.

sqlilongdata

It equals to sqldata if sqllen is larger than 32kB.

Example:

EXEC SQL INCLUDE sqlda.h;

    sqlda_t        *sqlda; /* This doesn't need to be under embedded DECLARE SECTION */

    EXEC SQL BEGIN DECLARE SECTION;
    char *prep_stmt = "select * from table1";
    int i;
    EXEC SQL END DECLARE SECTION;

    ...

    EXEC SQL PREPARE mystmt FROM :prep_stmt;

    EXEC SQL DESCRIBE mystmt INTO sqlda;

    printf("# of fields: %d\n", sqlda->sqld);
    for (i = 0; i < sqlda->sqld; i++)
      printf("field %d: \"%s\"\n", sqlda->sqlvar[i]->sqlname);

    EXEC SQL DECLARE mycursor CURSOR FOR mystmt;
    EXEC SQL OPEN mycursor;
    EXEC SQL WHENEVER NOT FOUND GOTO out;

    while (1)
    {
      EXEC SQL FETCH mycursor USING sqlda;
    }

    EXEC SQL CLOSE mycursor;

    free(sqlda); /* The main structure is all to be free(),
                  * sqlda and sqlda->sqlvar is in one allocated area */

For more information, see the sqlda.h header and the src/interfaces/ecpg/test/compat_informix/sqlda.pgc regression test.

36.15.4. Additional Functions

decadd

Add two decimal type values.

int decadd(decimal *arg1, decimal *arg2, decimal *sum);

The function receives a pointer to the first operand of type decimal (arg1), a pointer to the second operand of type decimal (arg2) and a pointer to a value of type decimal that will contain the sum (sum). On success, the function returns 0. ECPG_INFORMIX_NUM_OVERFLOW is returned in case of overflow and ECPG_INFORMIX_NUM_UNDERFLOW in case of underflow. -1 is returned for other failures and errno is set to the respective errno number of the pgtypeslib.

deccmp

Compare two variables of type decimal.

int deccmp(decimal *arg1, decimal *arg2);

The function receives a pointer to the first decimal value (arg1), a pointer to the second decimal value (arg2) and returns an integer value that indicates which is the bigger value.

  • 1, if the value that arg1 points to is bigger than the value that var2 points to

  • -1, if the value that arg1 points to is smaller than the value that arg2 points to

  • 0, if the value that arg1 points to and the value that arg2 points to are equal

deccopy

Copy a decimal value.

void deccopy(decimal *src, decimal *target);

The function receives a pointer to the decimal value that should be copied as the first argument (src) and a pointer to the target structure of type decimal (target) as the second argument.

deccvasc

Convert a value from its ASCII representation into a decimal type.

int deccvasc(char *cp, int len, decimal *np);

The function receives a pointer to string that contains the string representation of the number to be converted (cp) as well as its length len. np is a pointer to the decimal value that saves the result of the operation.

Valid formats are for example: -2, .794, +3.44, 592.49E07 or -32.84e-4.

The function returns 0 on success. If overflow or underflow occurred, ECPG_INFORMIX_NUM_OVERFLOW or ECPG_INFORMIX_NUM_UNDERFLOW is returned. If the ASCII representation could not be parsed, ECPG_INFORMIX_BAD_NUMERIC is returned or ECPG_INFORMIX_BAD_EXPONENT if this problem occurred while parsing the exponent.

deccvdbl

Convert a value of type double to a value of type decimal.

int deccvdbl(double dbl, decimal *np);

The function receives the variable of type double that should be converted as its first argument (dbl). As the second argument (np), the function receives a pointer to the decimal variable that should hold the result of the operation.

The function returns 0 on success and a negative value if the conversion failed.

deccvint

Convert a value of type int to a value of type decimal.

int deccvint(int in, decimal *np);

The function receives the variable of type int that should be converted as its first argument (in). As the second argument (np), the function receives a pointer to the decimal variable that should hold the result of the operation.

The function returns 0 on success and a negative value if the conversion failed.

deccvlong

Convert a value of type long to a value of type decimal.

int deccvlong(long lng, decimal *np);

The function receives the variable of type long that should be converted as its first argument (lng). As the second argument (np), the function receives a pointer to the decimal variable that should hold the result of the operation.

The function returns 0 on success and a negative value if the conversion failed.

decdiv

Divide two variables of type decimal.

int decdiv(decimal *n1, decimal *n2, decimal *result);

The function receives pointers to the variables that are the first (n1) and the second (n2) operands and calculates n1/n2. result is a pointer to the variable that should hold the result of the operation.

On success, 0 is returned and a negative value if the division fails. If overflow or underflow occurred, the function returns ECPG_INFORMIX_NUM_OVERFLOW or ECPG_INFORMIX_NUM_UNDERFLOW respectively. If an attempt to divide by zero is observed, the function returns ECPG_INFORMIX_DIVIDE_ZERO.

decmul

Multiply two decimal values.

int decmul(decimal *n1, decimal *n2, decimal *result);

The function receives pointers to the variables that are the first (n1) and the second (n2) operands and calculates n1*n2. result is a pointer to the variable that should hold the result of the operation.

On success, 0 is returned and a negative value if the multiplication fails. If overflow or underflow occurred, the function returns ECPG_INFORMIX_NUM_OVERFLOW or ECPG_INFORMIX_NUM_UNDERFLOW respectively.

decsub

Subtract one decimal value from another.

int decsub(decimal *n1, decimal *n2, decimal *result);

The function receives pointers to the variables that are the first (n1) and the second (n2) operands and calculates n1-n2. result is a pointer to the variable that should hold the result of the operation.

On success, 0 is returned and a negative value if the subtraction fails. If overflow or underflow occurred, the function returns ECPG_INFORMIX_NUM_OVERFLOW or ECPG_INFORMIX_NUM_UNDERFLOW respectively.

dectoasc

Convert a variable of type decimal to its ASCII representation in a C char* string.

int dectoasc(decimal *np, char *cp, int len, int right)

The function receives a pointer to a variable of type decimal (np) that it converts to its textual representation. cp is the buffer that should hold the result of the operation. The parameter right specifies, how many digits right of the decimal point should be included in the output. The result will be rounded to this number of decimal digits. Setting right to -1 indicates that all available decimal digits should be included in the output. If the length of the output buffer, which is indicated by len is not sufficient to hold the textual representation including the trailing zero byte, only a single * character is stored in the result and -1 is returned.

The function returns either -1 if the buffer cp was too small or ECPG_INFORMIX_OUT_OF_MEMORY if memory was exhausted.

dectodbl

Convert a variable of type decimal to a double.

int dectodbl(decimal *np, double *dblp);

The function receives a pointer to the decimal value to convert (np) and a pointer to the double variable that should hold the result of the operation (dblp).

On success, 0 is returned and a negative value if the conversion failed.

dectoint

Convert a variable to type decimal to an integer.

int dectoint(decimal *np, int *ip);

The function receives a pointer to the decimal value to convert (np) and a pointer to the integer variable that should hold the result of the operation (ip).

On success, 0 is returned and a negative value if the conversion failed. If an overflow occurred, ECPG_INFORMIX_NUM_OVERFLOW is returned.

Note that the ECPG implementation differs from the Informix implementation. Informix limits an integer to the range from -32767 to 32767, while the limits in the ECPG implementation depend on the architecture (INT_MIN .. INT_MAX).

dectolong

Convert a variable to type decimal to a long integer.

int dectolong(decimal *np, long *lngp);

The function receives a pointer to the decimal value to convert (np) and a pointer to the long variable that should hold the result of the operation (lngp).

On success, 0 is returned and a negative value if the conversion failed. If an overflow occurred, ECPG_INFORMIX_NUM_OVERFLOW is returned.

Note that the ECPG implementation differs from the Informix implementation. Informix limits a long integer to the range from -2,147,483,647 to 2,147,483,647, while the limits in the ECPG implementation depend on the architecture (-LONG_MAX .. LONG_MAX).

rdatestr

Converts a date to a C char* string.

int rdatestr(date d, char *str);

The function receives two arguments, the first one is the date to convert (d) and the second one is a pointer to the target string. The output format is always yyyy-mm-dd, so you need to allocate at least 11 bytes (including the zero-byte terminator) for the string.

The function returns 0 on success and a negative value in case of error.

Note that ECPG's implementation differs from the Informix implementation. In Informix the format can be influenced by setting environment variables. In ECPG however, you cannot change the output format.

rstrdate

Parse the textual representation of a date.

int rstrdate(char *str, date *d);

The function receives the textual representation of the date to convert (str) and a pointer to a variable of type date (d). This function does not allow you to specify a format mask. It uses the default format mask of Informix which is mm/dd/yyyy. Internally, this function is implemented by means of rdefmtdate. Therefore, rstrdate is not faster and if you have the choice you should opt for rdefmtdate which allows you to specify the format mask explicitly.

The function returns the same values as rdefmtdate.

rtoday

Get the current date.

void rtoday(date *d);

The function receives a pointer to a date variable (d) that it sets to the current date.

Internally this function uses the PGTYPESdate_today function.

rjulmdy

Extract the values for the day, the month and the year from a variable of type date.

int rjulmdy(date d, short mdy[3]);

The function receives the date d and a pointer to an array of 3 short integer values mdy. The variable name indicates the sequential order: mdy[0] will be set to contain the number of the month, mdy[1] will be set to the value of the day and mdy[2] will contain the year.

The function always returns 0 at the moment.

Internally the function uses the PGTYPESdate_julmdy function.

rdefmtdate

Use a format mask to convert a character string to a value of type date.

int rdefmtdate(date *d, char *fmt, char *str);

The function receives a pointer to the date value that should hold the result of the operation (d), the format mask to use for parsing the date (fmt) and the C char* string containing the textual representation of the date (str). The textual representation is expected to match the format mask. However you do not need to have a 1:1 mapping of the string to the format mask. The function only analyzes the sequential order and looks for the literals yy or yyyy that indicate the position of the year, mm to indicate the position of the month and dd to indicate the position of the day.

The function returns the following values:

  • 0 - The function terminated successfully.

  • ECPG_INFORMIX_ENOSHORTDATE - The date does not contain delimiters between day, month and year. In this case the input string must be exactly 6 or 8 bytes long but isn't.

  • ECPG_INFORMIX_ENOTDMY - The format string did not correctly indicate the sequential order of year, month and day.

  • ECPG_INFORMIX_BAD_DAY - The input string does not contain a valid day.

  • ECPG_INFORMIX_BAD_MONTH - The input string does not contain a valid month.

  • ECPG_INFORMIX_BAD_YEAR - The input string does not contain a valid year.

Internally this function is implemented to use the PGTYPESdate_defmt_asc function. See the reference there for a table of example input.

rfmtdate

Convert a variable of type date to its textual representation using a format mask.

int rfmtdate(date d, char *fmt, char *str);

The function receives the date to convert (d), the format mask (fmt) and the string that will hold the textual representation of the date (str).

On success, 0 is returned and a negative value if an error occurred.

Internally this function uses the PGTYPESdate_fmt_asc function, see the reference there for examples.

rmdyjul

Create a date value from an array of 3 short integers that specify the day, the month and the year of the date.

int rmdyjul(short mdy[3], date *d);

The function receives the array of the 3 short integers (mdy) and a pointer to a variable of type date that should hold the result of the operation.

Currently the function returns always 0.

Internally the function is implemented to use the function PGTYPESdate_mdyjul.

rdayofweek

Return a number representing the day of the week for a date value.

int rdayofweek(date d);

The function receives the date variable d as its only argument and returns an integer that indicates the day of the week for this date.

  • 0 - Sunday

  • 1 - Monday

  • 2 - Tuesday

  • 3 - Wednesday

  • 4 - Thursday

  • 5 - Friday

  • 6 - Saturday

Internally the function is implemented to use the function PGTYPESdate_dayofweek.

dtcurrent

Retrieve the current timestamp.

void dtcurrent(timestamp *ts);

The function retrieves the current timestamp and saves it into the timestamp variable that ts points to.

dtcvasc

Parses a timestamp from its textual representation into a timestamp variable.

int dtcvasc(char *str, timestamp *ts);

The function receives the string to parse (str) and a pointer to the timestamp variable that should hold the result of the operation (ts).

The function returns 0 on success and a negative value in case of error.

Internally this function uses the PGTYPEStimestamp_from_asc function. See the reference there for a table with example inputs.

dtcvfmtasc

Parses a timestamp from its textual representation using a format mask into a timestamp variable.

dtcvfmtasc(char *inbuf, char *fmtstr, timestamp *dtvalue)

The function receives the string to parse (inbuf), the format mask to use (fmtstr) and a pointer to the timestamp variable that should hold the result of the operation (dtvalue).

This function is implemented by means of the PGTYPEStimestamp_defmt_asc function. See the documentation there for a list of format specifiers that can be used.

The function returns 0 on success and a negative value in case of error.

dtsub

Subtract one timestamp from another and return a variable of type interval.

int dtsub(timestamp *ts1, timestamp *ts2, interval *iv);

The function will subtract the timestamp variable that ts2 points to from the timestamp variable that ts1 points to and will store the result in the interval variable that iv points to.

Upon success, the function returns 0 and a negative value if an error occurred.

dttoasc

Convert a timestamp variable to a C char* string.

int dttoasc(timestamp *ts, char *output);

The function receives a pointer to the timestamp variable to convert (ts) and the string that should hold the result of the operation (output). It converts ts to its textual representation according to the SQL standard, which is be YYYY-MM-DD HH:MM:SS.

Upon success, the function returns 0 and a negative value if an error occurred.

dttofmtasc

Convert a timestamp variable to a C char* using a format mask.

int dttofmtasc(timestamp *ts, char *output, int str_len, char *fmtstr);

The function receives a pointer to the timestamp to convert as its first argument (ts), a pointer to the output buffer (output), the maximal length that has been allocated for the output buffer (str_len) and the format mask to use for the conversion (fmtstr).

Upon success, the function returns 0 and a negative value if an error occurred.

Internally, this function uses the PGTYPEStimestamp_fmt_asc function. See the reference there for information on what format mask specifiers can be used.

intoasc

Convert an interval variable to a C char* string.

int intoasc(interval *i, char *str);

The function receives a pointer to the interval variable to convert (i) and the string that should hold the result of the operation (str). It converts i to its textual representation according to the SQL standard, which is be YYYY-MM-DD HH:MM:SS.

Upon success, the function returns 0 and a negative value if an error occurred.

rfmtlong

Convert a long integer value to its textual representation using a format mask.

int rfmtlong(long lng_val, char *fmt, char *outbuf);

The function receives the long value lng_val, the format mask fmt and a pointer to the output buffer outbuf. It converts the long value according to the format mask to its textual representation.

The format mask can be composed of the following format specifying characters:

  • * (asterisk) - if this position would be blank otherwise, fill it with an asterisk.

  • & (ampersand) - if this position would be blank otherwise, fill it with a zero.

  • # - turn leading zeroes into blanks.

  • < - left-justify the number in the string.

  • , (comma) - group numbers of four or more digits into groups of three digits separated by a comma.

  • . (period) - this character separates the whole-number part of the number from the fractional part.

  • - (minus) - the minus sign appears if the number is a negative value.

  • + (plus) - the plus sign appears if the number is a positive value.

  • ( - this replaces the minus sign in front of the negative number. The minus sign will not appear.

  • ) - this character replaces the minus and is printed behind the negative value.

  • $ - the currency symbol.

rupshift

Convert a string to upper case.

void rupshift(char *str);

The function receives a pointer to the string and transforms every lower case character to upper case.

byleng

Return the number of characters in a string without counting trailing blanks.

int byleng(char *str, int len);

The function expects a fixed-length string as its first argument (str) and its length as its second argument (len). It returns the number of significant characters, that is the length of the string without trailing blanks.

ldchar

Copy a fixed-length string into a null-terminated string.

void ldchar(char *src, int len, char *dest);

The function receives the fixed-length string to copy (src), its length (len) and a pointer to the destination memory (dest). Note that you need to reserve at least len+1 bytes for the string that dest points to. The function copies at most len bytes to the new location (less if the source string has trailing blanks) and adds the null-terminator.

rgetmsg

int rgetmsg(int msgnum, char *s, int maxsize);

This function exists but is not implemented at the moment!

rtypalign

int rtypalign(int offset, int type);

This function exists but is not implemented at the moment!

rtypmsize

int rtypmsize(int type, int len);

This function exists but is not implemented at the moment!

rtypwidth

int rtypwidth(int sqltype, int sqllen);

This function exists but is not implemented at the moment!

rsetnull

Set a variable to NULL.

int rsetnull(int t, char *ptr);

The function receives an integer that indicates the type of the variable and a pointer to the variable itself that is cast to a C char* pointer.

The following types exist:

  • CCHARTYPE - For a variable of type char or char*

  • CSHORTTYPE - For a variable of type short int

  • CINTTYPE - For a variable of type int

  • CBOOLTYPE - For a variable of type boolean

  • CFLOATTYPE - For a variable of type float

  • CLONGTYPE - For a variable of type long

  • CDOUBLETYPE - For a variable of type double

  • CDECIMALTYPE - For a variable of type decimal

  • CDATETYPE - For a variable of type date

  • CDTIMETYPE - For a variable of type timestamp

Here is an example of a call to this function:

$char c[] = "abc       ";
$short s = 17;
$int i = -74874;

rsetnull(CCHARTYPE, (char *) c);
rsetnull(CSHORTTYPE, (char *) &s);
rsetnull(CINTTYPE, (char *) &i);

risnull

Test if a variable is NULL.

int risnull(int t, char *ptr);

The function receives the type of the variable to test (t) as well a pointer to this variable (ptr). Note that the latter needs to be cast to a char*. See the function rsetnull for a list of possible variable types.

Here is an example of how to use this function:

$char c[] = "abc       ";
$short s = 17;
$int i = -74874;

risnull(CCHARTYPE, (char *) c);
risnull(CSHORTTYPE, (char *) &s);
risnull(CINTTYPE, (char *) &i);

36.15.5. Additional Constants

Note that all constants here describe errors and all of them are defined to represent negative values. In the descriptions of the different constants you can also find the value that the constants represent in the current implementation. However you should not rely on this number. You can however rely on the fact all of them are defined to represent negative values.

ECPG_INFORMIX_NUM_OVERFLOW

Functions return this value if an overflow occurred in a calculation. Internally it is defined as -1200 (the Informix definition).

ECPG_INFORMIX_NUM_UNDERFLOW

Functions return this value if an underflow occurred in a calculation. Internally it is defined as -1201 (the Informix definition).

ECPG_INFORMIX_DIVIDE_ZERO

Functions return this value if an attempt to divide by zero is observed. Internally it is defined as -1202 (the Informix definition).

ECPG_INFORMIX_BAD_YEAR

Functions return this value if a bad value for a year was found while parsing a date. Internally it is defined as -1204 (the Informix definition).

ECPG_INFORMIX_BAD_MONTH

Functions return this value if a bad value for a month was found while parsing a date. Internally it is defined as -1205 (the Informix definition).

ECPG_INFORMIX_BAD_DAY

Functions return this value if a bad value for a day was found while parsing a date. Internally it is defined as -1206 (the Informix definition).

ECPG_INFORMIX_ENOSHORTDATE

Functions return this value if a parsing routine needs a short date representation but did not get the date string in the right length. Internally it is defined as -1209 (the Informix definition).

ECPG_INFORMIX_DATE_CONVERT

Functions return this value if an error occurred during date formatting. Internally it is defined as -1210 (the Informix definition).

ECPG_INFORMIX_OUT_OF_MEMORY

Functions return this value if memory was exhausted during their operation. Internally it is defined as -1211 (the Informix definition).

ECPG_INFORMIX_ENOTDMY

Functions return this value if a parsing routine was supposed to get a format mask (like mmddyy) but not all fields were listed correctly. Internally it is defined as -1212 (the Informix definition).

ECPG_INFORMIX_BAD_NUMERIC

Functions return this value either if a parsing routine cannot parse the textual representation for a numeric value because it contains errors or if a routine cannot complete a calculation involving numeric variables because at least one of the numeric variables is invalid. Internally it is defined as -1213 (the Informix definition).

ECPG_INFORMIX_BAD_EXPONENT

Functions return this value if a parsing routine cannot parse an exponent. Internally it is defined as -1216 (the Informix definition).

ECPG_INFORMIX_BAD_DATE

Functions return this value if a parsing routine cannot parse a date. Internally it is defined as -1218 (the Informix definition).

ECPG_INFORMIX_EXTRA_CHARS

Functions return this value if a parsing routine is passed extra characters it cannot parse. Internally it is defined as -1264 (the Informix definition).

Prev | Up | Next
WHENEVER | Home | 36.16. Internals


36.16. Internals
Prev Up Chapter 36. ECPG — Embedded SQL in C Home Next

36.16. Internals

This section explains how ECPG works internally. This information can occasionally be useful to help users understand how to use ECPG.

The first four lines written by ecpg to the output are fixed lines. Two are comments and two are include lines necessary to interface to the library. Then the preprocessor reads through the file and writes output. Normally it just echoes everything to the output.

When it sees an EXEC SQL statement, it intervenes and changes it. The command starts with EXEC SQL and ends with ;. Everything in between is treated as an SQL statement and parsed for variable substitution.

Variable substitution occurs when a symbol starts with a colon (:). The variable with that name is looked up among the variables that were previously declared within a EXEC SQL DECLARE section.

The most important function in the library is ECPGdo, which takes care of executing most commands. It takes a variable number of arguments. This can easily add up to 50 or so arguments, and we hope this will not be a problem on any platform.

The arguments are:

A line number

This is the line number of the original line; used in error messages only.

A string

This is the SQL command that is to be issued. It is modified by the input variables, i.e., the variables that where not known at compile time but are to be entered in the command. Where the variables should go the string contains ?.

Input variables

Every input variable causes ten arguments to be created. (See below.)

ECPGt_EOIT

An enum telling that there are no more input variables.

Output variables

Every output variable causes ten arguments to be created. (See below.) These variables are filled by the function.

ECPGt_EORT

An enum telling that there are no more variables.

For every variable that is part of the SQL command, the function gets ten arguments:

  1. The type as a special symbol.

  2. A pointer to the value or a pointer to the pointer.

  3. The size of the variable if it is a char or varchar.

  4. The number of elements in the array (for array fetches).

  5. The offset to the next element in the array (for array fetches).

  6. The type of the indicator variable as a special symbol.

  7. A pointer to the indicator variable.

  8. 0

  9. The number of elements in the indicator array (for array fetches).

  10. The offset to the next element in the indicator array (for array fetches).

Note that not all SQL commands are treated in this way. For instance, an open cursor statement like:

EXEC SQL OPEN cursor;

is not copied to the output. Instead, the cursor's DECLARE command is used at the position of the OPEN command because it indeed opens the cursor.

Here is a complete example describing the output of the preprocessor of a file foo.pgc (details might change with each particular version of the preprocessor):

EXEC SQL BEGIN DECLARE SECTION;
int index;
int result;
EXEC SQL END DECLARE SECTION;
...
EXEC SQL SELECT res INTO :result FROM mytable WHERE index = :index;

is translated into:

/* Processed by ecpg (2.6.0) */
/* These two include files are added by the preprocessor */
#include <ecpgtype.h>;
#include <ecpglib.h>;

/* exec sql begin declare section */

#line 1 "foo.pgc"

 int index;
 int result;
/* exec sql end declare section */
...
ECPGdo(__LINE__, NULL, "SELECT res FROM mytable WHERE index = ?     ",
        ECPGt_int,&(index),1L,1L,sizeof(int),
        ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L, ECPGt_EOIT,
        ECPGt_int,&(result),1L,1L,sizeof(int),
        ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L, ECPGt_EORT);
#line 147 "foo.pgc"

(The indentation here is added for readability and not something the preprocessor does.)

Prev | Up | Next
36.15. Informix Compatibility Mode | Home | Chapter 37. The Information Schema


Chapter 37. The Information Schema
Prev Up Part IV. Client Interfaces Home Next

Chapter 37. The Information Schema

Table of Contents

37.1. The Schema
37.2. Data Types
37.3. information_schema_catalog_name
37.4. administrable_role_​authorizations
37.5. applicable_roles
37.6. attributes
37.7. character_sets
37.8. check_constraint_routine_usage
37.9. check_constraints
37.10. collations
37.11. collation_character_set_​applicability
37.12. column_column_usage
37.13. column_domain_usage
37.14. column_options
37.15. column_privileges
37.16. column_udt_usage
37.17. columns
37.18. constraint_column_usage
37.19. constraint_table_usage
37.20. data_type_privileges
37.21. domain_constraints
37.22. domain_udt_usage
37.23. domains
37.24. element_types
37.25. enabled_roles
37.26. foreign_data_wrapper_options
37.27. foreign_data_wrappers
37.28. foreign_server_options
37.29. foreign_servers
37.30. foreign_table_options
37.31. foreign_tables
37.32. key_column_usage
37.33. parameters
37.34. referential_constraints
37.35. role_column_grants
37.36. role_routine_grants
37.37. role_table_grants
37.38. role_udt_grants
37.39. role_usage_grants
37.40. routine_column_usage
37.41. routine_privileges
37.42. routine_routine_usage
37.43. routine_sequence_usage
37.44. routine_table_usage
37.45. routines
37.46. schemata
37.47. sequences
37.48. sql_features
37.49. sql_implementation_info
37.50. sql_parts
37.51. sql_sizing
37.52. table_constraints
37.53. table_privileges
37.54. tables
37.55. transforms
37.56. triggered_update_columns
37.57. triggers
37.58. udt_privileges
37.59. usage_privileges
37.60. user_defined_types
37.61. user_mapping_options
37.62. user_mappings
37.63. view_column_usage
37.64. view_routine_usage
37.65. view_table_usage
37.66. views

The information schema consists of a set of views that contain information about the objects defined in the current database. The information schema is defined in the SQL standard and can therefore be expected to be portable and remain stable — unlike the system catalogs, which are specific to PostgreSQL and are modeled after implementation concerns. The information schema views do not, however, contain information about PostgreSQL-specific features; to inquire about those you need to query the system catalogs or other PostgreSQL-specific views.

Note

When querying the database for constraint information, it is possible for a standard-compliant query that expects to return one row to return several. This is because the SQL standard requires constraint names to be unique within a schema, but PostgreSQL does not enforce this restriction. PostgreSQL automatically-generated constraint names avoid duplicates in the same schema, but users can specify such duplicate names.

This problem can appear when querying information schema views such as check_constraint_routine_usage, check_constraints, domain_constraints, and referential_constraints. Some other views have similar issues but contain the table name to help distinguish duplicate rows, e.g., constraint_column_usage, constraint_table_usage, table_constraints.

Prev | Up | Next
36.16. Internals | Home | 37.1. The Schema


37.1. The Schema
Prev Up Chapter 37. The Information Schema Home Next

37.1. The Schema

The information schema itself is a schema named information_schema. This schema automatically exists in all databases. The owner of this schema is the initial database user in the cluster, and that user naturally has all the privileges on this schema, including the ability to drop it (but the space savings achieved by that are minuscule).

By default, the information schema is not in the schema search path, so you need to access all objects in it through qualified names. Since the names of some of the objects in the information schema are generic names that might occur in user applications, you should be careful if you want to put the information schema in the path.

Prev | Up | Next
Chapter 37. The Information Schema | Home | 37.2. Data Types


37.2. Data Types
Prev Up Chapter 37. The Information Schema Home Next

37.2. Data Types

The columns of the information schema views use special data types that are defined in the information schema. These are defined as simple domains over ordinary built-in types. You should not use these types for work outside the information schema, but your applications must be prepared for them if they select from the information schema.

These types are:

cardinal_number

A nonnegative integer.

character_data

A character string (without specific maximum length).

sql_identifier

A character string. This type is used for SQL identifiers, the type character_data is used for any other kind of text data.

time_stamp

A domain over the type timestamp with time zone

yes_or_no

A character string domain that contains either YES or NO. This is used to represent Boolean (true/false) data in the information schema. (The information schema was invented before the type boolean was added to the SQL standard, so this convention is necessary to keep the information schema backward compatible.)

Every column in the information schema has one of these five types.

Prev | Up | Next
37.1. The Schema | Home | 37.3. information_schema_catalog_name


37.3. information_schema_catalog_name
Prev Up Chapter 37. The Information Schema Home Next

37.3. information_schema_catalog_name

information_schema_catalog_name is a table that always contains one row and one column containing the name of the current database (current catalog, in SQL terminology).

Table 37.1. information_schema_catalog_name Columns

Column Type

Description

catalog_name sql_identifier

Name of the database that contains this information schema

Prev | Up | Next
37.2. Data Types | Home | 37.4. administrable_role_​authorizations


37.4. administrable_role_​authorizations
Prev Up Chapter 37. The Information Schema Home Next

37.4. administrable_role_​authorizations

The view administrable_role_authorizations identifies all roles that the current user has the admin option for.

Table 37.2. administrable_role_authorizations Columns

Column Type

Description

grantee sql_identifier

Name of the role to which this role membership was granted (can be the current user, or a different role in case of nested role memberships)

role_name sql_identifier

Name of a role

is_grantable yes_or_no

Always YES

Prev | Up | Next
37.3. information_schema_catalog_name | Home | 37.5. applicable_roles


37.5. applicable_roles
Prev Up Chapter 37. The Information Schema Home Next

37.5. applicable_roles

The view applicable_roles identifies all roles whose privileges the current user can use. This means there is some chain of role grants from the current user to the role in question. The current user itself is also an applicable role. The set of applicable roles is generally used for permission checking.

Table 37.3. applicable_roles Columns

Column Type

Description

grantee sql_identifier

Name of the role to which this role membership was granted (can be the current user, or a different role in case of nested role memberships)

role_name sql_identifier

Name of a role

is_grantable yes_or_no

YES if the grantee has the admin option on the role, NO if not

Prev | Up | Next
37.4. administrable_role_​authorizations | Home | 37.6. attributes


37.6. attributes
Prev Up Chapter 37. The Information Schema Home Next

37.6. attributes

The view attributes contains information about the attributes of composite data types defined in the database. (Note that the view does not give information about table columns, which are sometimes called attributes in PostgreSQL contexts.) Only those attributes are shown that the current user has access to (by way of being the owner of or having some privilege on the type).

Table 37.4. attributes Columns

Column Type

Description

udt_catalog sql_identifier

Name of the database containing the data type (always the current database)

udt_schema sql_identifier

Name of the schema containing the data type

udt_name sql_identifier

Name of the data type

attribute_name sql_identifier

Name of the attribute

ordinal_position cardinal_number

Ordinal position of the attribute within the data type (count starts at 1)

attribute_default character_data

Default expression of the attribute

is_nullable yes_or_no

YES if the attribute is possibly nullable, NO if it is known not nullable.

data_type character_data

Data type of the attribute, if it is a built-in type, or ARRAY if it is some array (in that case, see the view element_types), else USER-DEFINED (in that case, the type is identified in attribute_udt_name and associated columns).

character_maximum_length cardinal_number

If data_type identifies a character or bit string type, the declared maximum length; null for all other data types or if no maximum length was declared.

character_octet_length cardinal_number

If data_type identifies a character type, the maximum possible length in octets (bytes) of a datum; null for all other data types. The maximum octet length depends on the declared character maximum length (see above) and the server encoding.

character_set_catalog sql_identifier

Applies to a feature not available in PostgreSQL

character_set_schema sql_identifier

Applies to a feature not available in PostgreSQL

character_set_name sql_identifier

Applies to a feature not available in PostgreSQL

collation_catalog sql_identifier

Name of the database containing the collation of the attribute (always the current database), null if default or the data type of the attribute is not collatable

collation_schema sql_identifier

Name of the schema containing the collation of the attribute, null if default or the data type of the attribute is not collatable

collation_name sql_identifier

Name of the collation of the attribute, null if default or the data type of the attribute is not collatable

numeric_precision cardinal_number

If data_type identifies a numeric type, this column contains the (declared or implicit) precision of the type for this attribute. The precision indicates the number of significant digits. It can be expressed in decimal (base 10) or binary (base 2) terms, as specified in the column numeric_precision_radix. For all other data types, this column is null.

numeric_precision_radix cardinal_number

If data_type identifies a numeric type, this column indicates in which base the values in the columns numeric_precision and numeric_scale are expressed. The value is either 2 or 10. For all other data types, this column is null.

numeric_scale cardinal_number

If data_type identifies an exact numeric type, this column contains the (declared or implicit) scale of the type for this attribute. The scale indicates the number of significant digits to the right of the decimal point. It can be expressed in decimal (base 10) or binary (base 2) terms, as specified in the column numeric_precision_radix. For all other data types, this column is null.

datetime_precision cardinal_number

If data_type identifies a date, time, timestamp, or interval type, this column contains the (declared or implicit) fractional seconds precision of the type for this attribute, that is, the number of decimal digits maintained following the decimal point in the seconds value. For all other data types, this column is null.

interval_type character_data

If data_type identifies an interval type, this column contains the specification which fields the intervals include for this attribute, e.g., YEAR TO MONTH, DAY TO SECOND, etc. If no field restrictions were specified (that is, the interval accepts all fields), and for all other data types, this field is null.

interval_precision cardinal_number

Applies to a feature not available in PostgreSQL (see datetime_precision for the fractional seconds precision of interval type attributes)

attribute_udt_catalog sql_identifier

Name of the database that the attribute data type is defined in (always the current database)

attribute_udt_schema sql_identifier

Name of the schema that the attribute data type is defined in

attribute_udt_name sql_identifier

Name of the attribute data type

scope_catalog sql_identifier

Applies to a feature not available in PostgreSQL

scope_schema sql_identifier

Applies to a feature not available in PostgreSQL

scope_name sql_identifier

Applies to a feature not available in PostgreSQL

maximum_cardinality cardinal_number

Always null, because arrays always have unlimited maximum cardinality in PostgreSQL

dtd_identifier sql_identifier

An identifier of the data type descriptor of the column, unique among the data type descriptors pertaining to the table. This is mainly useful for joining with other instances of such identifiers. (The specific format of the identifier is not defined and not guaranteed to remain the same in future versions.)

is_derived_reference_attribute yes_or_no

Applies to a feature not available in PostgreSQL

See also under Section 37.17, a similarly structured view, for further information on some of the columns.

Prev | Up | Next
37.5. applicable_roles | Home | 37.7. character_sets


37.7. character_sets
Prev Up Chapter 37. The Information Schema Home Next

37.7. character_sets

The view character_sets identifies the character sets available in the current database. Since PostgreSQL does not support multiple character sets within one database, this view only shows one, which is the database encoding.

Take note of how the following terms are used in the SQL standard:

character repertoire

An abstract collection of characters, for example UNICODE, UCS, or LATIN1. Not exposed as an SQL object, but visible in this view.

character encoding form

An encoding of some character repertoire. Most older character repertoires only use one encoding form, and so there are no separate names for them (e.g., LATIN1 is an encoding form applicable to the LATIN1 repertoire). But for example Unicode has the encoding forms UTF8, UTF16, etc. (not all supported by PostgreSQL). Encoding forms are not exposed as an SQL object, but are visible in this view.

character set

A named SQL object that identifies a character repertoire, a character encoding, and a default collation. A predefined character set would typically have the same name as an encoding form, but users could define other names. For example, the character set UTF8 would typically identify the character repertoire UCS, encoding form UTF8, and some default collation.

You can think of an “encoding” in PostgreSQL either as a character set or a character encoding form. They will have the same name, and there can only be one in one database.

Table 37.5. character_sets Columns

Column Type

Description

character_set_catalog sql_identifier

Character sets are currently not implemented as schema objects, so this column is null.

character_set_schema sql_identifier

Character sets are currently not implemented as schema objects, so this column is null.

character_set_name sql_identifier

Name of the character set, currently implemented as showing the name of the database encoding

character_repertoire sql_identifier

Character repertoire, showing UCS if the encoding is UTF8, else just the encoding name

form_of_use sql_identifier

Character encoding form, same as the database encoding

default_collate_catalog sql_identifier

Name of the database containing the default collation (always the current database, if any collation is identified)

default_collate_schema sql_identifier

Name of the schema containing the default collation

default_collate_name sql_identifier

Name of the default collation. The default collation is identified as the collation that matches the COLLATE and CTYPE settings of the current database. If there is no such collation, then this column and the associated schema and catalog columns are null.

Prev | Up | Next
37.6. attributes | Home | 37.8. check_constraint_routine_usage


37.8. check_constraint_routine_usage
Prev Up Chapter 37. The Information Schema Home Next

37.8. check_constraint_routine_usage

The view check_constraint_routine_usage identifies routines (functions and procedures) that are used by a check constraint. Only those routines are shown that are owned by a currently enabled role.

Table 37.6. check_constraint_routine_usage Columns

Column Type

Description

constraint_catalog sql_identifier

Name of the database containing the constraint (always the current database)

constraint_schema sql_identifier

Name of the schema containing the constraint

constraint_name sql_identifier

Name of the constraint

specific_catalog sql_identifier

Name of the database containing the function (always the current database)

specific_schema sql_identifier

Name of the schema containing the function

specific_name sql_identifier

The “specific name” of the function. See Section 37.45 for more information.

Prev | Up | Next
37.7. character_sets | Home | 37.9. check_constraints


37.9. check_constraints
Prev Up Chapter 37. The Information Schema Home Next

37.9. check_constraints

The view check_constraints contains all check constraints, either defined on a table or on a domain, that are owned by a currently enabled role. (The owner of the table or domain is the owner of the constraint.)

Table 37.7. check_constraints Columns

Column Type

Description

constraint_catalog sql_identifier

Name of the database containing the constraint (always the current database)

constraint_schema sql_identifier

Name of the schema containing the constraint

constraint_name sql_identifier

Name of the constraint

check_clause character_data

The check expression of the check constraint

Prev | Up | Next
37.8. check_constraint_routine_usage | Home | 37.10. collations


37.10. collations
Prev Up Chapter 37. The Information Schema Home Next

37.10. collations

The view collations contains the collations available in the current database.

Table 37.8. collations Columns

Column Type

Description

collation_catalog sql_identifier

Name of the database containing the collation (always the current database)

collation_schema sql_identifier

Name of the schema containing the collation

collation_name sql_identifier

Name of the default collation

pad_attribute character_data

Always NO PAD (The alternative PAD SPACE is not supported by PostgreSQL.)

Prev | Up | Next
37.9. check_constraints | Home | 37.11. collation_character_set_​applicability


37.11. collation_character_set_​applicability
Prev Up Chapter 37. The Information Schema Home Next

37.11. collation_character_set_​applicability

The view collation_character_set_applicability identifies which character set the available collations are applicable to. In PostgreSQL, there is only one character set per database (see explanation in Section 37.7), so this view does not provide much useful information.

Table 37.9. collation_character_set_applicability Columns

Column Type

Description

collation_catalog sql_identifier

Name of the database containing the collation (always the current database)

collation_schema sql_identifier

Name of the schema containing the collation

collation_name sql_identifier

Name of the default collation

character_set_catalog sql_identifier

Character sets are currently not implemented as schema objects, so this column is null

character_set_schema sql_identifier

Character sets are currently not implemented as schema objects, so this column is null

character_set_name sql_identifier

Name of the character set

Prev | Up | Next
37.10. collations | Home | 37.12. column_column_usage


37.12. column_column_usage
Prev Up Chapter 37. The Information Schema Home Next

37.12. column_column_usage

The view column_column_usage identifies all generated columns that depend on another base column in the same table. Only tables owned by a currently enabled role are included.

Table 37.10. column_column_usage Columns

Column Type

Description

table_catalog sql_identifier

Name of the database containing the table (always the current database)

table_schema sql_identifier

Name of the schema containing the table

table_name sql_identifier

Name of the table

column_name sql_identifier

Name of the base column that a generated column depends on

dependent_column sql_identifier

Name of the generated column

Prev | Up | Next
37.11. collation_character_set_​applicability | Home | 37.13. column_domain_usage


37.13. column_domain_usage
Prev Up Chapter 37. The Information Schema Home Next

37.13. column_domain_usage

The view column_domain_usage identifies all columns (of a table or a view) that make use of some domain defined in the current database and owned by a currently enabled role.

Table 37.11. column_domain_usage Columns

Column Type

Description

domain_catalog sql_identifier

Name of the database containing the domain (always the current database)

domain_schema sql_identifier

Name of the schema containing the domain

domain_name sql_identifier

Name of the domain

table_catalog sql_identifier

Name of the database containing the table (always the current database)

table_schema sql_identifier

Name of the schema containing the table

table_name sql_identifier

Name of the table

column_name sql_identifier

Name of the column

Prev | Up | Next
37.12. column_column_usage | Home | 37.14. column_options


37.14. column_options
Prev Up Chapter 37. The Information Schema Home Next

37.14. column_options

The view column_options contains all the options defined for foreign table columns in the current database. Only those foreign table columns are shown that the current user has access to (by way of being the owner or having some privilege).

Table 37.12. column_options Columns

Column Type

Description

table_catalog sql_identifier

Name of the database that contains the foreign table (always the current database)

table_schema sql_identifier

Name of the schema that contains the foreign table

table_name sql_identifier

Name of the foreign table

column_name sql_identifier

Name of the column

option_name sql_identifier

Name of an option

option_value character_data

Value of the option

Prev | Up | Next
37.13. column_domain_usage | Home | 37.15. column_privileges


37.15. column_privileges
Prev Up Chapter 37. The Information Schema Home Next

37.15. column_privileges

The view column_privileges identifies all privileges granted on columns to a currently enabled role or by a currently enabled role. There is one row for each combination of column, grantor, and grantee.

If a privilege has been granted on an entire table, it will show up in this view as a grant for each column, but only for the privilege types where column granularity is possible: SELECT, INSERT, UPDATE, REFERENCES.

Table 37.13. column_privileges Columns

Column Type

Description

grantor sql_identifier

Name of the role that granted the privilege

grantee sql_identifier

Name of the role that the privilege was granted to

table_catalog sql_identifier

Name of the database that contains the table that contains the column (always the current database)

table_schema sql_identifier

Name of the schema that contains the table that contains the column

table_name sql_identifier

Name of the table that contains the column

column_name sql_identifier

Name of the column

privilege_type character_data

Type of the privilege: SELECT, INSERT, UPDATE, or REFERENCES

is_grantable yes_or_no

YES if the privilege is grantable, NO if not

Prev | Up | Next
37.14. column_options | Home | 37.16. column_udt_usage


37.16. column_udt_usage
Prev Up Chapter 37. The Information Schema Home Next

37.16. column_udt_usage

The view column_udt_usage identifies all columns that use data types owned by a currently enabled role. Note that in PostgreSQL, built-in data types behave like user-defined types, so they are included here as well. See also Section 37.17 for details.

Table 37.14. column_udt_usage Columns

Column Type

Description

udt_catalog sql_identifier

Name of the database that the column data type (the underlying type of the domain, if applicable) is defined in (always the current database)

udt_schema sql_identifier

Name of the schema that the column data type (the underlying type of the domain, if applicable) is defined in

udt_name sql_identifier

Name of the column data type (the underlying type of the domain, if applicable)

table_catalog sql_identifier

Name of the database containing the table (always the current database)

table_schema sql_identifier

Name of the schema containing the table

table_name sql_identifier

Name of the table

column_name sql_identifier

Name of the column

Prev | Up | Next
37.15. column_privileges | Home | 37.17. columns


37.17. columns
Prev Up Chapter 37. The Information Schema Home Next

37.17. columns

The view columns contains information about all table columns (or view columns) in the database. System columns (ctid, etc.) are not included. Only those columns are shown that the current user has access to (by way of being the owner or having some privilege).

Table 37.15. columns Columns

Column Type

Description

table_catalog sql_identifier

Name of the database containing the table (always the current database)

table_schema sql_identifier

Name of the schema containing the table

table_name sql_identifier

Name of the table

column_name sql_identifier

Name of the column

ordinal_position cardinal_number

Ordinal position of the column within the table (count starts at 1)

column_default character_data

Default expression of the column

is_nullable yes_or_no

YES if the column is possibly nullable, NO if it is known not nullable. A not-null constraint is one way a column can be known not nullable, but there can be others.

data_type character_data

Data type of the column, if it is a built-in type, or ARRAY if it is some array (in that case, see the view element_types), else USER-DEFINED (in that case, the type is identified in udt_name and associated columns). If the column is based on a domain, this column refers to the type underlying the domain (and the domain is identified in domain_name and associated columns).

character_maximum_length cardinal_number

If data_type identifies a character or bit string type, the declared maximum length; null for all other data types or if no maximum length was declared.

character_octet_length cardinal_number

If data_type identifies a character type, the maximum possible length in octets (bytes) of a datum; null for all other data types. The maximum octet length depends on the declared character maximum length (see above) and the server encoding.

numeric_precision cardinal_number

If data_type identifies a numeric type, this column contains the (declared or implicit) precision of the type for this column. The precision indicates the number of significant digits. It can be expressed in decimal (base 10) or binary (base 2) terms, as specified in the column numeric_precision_radix. For all other data types, this column is null.

numeric_precision_radix cardinal_number

If data_type identifies a numeric type, this column indicates in which base the values in the columns numeric_precision and numeric_scale are expressed. The value is either 2 or 10. For all other data types, this column is null.

numeric_scale cardinal_number

If data_type identifies an exact numeric type, this column contains the (declared or implicit) scale of the type for this column. The scale indicates the number of significant digits to the right of the decimal point. It can be expressed in decimal (base 10) or binary (base 2) terms, as specified in the column numeric_precision_radix. For all other data types, this column is null.

datetime_precision cardinal_number

If data_type identifies a date, time, timestamp, or interval type, this column contains the (declared or implicit) fractional seconds precision of the type for this column, that is, the number of decimal digits maintained following the decimal point in the seconds value. For all other data types, this column is null.

interval_type character_data

If data_type identifies an interval type, this column contains the specification which fields the intervals include for this column, e.g., YEAR TO MONTH, DAY TO SECOND, etc. If no field restrictions were specified (that is, the interval accepts all fields), and for all other data types, this field is null.

interval_precision cardinal_number

Applies to a feature not available in PostgreSQL (see datetime_precision for the fractional seconds precision of interval type columns)

character_set_catalog sql_identifier

Applies to a feature not available in PostgreSQL

character_set_schema sql_identifier

Applies to a feature not available in PostgreSQL

character_set_name sql_identifier

Applies to a feature not available in PostgreSQL

collation_catalog sql_identifier

Name of the database containing the collation of the column (always the current database), null if default or the data type of the column is not collatable

collation_schema sql_identifier

Name of the schema containing the collation of the column, null if default or the data type of the column is not collatable

collation_name sql_identifier

Name of the collation of the column, null if default or the data type of the column is not collatable

domain_catalog sql_identifier

If the column has a domain type, the name of the database that the domain is defined in (always the current database), else null.

domain_schema sql_identifier

If the column has a domain type, the name of the schema that the domain is defined in, else null.

domain_name sql_identifier

If the column has a domain type, the name of the domain, else null.

udt_catalog sql_identifier

Name of the database that the column data type (the underlying type of the domain, if applicable) is defined in (always the current database)

udt_schema sql_identifier

Name of the schema that the column data type (the underlying type of the domain, if applicable) is defined in

udt_name sql_identifier

Name of the column data type (the underlying type of the domain, if applicable)

scope_catalog sql_identifier

Applies to a feature not available in PostgreSQL

scope_schema sql_identifier

Applies to a feature not available in PostgreSQL

scope_name sql_identifier

Applies to a feature not available in PostgreSQL

maximum_cardinality cardinal_number

Always null, because arrays always have unlimited maximum cardinality in PostgreSQL

dtd_identifier sql_identifier

An identifier of the data type descriptor of the column, unique among the data type descriptors pertaining to the table. This is mainly useful for joining with other instances of such identifiers. (The specific format of the identifier is not defined and not guaranteed to remain the same in future versions.)

is_self_referencing yes_or_no

Applies to a feature not available in PostgreSQL

is_identity yes_or_no

If the column is an identity column, then YES, else NO.

identity_generation character_data

If the column is an identity column, then ALWAYS or BY DEFAULT, reflecting the definition of the column.

identity_start character_data

If the column is an identity column, then the start value of the internal sequence, else null.

identity_increment character_data

If the column is an identity column, then the increment of the internal sequence, else null.

identity_maximum character_data

If the column is an identity column, then the maximum value of the internal sequence, else null.

identity_minimum character_data

If the column is an identity column, then the minimum value of the internal sequence, else null.

identity_cycle yes_or_no

If the column is an identity column, then YES if the internal sequence cycles or NO if it does not; otherwise null.

is_generated character_data

If the column is a generated column, then ALWAYS, else NEVER.

generation_expression character_data

If the column is a generated column, then the generation expression, else null.

is_updatable yes_or_no

YES if the column is updatable, NO if not (Columns in base tables are always updatable, columns in views not necessarily)

Since data types can be defined in a variety of ways in SQL, and PostgreSQL contains additional ways to define data types, their representation in the information schema can be somewhat difficult. The column data_type is supposed to identify the underlying built-in type of the column. In PostgreSQL, this means that the type is defined in the system catalog schema pg_catalog. This column might be useful if the application can handle the well-known built-in types specially (for example, format the numeric types differently or use the data in the precision columns). The columns udt_name, udt_schema, and udt_catalog always identify the underlying data type of the column, even if the column is based on a domain. (Since PostgreSQL treats built-in types like user-defined types, built-in types appear here as well. This is an extension of the SQL standard.) These columns should be used if an application wants to process data differently according to the type, because in that case it wouldn't matter if the column is really based on a domain. If the column is based on a domain, the identity of the domain is stored in the columns domain_name, domain_schema, and domain_catalog. If you want to pair up columns with their associated data types and treat domains as separate types, you could write coalesce(domain_name, udt_name), etc.

Prev | Up | Next
37.16. column_udt_usage | Home | 37.18. constraint_column_usage


37.18. constraint_column_usage
Prev Up Chapter 37. The Information Schema Home Next

37.18. constraint_column_usage

The view constraint_column_usage identifies all columns in the current database that are used by some constraint. Only those columns are shown that are contained in a table owned by a currently enabled role. For a check constraint, this view identifies the columns that are used in the check expression. For a foreign key constraint, this view identifies the columns that the foreign key references. For a unique or primary key constraint, this view identifies the constrained columns.

Table 37.16. constraint_column_usage Columns

Column Type

Description

table_catalog sql_identifier

Name of the database that contains the table that contains the column that is used by some constraint (always the current database)

table_schema sql_identifier

Name of the schema that contains the table that contains the column that is used by some constraint

table_name sql_identifier

Name of the table that contains the column that is used by some constraint

column_name sql_identifier

Name of the column that is used by some constraint

constraint_catalog sql_identifier

Name of the database that contains the constraint (always the current database)

constraint_schema sql_identifier

Name of the schema that contains the constraint

constraint_name sql_identifier

Name of the constraint

Prev | Up | Next
37.17. columns | Home | 37.19. constraint_table_usage


37.19. constraint_table_usage
Prev Up Chapter 37. The Information Schema Home Next

37.19. constraint_table_usage

The view constraint_table_usage identifies all tables in the current database that are used by some constraint and are owned by a currently enabled role. (This is different from the view table_constraints, which identifies all table constraints along with the table they are defined on.) For a foreign key constraint, this view identifies the table that the foreign key references. For a unique or primary key constraint, this view simply identifies the table the constraint belongs to. Check constraints and not-null constraints are not included in this view.

Table 37.17. constraint_table_usage Columns

Column Type

Description

table_catalog sql_identifier

Name of the database that contains the table that is used by some constraint (always the current database)

table_schema sql_identifier

Name of the schema that contains the table that is used by some constraint

table_name sql_identifier

Name of the table that is used by some constraint

constraint_catalog sql_identifier

Name of the database that contains the constraint (always the current database)

constraint_schema sql_identifier

Name of the schema that contains the constraint

constraint_name sql_identifier

Name of the constraint

Prev | Up | Next
37.18. constraint_column_usage | Home | 37.20. data_type_privileges


37.20. data_type_privileges
Prev Up Chapter 37. The Information Schema Home Next

37.20. data_type_privileges

The view data_type_privileges identifies all data type descriptors that the current user has access to, by way of being the owner of the described object or having some privilege for it. A data type descriptor is generated whenever a data type is used in the definition of a table column, a domain, or a function (as parameter or return type) and stores some information about how the data type is used in that instance (for example, the declared maximum length, if applicable). Each data type descriptor is assigned an arbitrary identifier that is unique among the data type descriptor identifiers assigned for one object (table, domain, function). This view is probably not useful for applications, but it is used to define some other views in the information schema.

Table 37.18. data_type_privileges Columns

Column Type

Description

object_catalog sql_identifier

Name of the database that contains the described object (always the current database)

object_schema sql_identifier

Name of the schema that contains the described object

object_name sql_identifier

Name of the described object

object_type character_data

The type of the described object: one of TABLE (the data type descriptor pertains to a column of that table), DOMAIN (the data type descriptors pertains to that domain), ROUTINE (the data type descriptor pertains to a parameter or the return data type of that function).

dtd_identifier sql_identifier

The identifier of the data type descriptor, which is unique among the data type descriptors for that same object.

Prev | Up | Next
37.19. constraint_table_usage | Home | 37.21. domain_constraints


37.21. domain_constraints
Prev Up Chapter 37. The Information Schema Home Next

37.21. domain_constraints

The view domain_constraints contains all constraints belonging to domains defined in the current database. Only those domains are shown that the current user has access to (by way of being the owner or having some privilege).

Table 37.19. domain_constraints Columns

Column Type

Description

constraint_catalog sql_identifier

Name of the database that contains the constraint (always the current database)

constraint_schema sql_identifier

Name of the schema that contains the constraint

constraint_name sql_identifier

Name of the constraint

domain_catalog sql_identifier

Name of the database that contains the domain (always the current database)

domain_schema sql_identifier

Name of the schema that contains the domain

domain_name sql_identifier

Name of the domain

is_deferrable yes_or_no

YES if the constraint is deferrable, NO if not

initially_deferred yes_or_no

YES if the constraint is deferrable and initially deferred, NO if not

Prev | Up | Next
37.20. data_type_privileges | Home | 37.22. domain_udt_usage


37.22. domain_udt_usage
Prev Up Chapter 37. The Information Schema Home Next

37.22. domain_udt_usage

The view domain_udt_usage identifies all domains that are based on data types owned by a currently enabled role. Note that in PostgreSQL, built-in data types behave like user-defined types, so they are included here as well.

Table 37.20. domain_udt_usage Columns

Column Type

Description

udt_catalog sql_identifier

Name of the database that the domain data type is defined in (always the current database)

udt_schema sql_identifier

Name of the schema that the domain data type is defined in

udt_name sql_identifier

Name of the domain data type

domain_catalog sql_identifier

Name of the database that contains the domain (always the current database)

domain_schema sql_identifier

Name of the schema that contains the domain

domain_name sql_identifier

Name of the domain

Prev | Up | Next
37.21. domain_constraints | Home | 37.23. domains


37.23. domains
Prev Up Chapter 37. The Information Schema Home Next

37.23. domains

The view domains contains all domains defined in the current database. Only those domains are shown that the current user has access to (by way of being the owner or having some privilege).

Table 37.21. domains Columns

Column Type

Description

domain_catalog sql_identifier

Name of the database that contains the domain (always the current database)

domain_schema sql_identifier

Name of the schema that contains the domain

domain_name sql_identifier

Name of the domain

data_type character_data

Data type of the domain, if it is a built-in type, or ARRAY if it is some array (in that case, see the view element_types), else USER-DEFINED (in that case, the type is identified in udt_name and associated columns).

character_maximum_length cardinal_number

If the domain has a character or bit string type, the declared maximum length; null for all other data types or if no maximum length was declared.

character_octet_length cardinal_number

If the domain has a character type, the maximum possible length in octets (bytes) of a datum; null for all other data types. The maximum octet length depends on the declared character maximum length (see above) and the server encoding.

character_set_catalog sql_identifier

Applies to a feature not available in PostgreSQL

character_set_schema sql_identifier

Applies to a feature not available in PostgreSQL

character_set_name sql_identifier

Applies to a feature not available in PostgreSQL

collation_catalog sql_identifier

Name of the database containing the collation of the domain (always the current database), null if default or the data type of the domain is not collatable

collation_schema sql_identifier

Name of the schema containing the collation of the domain, null if default or the data type of the domain is not collatable

collation_name sql_identifier

Name of the collation of the domain, null if default or the data type of the domain is not collatable

numeric_precision cardinal_number

If the domain has a numeric type, this column contains the (declared or implicit) precision of the type for this domain. The precision indicates the number of significant digits. It can be expressed in decimal (base 10) or binary (base 2) terms, as specified in the column numeric_precision_radix. For all other data types, this column is null.

numeric_precision_radix cardinal_number

If the domain has a numeric type, this column indicates in which base the values in the columns numeric_precision and numeric_scale are expressed. The value is either 2 or 10. For all other data types, this column is null.

numeric_scale cardinal_number

If the domain has an exact numeric type, this column contains the (declared or implicit) scale of the type for this domain. The scale indicates the number of significant digits to the right of the decimal point. It can be expressed in decimal (base 10) or binary (base 2) terms, as specified in the column numeric_precision_radix. For all other data types, this column is null.

datetime_precision cardinal_number

If data_type identifies a date, time, timestamp, or interval type, this column contains the (declared or implicit) fractional seconds precision of the type for this domain, that is, the number of decimal digits maintained following the decimal point in the seconds value. For all other data types, this column is null.

interval_type character_data

If data_type identifies an interval type, this column contains the specification which fields the intervals include for this domain, e.g., YEAR TO MONTH, DAY TO SECOND, etc. If no field restrictions were specified (that is, the interval accepts all fields), and for all other data types, this field is null.

interval_precision cardinal_number

Applies to a feature not available in PostgreSQL (see datetime_precision for the fractional seconds precision of interval type domains)

domain_default character_data

Default expression of the domain

udt_catalog sql_identifier

Name of the database that the domain data type is defined in (always the current database)

udt_schema sql_identifier

Name of the schema that the domain data type is defined in

udt_name sql_identifier

Name of the domain data type

scope_catalog sql_identifier

Applies to a feature not available in PostgreSQL

scope_schema sql_identifier

Applies to a feature not available in PostgreSQL

scope_name sql_identifier

Applies to a feature not available in PostgreSQL

maximum_cardinality cardinal_number

Always null, because arrays always have unlimited maximum cardinality in PostgreSQL

dtd_identifier sql_identifier

An identifier of the data type descriptor of the domain, unique among the data type descriptors pertaining to the domain (which is trivial, because a domain only contains one data type descriptor). This is mainly useful for joining with other instances of such identifiers. (The specific format of the identifier is not defined and not guaranteed to remain the same in future versions.)

Prev | Up | Next
37.22. domain_udt_usage | Home | 37.24. element_types


37.24. element_types
Prev Up Chapter 37. The Information Schema Home Next

37.24. element_types

The view element_types contains the data type descriptors of the elements of arrays. When a table column, composite-type attribute, domain, function parameter, or function return value is defined to be of an array type, the respective information schema view only contains ARRAY in the column data_type. To obtain information on the element type of the array, you can join the respective view with this view. For example, to show the columns of a table with data types and array element types, if applicable, you could do:

SELECT c.column_name, c.data_type, e.data_type AS element_type
FROM information_schema.columns c LEFT JOIN information_schema.element_types e
     ON ((c.table_catalog, c.table_schema, c.table_name, 'TABLE', c.dtd_identifier)
       = (e.object_catalog, e.object_schema, e.object_name, e.object_type, e.collection_type_identifier))
WHERE c.table_schema = '...' AND c.table_name = '...'
ORDER BY c.ordinal_position;

This view only includes objects that the current user has access to, by way of being the owner or having some privilege.

Table 37.22. element_types Columns

Column Type

Description

object_catalog sql_identifier

Name of the database that contains the object that uses the array being described (always the current database)

object_schema sql_identifier

Name of the schema that contains the object that uses the array being described

object_name sql_identifier

Name of the object that uses the array being described

object_type character_data

The type of the object that uses the array being described: one of TABLE (the array is used by a column of that table), USER-DEFINED TYPE (the array is used by an attribute of that composite type), DOMAIN (the array is used by that domain), ROUTINE (the array is used by a parameter or the return data type of that function).

collection_type_identifier sql_identifier

The identifier of the data type descriptor of the array being described. Use this to join with the dtd_identifier columns of other information schema views.

data_type character_data

Data type of the array elements, if it is a built-in type, else USER-DEFINED (in that case, the type is identified in udt_name and associated columns).

character_maximum_length cardinal_number

Always null, since this information is not applied to array element data types in PostgreSQL

character_octet_length cardinal_number

Always null, since this information is not applied to array element data types in PostgreSQL

character_set_catalog sql_identifier

Applies to a feature not available in PostgreSQL

character_set_schema sql_identifier

Applies to a feature not available in PostgreSQL

character_set_name sql_identifier

Applies to a feature not available in PostgreSQL

collation_catalog sql_identifier

Name of the database containing the collation of the element type (always the current database), null if default or the data type of the element is not collatable

collation_schema sql_identifier

Name of the schema containing the collation of the element type, null if default or the data type of the element is not collatable

collation_name sql_identifier

Name of the collation of the element type, null if default or the data type of the element is not collatable

numeric_precision cardinal_number

Always null, since this information is not applied to array element data types in PostgreSQL

numeric_precision_radix cardinal_number

Always null, since this information is not applied to array element data types in PostgreSQL

numeric_scale cardinal_number

Always null, since this information is not applied to array element data types in PostgreSQL

datetime_precision cardinal_number

Always null, since this information is not applied to array element data types in PostgreSQL

interval_type character_data

Always null, since this information is not applied to array element data types in PostgreSQL

interval_precision cardinal_number

Always null, since this information is not applied to array element data types in PostgreSQL

domain_default character_data

Not yet implemented

udt_catalog sql_identifier

Name of the database that the data type of the elements is defined in (always the current database)

udt_schema sql_identifier

Name of the schema that the data type of the elements is defined in

udt_name sql_identifier

Name of the data type of the elements

scope_catalog sql_identifier

Applies to a feature not available in PostgreSQL

scope_schema sql_identifier

Applies to a feature not available in PostgreSQL

scope_name sql_identifier

Applies to a feature not available in PostgreSQL

maximum_cardinality cardinal_number

Always null, because arrays always have unlimited maximum cardinality in PostgreSQL

dtd_identifier sql_identifier

An identifier of the data type descriptor of the element. This is currently not useful.

Prev | Up | Next
37.23. domains | Home | 37.25. enabled_roles


37.25. enabled_roles
Prev Up Chapter 37. The Information Schema Home Next

37.25. enabled_roles

The view enabled_roles identifies the currently “enabled roles”. The enabled roles are recursively defined as the current user together with all roles that have been granted to the enabled roles with automatic inheritance. In other words, these are all roles that the current user has direct or indirect, automatically inheriting membership in.

For permission checking, the set of “applicable roles” is applied, which can be broader than the set of enabled roles. So generally, it is better to use the view applicable_roles instead of this one; See Section 37.5 for details on applicable_roles view.

Table 37.23. enabled_roles Columns

Column Type

Description

role_name sql_identifier

Name of a role

Prev | Up | Next
37.24. element_types | Home | 37.26. foreign_data_wrapper_options


37.26. foreign_data_wrapper_options
Prev Up Chapter 37. The Information Schema Home Next

37.26. foreign_data_wrapper_options

The view foreign_data_wrapper_options contains all the options defined for foreign-data wrappers in the current database. Only those foreign-data wrappers are shown that the current user has access to (by way of being the owner or having some privilege).

Table 37.24. foreign_data_wrapper_options Columns

Column Type

Description

foreign_data_wrapper_catalog sql_identifier

Name of the database that the foreign-data wrapper is defined in (always the current database)

foreign_data_wrapper_name sql_identifier

Name of the foreign-data wrapper

option_name sql_identifier

Name of an option

option_value character_data

Value of the option

Prev | Up | Next
37.25. enabled_roles | Home | 37.27. foreign_data_wrappers


37.27. foreign_data_wrappers
Prev Up Chapter 37. The Information Schema Home Next

37.27. foreign_data_wrappers

The view foreign_data_wrappers contains all foreign-data wrappers defined in the current database. Only those foreign-data wrappers are shown that the current user has access to (by way of being the owner or having some privilege).

Table 37.25. foreign_data_wrappers Columns

Column Type

Description

foreign_data_wrapper_catalog sql_identifier

Name of the database that contains the foreign-data wrapper (always the current database)

foreign_data_wrapper_name sql_identifier

Name of the foreign-data wrapper

authorization_identifier sql_identifier

Name of the owner of the foreign server

library_name character_data

File name of the library that implementing this foreign-data wrapper

foreign_data_wrapper_language character_data

Language used to implement this foreign-data wrapper

Prev | Up | Next
37.26. foreign_data_wrapper_options | Home | 37.28. foreign_server_options


37.28. foreign_server_options
Prev Up Chapter 37. The Information Schema Home Next

37.28. foreign_server_options

The view foreign_server_options contains all the options defined for foreign servers in the current database. Only those foreign servers are shown that the current user has access to (by way of being the owner or having some privilege).

Table 37.26. foreign_server_options Columns

Column Type

Description

foreign_server_catalog sql_identifier

Name of the database that the foreign server is defined in (always the current database)

foreign_server_name sql_identifier

Name of the foreign server

option_name sql_identifier

Name of an option

option_value character_data

Value of the option

Prev | Up | Next
37.27. foreign_data_wrappers | Home | 37.29. foreign_servers


37.29. foreign_servers
Prev Up Chapter 37. The Information Schema Home Next

37.29. foreign_servers

The view foreign_servers contains all foreign servers defined in the current database. Only those foreign servers are shown that the current user has access to (by way of being the owner or having some privilege).

Table 37.27. foreign_servers Columns

Column Type

Description

foreign_server_catalog sql_identifier

Name of the database that the foreign server is defined in (always the current database)

foreign_server_name sql_identifier

Name of the foreign server

foreign_data_wrapper_catalog sql_identifier

Name of the database that contains the foreign-data wrapper used by the foreign server (always the current database)

foreign_data_wrapper_name sql_identifier

Name of the foreign-data wrapper used by the foreign server

foreign_server_type character_data

Foreign server type information, if specified upon creation

foreign_server_version character_data

Foreign server version information, if specified upon creation

authorization_identifier sql_identifier

Name of the owner of the foreign server

Prev | Up | Next
37.28. foreign_server_options | Home | 37.30. foreign_table_options


37.30. foreign_table_options
Prev Up Chapter 37. The Information Schema Home Next

37.30. foreign_table_options

The view foreign_table_options contains all the options defined for foreign tables in the current database. Only those foreign tables are shown that the current user has access to (by way of being the owner or having some privilege).

Table 37.28. foreign_table_options Columns

Column Type

Description

foreign_table_catalog sql_identifier

Name of the database that contains the foreign table (always the current database)

foreign_table_schema sql_identifier

Name of the schema that contains the foreign table

foreign_table_name sql_identifier

Name of the foreign table

option_name sql_identifier

Name of an option

option_value character_data

Value of the option

Prev | Up | Next
37.29. foreign_servers | Home | 37.31. foreign_tables


37.31. foreign_tables
Prev Up Chapter 37. The Information Schema Home Next

37.31. foreign_tables

The view foreign_tables contains all foreign tables defined in the current database. Only those foreign tables are shown that the current user has access to (by way of being the owner or having some privilege).

Table 37.29. foreign_tables Columns

Column Type

Description

foreign_table_catalog sql_identifier

Name of the database that the foreign table is defined in (always the current database)

foreign_table_schema sql_identifier

Name of the schema that contains the foreign table

foreign_table_name sql_identifier

Name of the foreign table

foreign_server_catalog sql_identifier

Name of the database that the foreign server is defined in (always the current database)

foreign_server_name sql_identifier

Name of the foreign server

Prev | Up | Next
37.30. foreign_table_options | Home | 37.32. key_column_usage


37.32. key_column_usage
Prev Up Chapter 37. The Information Schema Home Next

37.32. key_column_usage

The view key_column_usage identifies all columns in the current database that are restricted by some unique, primary key, or foreign key constraint. Check constraints are not included in this view. Only those columns are shown that the current user has access to, by way of being the owner or having some privilege.

Table 37.30. key_column_usage Columns

Column Type

Description

constraint_catalog sql_identifier

Name of the database that contains the constraint (always the current database)

constraint_schema sql_identifier

Name of the schema that contains the constraint

constraint_name sql_identifier

Name of the constraint

table_catalog sql_identifier

Name of the database that contains the table that contains the column that is restricted by this constraint (always the current database)

table_schema sql_identifier

Name of the schema that contains the table that contains the column that is restricted by this constraint

table_name sql_identifier

Name of the table that contains the column that is restricted by this constraint

column_name sql_identifier

Name of the column that is restricted by this constraint

ordinal_position cardinal_number

Ordinal position of the column within the constraint key (count starts at 1)

position_in_unique_constraint cardinal_number

For a foreign-key constraint, ordinal position of the referenced column within its unique constraint (count starts at 1); otherwise null

Prev | Up | Next
37.31. foreign_tables | Home | 37.33. parameters


37.33. parameters
Prev Up Chapter 37. The Information Schema Home Next

37.33. parameters

The view parameters contains information about the parameters (arguments) of all functions in the current database. Only those functions are shown that the current user has access to (by way of being the owner or having some privilege).

Table 37.31. parameters Columns

Column Type

Description

specific_catalog sql_identifier

Name of the database containing the function (always the current database)

specific_schema sql_identifier

Name of the schema containing the function

specific_name sql_identifier

The “specific name” of the function. See Section 37.45 for more information.

ordinal_position cardinal_number

Ordinal position of the parameter in the argument list of the function (count starts at 1)

parameter_mode character_data

IN for input parameter, OUT for output parameter, and INOUT for input/output parameter.

is_result yes_or_no

Applies to a feature not available in PostgreSQL

as_locator yes_or_no

Applies to a feature not available in PostgreSQL

parameter_name sql_identifier

Name of the parameter, or null if the parameter has no name

data_type character_data

Data type of the parameter, if it is a built-in type, or ARRAY if it is some array (in that case, see the view element_types), else USER-DEFINED (in that case, the type is identified in udt_name and associated columns).

character_maximum_length cardinal_number

Always null, since this information is not applied to parameter data types in PostgreSQL

character_octet_length cardinal_number

Always null, since this information is not applied to parameter data types in PostgreSQL

character_set_catalog sql_identifier

Applies to a feature not available in PostgreSQL

character_set_schema sql_identifier

Applies to a feature not available in PostgreSQL

character_set_name sql_identifier

Applies to a feature not available in PostgreSQL

collation_catalog sql_identifier

Always null, since this information is not applied to parameter data types in PostgreSQL

collation_schema sql_identifier

Always null, since this information is not applied to parameter data types in PostgreSQL

collation_name sql_identifier

Always null, since this information is not applied to parameter data types in PostgreSQL

numeric_precision cardinal_number

Always null, since this information is not applied to parameter data types in PostgreSQL

numeric_precision_radix cardinal_number

Always null, since this information is not applied to parameter data types in PostgreSQL

numeric_scale cardinal_number

Always null, since this information is not applied to parameter data types in PostgreSQL

datetime_precision cardinal_number

Always null, since this information is not applied to parameter data types in PostgreSQL

interval_type character_data

Always null, since this information is not applied to parameter data types in PostgreSQL

interval_precision cardinal_number

Always null, since this information is not applied to parameter data types in PostgreSQL

udt_catalog sql_identifier

Name of the database that the data type of the parameter is defined in (always the current database)

udt_schema sql_identifier

Name of the schema that the data type of the parameter is defined in

udt_name sql_identifier

Name of the data type of the parameter

scope_catalog sql_identifier

Applies to a feature not available in PostgreSQL

scope_schema sql_identifier

Applies to a feature not available in PostgreSQL

scope_name sql_identifier

Applies to a feature not available in PostgreSQL

maximum_cardinality cardinal_number

Always null, because arrays always have unlimited maximum cardinality in PostgreSQL

dtd_identifier sql_identifier

An identifier of the data type descriptor of the parameter, unique among the data type descriptors pertaining to the function. This is mainly useful for joining with other instances of such identifiers. (The specific format of the identifier is not defined and not guaranteed to remain the same in future versions.)

parameter_default character_data

The default expression of the parameter, or null if none or if the function is not owned by a currently enabled role.

Prev | Up | Next
37.32. key_column_usage | Home | 37.34. referential_constraints


37.34. referential_constraints
Prev Up Chapter 37. The Information Schema Home Next

37.34. referential_constraints

The view referential_constraints contains all referential (foreign key) constraints in the current database. Only those constraints are shown for which the current user has write access to the referencing table (by way of being the owner or having some privilege other than SELECT).

Table 37.32. referential_constraints Columns

Column Type

Description

constraint_catalog sql_identifier

Name of the database containing the constraint (always the current database)

constraint_schema sql_identifier

Name of the schema containing the constraint

constraint_name sql_identifier

Name of the constraint

unique_constraint_catalog sql_identifier

Name of the database that contains the unique or primary key constraint that the foreign key constraint references (always the current database)

unique_constraint_schema sql_identifier

Name of the schema that contains the unique or primary key constraint that the foreign key constraint references

unique_constraint_name sql_identifier

Name of the unique or primary key constraint that the foreign key constraint references

match_option character_data

Match option of the foreign key constraint: FULL, PARTIAL, or NONE.

update_rule character_data

Update rule of the foreign key constraint: CASCADE, SET NULL, SET DEFAULT, RESTRICT, or NO ACTION.

delete_rule character_data

Delete rule of the foreign key constraint: CASCADE, SET NULL, SET DEFAULT, RESTRICT, or NO ACTION.

Prev | Up | Next
37.33. parameters | Home | 37.35. role_column_grants


37.35. role_column_grants
Prev Up Chapter 37. The Information Schema Home Next

37.35. role_column_grants

The view role_column_grants identifies all privileges granted on columns where the grantor or grantee is a currently enabled role. Further information can be found under column_privileges. The only effective difference between this view and column_privileges is that this view omits columns that have been made accessible to the current user by way of a grant to PUBLIC.

Table 37.33. role_column_grants Columns

Column Type

Description

grantor sql_identifier

Name of the role that granted the privilege

grantee sql_identifier

Name of the role that the privilege was granted to

table_catalog sql_identifier

Name of the database that contains the table that contains the column (always the current database)

table_schema sql_identifier

Name of the schema that contains the table that contains the column

table_name sql_identifier

Name of the table that contains the column

column_name sql_identifier

Name of the column

privilege_type character_data

Type of the privilege: SELECT, INSERT, UPDATE, or REFERENCES

is_grantable yes_or_no

YES if the privilege is grantable, NO if not

Prev | Up | Next
37.34. referential_constraints | Home | 37.36. role_routine_grants


37.36. role_routine_grants
Prev Up Chapter 37. The Information Schema Home Next

37.36. role_routine_grants

The view role_routine_grants identifies all privileges granted on functions where the grantor or grantee is a currently enabled role. Further information can be found under routine_privileges. The only effective difference between this view and routine_privileges is that this view omits functions that have been made accessible to the current user by way of a grant to PUBLIC.

Table 37.34. role_routine_grants Columns

Column Type

Description

grantor sql_identifier

Name of the role that granted the privilege

grantee sql_identifier

Name of the role that the privilege was granted to

specific_catalog sql_identifier

Name of the database containing the function (always the current database)

specific_schema sql_identifier

Name of the schema containing the function

specific_name sql_identifier

The “specific name” of the function. See Section 37.45 for more information.

routine_catalog sql_identifier

Name of the database containing the function (always the current database)

routine_schema sql_identifier

Name of the schema containing the function

routine_name sql_identifier

Name of the function (might be duplicated in case of overloading)

privilege_type character_data

Always EXECUTE (the only privilege type for functions)

is_grantable yes_or_no

YES if the privilege is grantable, NO if not

Prev | Up | Next
37.35. role_column_grants | Home | 37.37. role_table_grants


37.37. role_table_grants
Prev Up Chapter 37. The Information Schema Home Next

37.37. role_table_grants

The view role_table_grants identifies all privileges granted on tables or views where the grantor or grantee is a currently enabled role. Further information can be found under table_privileges. The only effective difference between this view and table_privileges is that this view omits tables that have been made accessible to the current user by way of a grant to PUBLIC.

Table 37.35. role_table_grants Columns

Column Type

Description

grantor sql_identifier

Name of the role that granted the privilege

grantee sql_identifier

Name of the role that the privilege was granted to

table_catalog sql_identifier

Name of the database that contains the table (always the current database)

table_schema sql_identifier

Name of the schema that contains the table

table_name sql_identifier

Name of the table

privilege_type character_data

Type of the privilege: SELECT, INSERT, UPDATE, DELETE, TRUNCATE, REFERENCES, or TRIGGER

is_grantable yes_or_no

YES if the privilege is grantable, NO if not

with_hierarchy yes_or_no

In the SQL standard, WITH HIERARCHY OPTION is a separate (sub-)privilege allowing certain operations on table inheritance hierarchies. In PostgreSQL, this is included in the SELECT privilege, so this column shows YES if the privilege is SELECT, else NO.

Prev | Up | Next
37.36. role_routine_grants | Home | 37.38. role_udt_grants


37.38. role_udt_grants
Prev Up Chapter 37. The Information Schema Home Next

37.38. role_udt_grants

The view role_udt_grants is intended to identify USAGE privileges granted on user-defined types where the grantor or grantee is a currently enabled role. Further information can be found under udt_privileges. The only effective difference between this view and udt_privileges is that this view omits objects that have been made accessible to the current user by way of a grant to PUBLIC. Since data types do not have real privileges in PostgreSQL, but only an implicit grant to PUBLIC, this view is empty.

Table 37.36. role_udt_grants Columns

Column Type

Description

grantor sql_identifier

The name of the role that granted the privilege

grantee sql_identifier

The name of the role that the privilege was granted to

udt_catalog sql_identifier

Name of the database containing the type (always the current database)

udt_schema sql_identifier

Name of the schema containing the type

udt_name sql_identifier

Name of the type

privilege_type character_data

Always TYPE USAGE

is_grantable yes_or_no

YES if the privilege is grantable, NO if not

Prev | Up | Next
37.37. role_table_grants | Home | 37.39. role_usage_grants


37.39. role_usage_grants
Prev Up Chapter 37. The Information Schema Home Next

37.39. role_usage_grants

The view role_usage_grants identifies USAGE privileges granted on various kinds of objects where the grantor or grantee is a currently enabled role. Further information can be found under usage_privileges. The only effective difference between this view and usage_privileges is that this view omits objects that have been made accessible to the current user by way of a grant to PUBLIC.

Table 37.37. role_usage_grants Columns

Column Type

Description

grantor sql_identifier

The name of the role that granted the privilege

grantee sql_identifier

The name of the role that the privilege was granted to

object_catalog sql_identifier

Name of the database containing the object (always the current database)

object_schema sql_identifier

Name of the schema containing the object, if applicable, else an empty string

object_name sql_identifier

Name of the object

object_type character_data

COLLATION or DOMAIN or FOREIGN DATA WRAPPER or FOREIGN SERVER or SEQUENCE

privilege_type character_data

Always USAGE

is_grantable yes_or_no

YES if the privilege is grantable, NO if not

Prev | Up | Next
37.38. role_udt_grants | Home | 37.40. routine_column_usage


37.40. routine_column_usage
Prev Up Chapter 37. The Information Schema Home Next

37.40. routine_column_usage

The view routine_column_usage is meant to identify all columns that are used by a function or procedure. This information is currently not tracked by PostgreSQL.

Table 37.38. routine_column_usage Columns

Column Type

Description

specific_catalog sql_identifier

Name of the database containing the function (always the current database)

specific_schema sql_identifier

Name of the schema containing the function

specific_name sql_identifier

The “specific name” of the function. See Section 37.45 for more information.

routine_catalog sql_identifier

Name of the database containing the function (always the current database)

routine_schema sql_identifier

Name of the schema containing the function

routine_name sql_identifier

Name of the function (might be duplicated in case of overloading)

table_catalog sql_identifier

Name of the database that contains the table that is used by the function (always the current database)

table_schema sql_identifier

Name of the schema that contains the table that is used by the function

table_name sql_identifier

Name of the table that is used by the function

column_name sql_identifier

Name of the column that is used by the function

Prev | Up | Next
37.39. role_usage_grants | Home | 37.41. routine_privileges


37.41. routine_privileges
Prev Up Chapter 37. The Information Schema Home Next

37.41. routine_privileges

The view routine_privileges identifies all privileges granted on functions to a currently enabled role or by a currently enabled role. There is one row for each combination of function, grantor, and grantee.

Table 37.39. routine_privileges Columns

Column Type

Description

grantor sql_identifier

Name of the role that granted the privilege

grantee sql_identifier

Name of the role that the privilege was granted to

specific_catalog sql_identifier

Name of the database containing the function (always the current database)

specific_schema sql_identifier

Name of the schema containing the function

specific_name sql_identifier

The “specific name” of the function. See Section 37.45 for more information.

routine_catalog sql_identifier

Name of the database containing the function (always the current database)

routine_schema sql_identifier

Name of the schema containing the function

routine_name sql_identifier

Name of the function (might be duplicated in case of overloading)

privilege_type character_data

Always EXECUTE (the only privilege type for functions)

is_grantable yes_or_no

YES if the privilege is grantable, NO if not

Prev | Up | Next
37.40. routine_column_usage | Home | 37.42. routine_routine_usage


37.42. routine_routine_usage
Prev Up Chapter 37. The Information Schema Home Next

37.42. routine_routine_usage

The view routine_routine_usage is meant to identify all functions or procedures that are used by another (or the same) function or procedure, either in the body or in parameter default expressions. Currently, only functions used in parameter default expressions are tracked. An entry is included here only if the used function is owned by a currently enabled role. (There is no such restriction on the using function.)

Note that the entries for both functions in the view refer to the “specific” name of the routine, even though the column names are used in a way that is inconsistent with other information schema views about routines. This is per SQL standard, although it is arguably a misdesign. See Section 37.45 for more information about specific names.

Table 37.40. routine_routine_usage Columns

Column Type

Description

specific_catalog sql_identifier

Name of the database containing the using function (always the current database)

specific_schema sql_identifier

Name of the schema containing the using function

specific_name sql_identifier

The “specific name” of the using function.

routine_catalog sql_identifier

Name of the database that contains the function that is used by the first function (always the current database)

routine_schema sql_identifier

Name of the schema that contains the function that is used by the first function

routine_name sql_identifier

The “specific name” of the function that is used by the first function.

Prev | Up | Next
37.41. routine_privileges | Home | 37.43. routine_sequence_usage


37.43. routine_sequence_usage
Prev Up Chapter 37. The Information Schema Home Next

37.43. routine_sequence_usage

The view routine_sequence_usage is meant to identify all sequences that are used by a function or procedure, either in the body or in parameter default expressions. Currently, only sequences used in parameter default expressions are tracked. A sequence is only included if that sequence is owned by a currently enabled role.

Table 37.41. routine_sequence_usage Columns

Column Type

Description

specific_catalog sql_identifier

Name of the database containing the function (always the current database)

specific_schema sql_identifier

Name of the schema containing the function

specific_name sql_identifier

The “specific name” of the function. See Section 37.45 for more information.

routine_catalog sql_identifier

Name of the database containing the function (always the current database)

routine_schema sql_identifier

Name of the schema containing the function

routine_name sql_identifier

Name of the function (might be duplicated in case of overloading)

schema_catalog sql_identifier

Name of the database that contains the sequence that is used by the function (always the current database)

sequence_schema sql_identifier

Name of the schema that contains the sequence that is used by the function

sequence_name sql_identifier

Name of the sequence that is used by the function

Prev | Up | Next
37.42. routine_routine_usage | Home | 37.44. routine_table_usage


37.44. routine_table_usage
Prev Up Chapter 37. The Information Schema Home Next

37.44. routine_table_usage

The view routine_table_usage is meant to identify all tables that are used by a function or procedure. This information is currently not tracked by PostgreSQL.

Table 37.42. routine_table_usage Columns

Column Type

Description

specific_catalog sql_identifier

Name of the database containing the function (always the current database)

specific_schema sql_identifier

Name of the schema containing the function

specific_name sql_identifier

The “specific name” of the function. See Section 37.45 for more information.

routine_catalog sql_identifier

Name of the database containing the function (always the current database)

routine_schema sql_identifier

Name of the schema containing the function

routine_name sql_identifier

Name of the function (might be duplicated in case of overloading)

table_catalog sql_identifier

Name of the database that contains the table that is used by the function (always the current database)

table_schema sql_identifier

Name of the schema that contains the table that is used by the function

table_name sql_identifier

Name of the table that is used by the function

Prev | Up | Next
37.43. routine_sequence_usage | Home | 37.45. routines


37.45. routines
Prev Up Chapter 37. The Information Schema Home Next

37.45. routines

The view routines contains all functions and procedures in the current database. Only those functions and procedures are shown that the current user has access to (by way of being the owner or having some privilege).

Table 37.43. routines Columns

Column Type

Description

specific_catalog sql_identifier

Name of the database containing the function (always the current database)

specific_schema sql_identifier

Name of the schema containing the function

specific_name sql_identifier

The “specific name” of the function. This is a name that uniquely identifies the function in the schema, even if the real name of the function is overloaded. The format of the specific name is not defined, it should only be used to compare it to other instances of specific routine names.

routine_catalog sql_identifier

Name of the database containing the function (always the current database)

routine_schema sql_identifier

Name of the schema containing the function

routine_name sql_identifier

Name of the function (might be duplicated in case of overloading)

routine_type character_data

FUNCTION for a function, PROCEDURE for a procedure

module_catalog sql_identifier

Applies to a feature not available in PostgreSQL

module_schema sql_identifier

Applies to a feature not available in PostgreSQL

module_name sql_identifier

Applies to a feature not available in PostgreSQL

udt_catalog sql_identifier

Applies to a feature not available in PostgreSQL

udt_schema sql_identifier

Applies to a feature not available in PostgreSQL

udt_name sql_identifier

Applies to a feature not available in PostgreSQL

data_type character_data

Return data type of the function, if it is a built-in type, or ARRAY if it is some array (in that case, see the view element_types), else USER-DEFINED (in that case, the type is identified in type_udt_name and associated columns). Null for a procedure.

character_maximum_length cardinal_number

Always null, since this information is not applied to return data types in PostgreSQL

character_octet_length cardinal_number

Always null, since this information is not applied to return data types in PostgreSQL

character_set_catalog sql_identifier

Applies to a feature not available in PostgreSQL

character_set_schema sql_identifier

Applies to a feature not available in PostgreSQL

character_set_name sql_identifier

Applies to a feature not available in PostgreSQL

collation_catalog sql_identifier

Always null, since this information is not applied to return data types in PostgreSQL

collation_schema sql_identifier

Always null, since this information is not applied to return data types in PostgreSQL

collation_name sql_identifier

Always null, since this information is not applied to return data types in PostgreSQL

numeric_precision cardinal_number

Always null, since this information is not applied to return data types in PostgreSQL

numeric_precision_radix cardinal_number

Always null, since this information is not applied to return data types in PostgreSQL

numeric_scale cardinal_number

Always null, since this information is not applied to return data types in PostgreSQL

datetime_precision cardinal_number

Always null, since this information is not applied to return data types in PostgreSQL

interval_type character_data

Always null, since this information is not applied to return data types in PostgreSQL

interval_precision cardinal_number

Always null, since this information is not applied to return data types in PostgreSQL

type_udt_catalog sql_identifier

Name of the database that the return data type of the function is defined in (always the current database). Null for a procedure.

type_udt_schema sql_identifier

Name of the schema that the return data type of the function is defined in. Null for a procedure.

type_udt_name sql_identifier

Name of the return data type of the function. Null for a procedure.

scope_catalog sql_identifier

Applies to a feature not available in PostgreSQL

scope_schema sql_identifier

Applies to a feature not available in PostgreSQL

scope_name sql_identifier

Applies to a feature not available in PostgreSQL

maximum_cardinality cardinal_number

Always null, because arrays always have unlimited maximum cardinality in PostgreSQL

dtd_identifier sql_identifier

An identifier of the data type descriptor of the return data type of this function, unique among the data type descriptors pertaining to the function. This is mainly useful for joining with other instances of such identifiers. (The specific format of the identifier is not defined and not guaranteed to remain the same in future versions.)

routine_body character_data

If the function is an SQL function, then SQL, else EXTERNAL.

routine_definition character_data

The source text of the function (null if the function is not owned by a currently enabled role). (According to the SQL standard, this column is only applicable if routine_body is SQL, but in PostgreSQL it will contain whatever source text was specified when the function was created.)

external_name character_data

If this function is a C function, then the external name (link symbol) of the function; else null. (This works out to be the same value that is shown in routine_definition.)

external_language character_data

The language the function is written in

parameter_style character_data

Always GENERAL (The SQL standard defines other parameter styles, which are not available in PostgreSQL.)

is_deterministic yes_or_no

If the function is declared immutable (called deterministic in the SQL standard), then YES, else NO. (You cannot query the other volatility levels available in PostgreSQL through the information schema.)

sql_data_access character_data

Always MODIFIES, meaning that the function possibly modifies SQL data. This information is not useful for PostgreSQL.

is_null_call yes_or_no

If the function automatically returns null if any of its arguments are null, then YES, else NO. Null for a procedure.

sql_path character_data

Applies to a feature not available in PostgreSQL

schema_level_routine yes_or_no

Always YES (The opposite would be a method of a user-defined type, which is a feature not available in PostgreSQL.)

max_dynamic_result_sets cardinal_number

Applies to a feature not available in PostgreSQL

is_user_defined_cast yes_or_no

Applies to a feature not available in PostgreSQL

is_implicitly_invocable yes_or_no

Applies to a feature not available in PostgreSQL

security_type character_data

If the function runs with the privileges of the current user, then INVOKER, if the function runs with the privileges of the user who defined it, then DEFINER.

to_sql_specific_catalog sql_identifier

Applies to a feature not available in PostgreSQL

to_sql_specific_schema sql_identifier

Applies to a feature not available in PostgreSQL

to_sql_specific_name sql_identifier

Applies to a feature not available in PostgreSQL

as_locator yes_or_no

Applies to a feature not available in PostgreSQL

created time_stamp

Applies to a feature not available in PostgreSQL

last_altered time_stamp

Applies to a feature not available in PostgreSQL

new_savepoint_level yes_or_no

Applies to a feature not available in PostgreSQL

is_udt_dependent yes_or_no

Currently always NO. The alternative YES applies to a feature not available in PostgreSQL.

result_cast_from_data_type character_data

Applies to a feature not available in PostgreSQL

result_cast_as_locator yes_or_no

Applies to a feature not available in PostgreSQL

result_cast_char_max_length cardinal_number

Applies to a feature not available in PostgreSQL

result_cast_char_octet_length cardinal_number

Applies to a feature not available in PostgreSQL

result_cast_char_set_catalog sql_identifier

Applies to a feature not available in PostgreSQL

result_cast_char_set_schema sql_identifier

Applies to a feature not available in PostgreSQL

result_cast_char_set_name sql_identifier

Applies to a feature not available in PostgreSQL

result_cast_collation_catalog sql_identifier

Applies to a feature not available in PostgreSQL

result_cast_collation_schema sql_identifier

Applies to a feature not available in PostgreSQL

result_cast_collation_name sql_identifier

Applies to a feature not available in PostgreSQL

result_cast_numeric_precision cardinal_number

Applies to a feature not available in PostgreSQL

result_cast_numeric_precision_radix cardinal_number

Applies to a feature not available in PostgreSQL

result_cast_numeric_scale cardinal_number

Applies to a feature not available in PostgreSQL

result_cast_datetime_precision cardinal_number

Applies to a feature not available in PostgreSQL

result_cast_interval_type character_data

Applies to a feature not available in PostgreSQL

result_cast_interval_precision cardinal_number

Applies to a feature not available in PostgreSQL

result_cast_type_udt_catalog sql_identifier

Applies to a feature not available in PostgreSQL

result_cast_type_udt_schema sql_identifier

Applies to a feature not available in PostgreSQL

result_cast_type_udt_name sql_identifier

Applies to a feature not available in PostgreSQL

result_cast_scope_catalog sql_identifier

Applies to a feature not available in PostgreSQL

result_cast_scope_schema sql_identifier

Applies to a feature not available in PostgreSQL

result_cast_scope_name sql_identifier

Applies to a feature not available in PostgreSQL

result_cast_maximum_cardinality cardinal_number

Applies to a feature not available in PostgreSQL

result_cast_dtd_identifier sql_identifier

Applies to a feature not available in PostgreSQL

Prev | Up | Next
37.44. routine_table_usage | Home | 37.46. schemata


37.46. schemata
Prev Up Chapter 37. The Information Schema Home Next

37.46. schemata

The view schemata contains all schemas in the current database that the current user has access to (by way of being the owner or having some privilege).

Table 37.44. schemata Columns

Column Type

Description

catalog_name sql_identifier

Name of the database that the schema is contained in (always the current database)

schema_name sql_identifier

Name of the schema

schema_owner sql_identifier

Name of the owner of the schema

default_character_set_catalog sql_identifier

Applies to a feature not available in PostgreSQL

default_character_set_schema sql_identifier

Applies to a feature not available in PostgreSQL

default_character_set_name sql_identifier

Applies to a feature not available in PostgreSQL

sql_path character_data

Applies to a feature not available in PostgreSQL

Prev | Up | Next
37.45. routines | Home | 37.47. sequences


37.47. sequences
Prev Up Chapter 37. The Information Schema Home Next

37.47. sequences

The view sequences contains all sequences defined in the current database. Only those sequences are shown that the current user has access to (by way of being the owner or having some privilege).

Table 37.45. sequences Columns

Column Type

Description

sequence_catalog sql_identifier

Name of the database that contains the sequence (always the current database)

sequence_schema sql_identifier

Name of the schema that contains the sequence

sequence_name sql_identifier

Name of the sequence

data_type character_data

The data type of the sequence.

numeric_precision cardinal_number

This column contains the (declared or implicit) precision of the sequence data type (see above). The precision indicates the number of significant digits. It can be expressed in decimal (base 10) or binary (base 2) terms, as specified in the column numeric_precision_radix.

numeric_precision_radix cardinal_number

This column indicates in which base the values in the columns numeric_precision and numeric_scale are expressed. The value is either 2 or 10.

numeric_scale cardinal_number

This column contains the (declared or implicit) scale of the sequence data type (see above). The scale indicates the number of significant digits to the right of the decimal point. It can be expressed in decimal (base 10) or binary (base 2) terms, as specified in the column numeric_precision_radix.

start_value character_data

The start value of the sequence

minimum_value character_data

The minimum value of the sequence

maximum_value character_data

The maximum value of the sequence

increment character_data

The increment of the sequence

cycle_option yes_or_no

YES if the sequence cycles, else NO

Note that in accordance with the SQL standard, the start, minimum, maximum, and increment values are returned as character strings.

Prev | Up | Next
37.46. schemata | Home | 37.48. sql_features


37.48. sql_features
Prev Up Chapter 37. The Information Schema Home Next

37.48. sql_features

The table sql_features contains information about which formal features defined in the SQL standard are supported by PostgreSQL. This is the same information that is presented in Appendix D. There you can also find some additional background information.

Table 37.46. sql_features Columns

Column Type

Description

feature_id character_data

Identifier string of the feature

feature_name character_data

Descriptive name of the feature

sub_feature_id character_data

Identifier string of the subfeature, or a zero-length string if not a subfeature

sub_feature_name character_data

Descriptive name of the subfeature, or a zero-length string if not a subfeature

is_supported yes_or_no

YES if the feature is fully supported by the current version of PostgreSQL, NO if not

is_verified_by character_data

Always null, since the PostgreSQL development group does not perform formal testing of feature conformance

comments character_data

Possibly a comment about the supported status of the feature

Prev | Up | Next
37.47. sequences | Home | 37.49. sql_implementation_info


37.49. sql_implementation_info
Prev Up Chapter 37. The Information Schema Home Next

37.49. sql_implementation_info

The table sql_implementation_info contains information about various aspects that are left implementation-defined by the SQL standard. This information is primarily intended for use in the context of the ODBC interface; users of other interfaces will probably find this information to be of little use. For this reason, the individual implementation information items are not described here; you will find them in the description of the ODBC interface.

Table 37.47. sql_implementation_info Columns

Column Type

Description

implementation_info_id character_data

Identifier string of the implementation information item

implementation_info_name character_data

Descriptive name of the implementation information item

integer_value cardinal_number

Value of the implementation information item, or null if the value is contained in the column character_value

character_value character_data

Value of the implementation information item, or null if the value is contained in the column integer_value

comments character_data

Possibly a comment pertaining to the implementation information item

Prev | Up | Next
37.48. sql_features | Home | 37.50. sql_parts


37.50. sql_parts
Prev Up Chapter 37. The Information Schema Home Next

37.50. sql_parts

The table sql_parts contains information about which of the several parts of the SQL standard are supported by PostgreSQL.

Table 37.48. sql_parts Columns

Column Type

Description

feature_id character_data

An identifier string containing the number of the part

feature_name character_data

Descriptive name of the part

is_supported yes_or_no

YES if the part is fully supported by the current version of PostgreSQL, NO if not

is_verified_by character_data

Always null, since the PostgreSQL development group does not perform formal testing of feature conformance

comments character_data

Possibly a comment about the supported status of the part

Prev | Up | Next
37.49. sql_implementation_info | Home | 37.51. sql_sizing


37.51. sql_sizing
Prev Up Chapter 37. The Information Schema Home Next

37.51. sql_sizing

The table sql_sizing contains information about various size limits and maximum values in PostgreSQL. This information is primarily intended for use in the context of the ODBC interface; users of other interfaces will probably find this information to be of little use. For this reason, the individual sizing items are not described here; you will find them in the description of the ODBC interface.

Table 37.49. sql_sizing Columns

Column Type

Description

sizing_id cardinal_number

Identifier of the sizing item

sizing_name character_data

Descriptive name of the sizing item

supported_value cardinal_number

Value of the sizing item, or 0 if the size is unlimited or cannot be determined, or null if the features for which the sizing item is applicable are not supported

comments character_data

Possibly a comment pertaining to the sizing item

Prev | Up | Next
37.50. sql_parts | Home | 37.52. table_constraints


37.52. table_constraints
Prev Up Chapter 37. The Information Schema Home Next

37.52. table_constraints

The view table_constraints contains all constraints belonging to tables that the current user owns or has some privilege other than SELECT on.

Table 37.50. table_constraints Columns

Column Type

Description

constraint_catalog sql_identifier

Name of the database that contains the constraint (always the current database)

constraint_schema sql_identifier

Name of the schema that contains the constraint

constraint_name sql_identifier

Name of the constraint

table_catalog sql_identifier

Name of the database that contains the table (always the current database)

table_schema sql_identifier

Name of the schema that contains the table

table_name sql_identifier

Name of the table

constraint_type character_data

Type of the constraint: CHECK, FOREIGN KEY, PRIMARY KEY, or UNIQUE

is_deferrable yes_or_no

YES if the constraint is deferrable, NO if not

initially_deferred yes_or_no

YES if the constraint is deferrable and initially deferred, NO if not

enforced yes_or_no

Applies to a feature not available in PostgreSQL (currently always YES)

Prev | Up | Next
37.51. sql_sizing | Home | 37.53. table_privileges


37.53. table_privileges
Prev Up Chapter 37. The Information Schema Home Next

37.53. table_privileges

The view table_privileges identifies all privileges granted on tables or views to a currently enabled role or by a currently enabled role. There is one row for each combination of table, grantor, and grantee.

Table 37.51. table_privileges Columns

Column Type

Description

grantor sql_identifier

Name of the role that granted the privilege

grantee sql_identifier

Name of the role that the privilege was granted to

table_catalog sql_identifier

Name of the database that contains the table (always the current database)

table_schema sql_identifier

Name of the schema that contains the table

table_name sql_identifier

Name of the table

privilege_type character_data

Type of the privilege: SELECT, INSERT, UPDATE, DELETE, TRUNCATE, REFERENCES, or TRIGGER

is_grantable yes_or_no

YES if the privilege is grantable, NO if not

with_hierarchy yes_or_no

In the SQL standard, WITH HIERARCHY OPTION is a separate (sub-)privilege allowing certain operations on table inheritance hierarchies. In PostgreSQL, this is included in the SELECT privilege, so this column shows YES if the privilege is SELECT, else NO.

Prev | Up | Next
37.52. table_constraints | Home | 37.54. tables


37.54. tables
Prev Up Chapter 37. The Information Schema Home Next

37.54. tables

The view tables contains all tables and views defined in the current database. Only those tables and views are shown that the current user has access to (by way of being the owner or having some privilege).

Table 37.52. tables Columns

Column Type

Description

table_catalog sql_identifier

Name of the database that contains the table (always the current database)

table_schema sql_identifier

Name of the schema that contains the table

table_name sql_identifier

Name of the table

table_type character_data

Type of the table: BASE TABLE for a persistent base table (the normal table type), VIEW for a view, FOREIGN for a foreign table, or LOCAL TEMPORARY for a temporary table

self_referencing_column_name sql_identifier

Applies to a feature not available in PostgreSQL

reference_generation character_data

Applies to a feature not available in PostgreSQL

user_defined_type_catalog sql_identifier

If the table is a typed table, the name of the database that contains the underlying data type (always the current database), else null.

user_defined_type_schema sql_identifier

If the table is a typed table, the name of the schema that contains the underlying data type, else null.

user_defined_type_name sql_identifier

If the table is a typed table, the name of the underlying data type, else null.

is_insertable_into yes_or_no

YES if the table is insertable into, NO if not (Base tables are always insertable into, views not necessarily.)

is_typed yes_or_no

YES if the table is a typed table, NO if not

commit_action character_data

Not yet implemented

Prev | Up | Next
37.53. table_privileges | Home | 37.55. transforms


37.55. transforms
Prev Up Chapter 37. The Information Schema Home Next

37.55. transforms

The view transforms contains information about the transforms defined in the current database. More precisely, it contains a row for each function contained in a transform (the “from SQL” or “to SQL” function).

Table 37.53. transforms Columns

Column Type

Description

udt_catalog sql_identifier

Name of the database that contains the type the transform is for (always the current database)

udt_schema sql_identifier

Name of the schema that contains the type the transform is for

udt_name sql_identifier

Name of the type the transform is for

specific_catalog sql_identifier

Name of the database containing the function (always the current database)

specific_schema sql_identifier

Name of the schema containing the function

specific_name sql_identifier

The “specific name” of the function. See Section 37.45 for more information.

group_name sql_identifier

The SQL standard allows defining transforms in “groups”, and selecting a group at run time. PostgreSQL does not support this. Instead, transforms are specific to a language. As a compromise, this field contains the language the transform is for.

transform_type character_data

FROM SQL or TO SQL

Prev | Up | Next
37.54. tables | Home | 37.56. triggered_update_columns


37.56. triggered_update_columns
Prev Up Chapter 37. The Information Schema Home Next

37.56. triggered_update_columns

For triggers in the current database that specify a column list (like UPDATE OF column1, column2), the view triggered_update_columns identifies these columns. Triggers that do not specify a column list are not included in this view. Only those columns are shown that the current user owns or has some privilege other than SELECT on.

Table 37.54. triggered_update_columns Columns

Column Type

Description

trigger_catalog sql_identifier

Name of the database that contains the trigger (always the current database)

trigger_schema sql_identifier

Name of the schema that contains the trigger

trigger_name sql_identifier

Name of the trigger

event_object_catalog sql_identifier

Name of the database that contains the table that the trigger is defined on (always the current database)

event_object_schema sql_identifier

Name of the schema that contains the table that the trigger is defined on

event_object_table sql_identifier

Name of the table that the trigger is defined on

event_object_column sql_identifier

Name of the column that the trigger is defined on

Prev | Up | Next
37.55. transforms | Home | 37.57. triggers


37.57. triggers
Prev Up Chapter 37. The Information Schema Home Next

37.57. triggers

The view triggers contains all triggers defined in the current database on tables and views that the current user owns or has some privilege other than SELECT on.

Table 37.55. triggers Columns

Column Type

Description

trigger_catalog sql_identifier

Name of the database that contains the trigger (always the current database)

trigger_schema sql_identifier

Name of the schema that contains the trigger

trigger_name sql_identifier

Name of the trigger

event_manipulation character_data

Event that fires the trigger (INSERT, UPDATE, or DELETE)

event_object_catalog sql_identifier

Name of the database that contains the table that the trigger is defined on (always the current database)

event_object_schema sql_identifier

Name of the schema that contains the table that the trigger is defined on

event_object_table sql_identifier

Name of the table that the trigger is defined on

action_order cardinal_number

Firing order among triggers on the same table having the same event_manipulation, action_timing, and action_orientation. In PostgreSQL, triggers are fired in name order, so this column reflects that.

action_condition character_data

WHEN condition of the trigger, null if none (also null if the table is not owned by a currently enabled role)

action_statement character_data

Statement that is executed by the trigger (currently always EXECUTE FUNCTION function(...))

action_orientation character_data

Identifies whether the trigger fires once for each processed row or once for each statement (ROW or STATEMENT)

action_timing character_data

Time at which the trigger fires (BEFORE, AFTER, or INSTEAD OF)

action_reference_old_table sql_identifier

Name of the “old” transition table, or null if none

action_reference_new_table sql_identifier

Name of the “new” transition table, or null if none

action_reference_old_row sql_identifier

Applies to a feature not available in PostgreSQL

action_reference_new_row sql_identifier

Applies to a feature not available in PostgreSQL

created time_stamp

Applies to a feature not available in PostgreSQL

Triggers in PostgreSQL have two incompatibilities with the SQL standard that affect the representation in the information schema. First, trigger names are local to each table in PostgreSQL, rather than being independent schema objects. Therefore there can be duplicate trigger names defined in one schema, so long as they belong to different tables. (trigger_catalog and trigger_schema are really the values pertaining to the table that the trigger is defined on.) Second, triggers can be defined to fire on multiple events in PostgreSQL (e.g., ON INSERT OR UPDATE), whereas the SQL standard only allows one. If a trigger is defined to fire on multiple events, it is represented as multiple rows in the information schema, one for each type of event. As a consequence of these two issues, the primary key of the view triggers is really (trigger_catalog, trigger_schema, event_object_table, trigger_name, event_manipulation) instead of (trigger_catalog, trigger_schema, trigger_name), which is what the SQL standard specifies. Nonetheless, if you define your triggers in a manner that conforms with the SQL standard (trigger names unique in the schema and only one event type per trigger), this will not affect you.

Note

Prior to PostgreSQL 9.1, this view's columns action_timing, action_reference_old_table, action_reference_new_table, action_reference_old_row, and action_reference_new_row were named condition_timing, condition_reference_old_table, condition_reference_new_table, condition_reference_old_row, and condition_reference_new_row respectively. That was how they were named in the SQL:1999 standard. The new naming conforms to SQL:2003 and later.

Prev | Up | Next
37.56. triggered_update_columns | Home | 37.58. udt_privileges


37.58. udt_privileges
Prev Up Chapter 37. The Information Schema Home Next

37.58. udt_privileges

The view udt_privileges identifies USAGE privileges granted on user-defined types to a currently enabled role or by a currently enabled role. There is one row for each combination of type, grantor, and grantee. This view shows only composite types (see under Section 37.60 for why); see Section 37.59 for domain privileges.

Table 37.56. udt_privileges Columns

Column Type

Description

grantor sql_identifier

Name of the role that granted the privilege

grantee sql_identifier

Name of the role that the privilege was granted to

udt_catalog sql_identifier

Name of the database containing the type (always the current database)

udt_schema sql_identifier

Name of the schema containing the type

udt_name sql_identifier

Name of the type

privilege_type character_data

Always TYPE USAGE

is_grantable yes_or_no

YES if the privilege is grantable, NO if not

Prev | Up | Next
37.57. triggers | Home | 37.59. usage_privileges


37.59. usage_privileges
Prev Up Chapter 37. The Information Schema Home Next

37.59. usage_privileges

The view usage_privileges identifies USAGE privileges granted on various kinds of objects to a currently enabled role or by a currently enabled role. In PostgreSQL, this currently applies to collations, domains, foreign-data wrappers, foreign servers, and sequences. There is one row for each combination of object, grantor, and grantee.

Since collations do not have real privileges in PostgreSQL, this view shows implicit non-grantable USAGE privileges granted by the owner to PUBLIC for all collations. The other object types, however, show real privileges.

In PostgreSQL, sequences also support SELECT and UPDATE privileges in addition to the USAGE privilege. These are nonstandard and therefore not visible in the information schema.

Table 37.57. usage_privileges Columns

Column Type

Description

grantor sql_identifier

Name of the role that granted the privilege

grantee sql_identifier

Name of the role that the privilege was granted to

object_catalog sql_identifier

Name of the database containing the object (always the current database)

object_schema sql_identifier

Name of the schema containing the object, if applicable, else an empty string

object_name sql_identifier

Name of the object

object_type character_data

COLLATION or DOMAIN or FOREIGN DATA WRAPPER or FOREIGN SERVER or SEQUENCE

privilege_type character_data

Always USAGE

is_grantable yes_or_no

YES if the privilege is grantable, NO if not

Prev | Up | Next
37.58. udt_privileges | Home | 37.60. user_defined_types


37.60. user_defined_types
Prev Up Chapter 37. The Information Schema Home Next

37.60. user_defined_types

The view user_defined_types currently contains all composite types defined in the current database. Only those types are shown that the current user has access to (by way of being the owner or having some privilege).

SQL knows about two kinds of user-defined types: structured types (also known as composite types in PostgreSQL) and distinct types (not implemented in PostgreSQL). To be future-proof, use the column user_defined_type_category to differentiate between these. Other user-defined types such as base types and enums, which are PostgreSQL extensions, are not shown here. For domains, see Section 37.23 instead.

Table 37.58. user_defined_types Columns

Column Type

Description

user_defined_type_catalog sql_identifier

Name of the database that contains the type (always the current database)

user_defined_type_schema sql_identifier

Name of the schema that contains the type

user_defined_type_name sql_identifier

Name of the type

user_defined_type_category character_data

Currently always STRUCTURED

is_instantiable yes_or_no

Applies to a feature not available in PostgreSQL

is_final yes_or_no

Applies to a feature not available in PostgreSQL

ordering_form character_data

Applies to a feature not available in PostgreSQL

ordering_category character_data

Applies to a feature not available in PostgreSQL

ordering_routine_catalog sql_identifier

Applies to a feature not available in PostgreSQL

ordering_routine_schema sql_identifier

Applies to a feature not available in PostgreSQL

ordering_routine_name sql_identifier

Applies to a feature not available in PostgreSQL

reference_type character_data

Applies to a feature not available in PostgreSQL

data_type character_data

Applies to a feature not available in PostgreSQL

character_maximum_length cardinal_number

Applies to a feature not available in PostgreSQL

character_octet_length cardinal_number

Applies to a feature not available in PostgreSQL

character_set_catalog sql_identifier

Applies to a feature not available in PostgreSQL

character_set_schema sql_identifier

Applies to a feature not available in PostgreSQL

character_set_name sql_identifier

Applies to a feature not available in PostgreSQL

collation_catalog sql_identifier

Applies to a feature not available in PostgreSQL

collation_schema sql_identifier

Applies to a feature not available in PostgreSQL

collation_name sql_identifier

Applies to a feature not available in PostgreSQL

numeric_precision cardinal_number

Applies to a feature not available in PostgreSQL

numeric_precision_radix cardinal_number

Applies to a feature not available in PostgreSQL

numeric_scale cardinal_number

Applies to a feature not available in PostgreSQL

datetime_precision cardinal_number

Applies to a feature not available in PostgreSQL

interval_type character_data

Applies to a feature not available in PostgreSQL

interval_precision cardinal_number

Applies to a feature not available in PostgreSQL

source_dtd_identifier sql_identifier

Applies to a feature not available in PostgreSQL

ref_dtd_identifier sql_identifier

Applies to a feature not available in PostgreSQL

Prev | Up | Next
37.59. usage_privileges | Home | 37.61. user_mapping_options


37.61. user_mapping_options
Prev Up Chapter 37. The Information Schema Home Next

37.61. user_mapping_options

The view user_mapping_options contains all the options defined for user mappings in the current database. Only those user mappings are shown where the current user has access to the corresponding foreign server (by way of being the owner or having some privilege).

Table 37.59. user_mapping_options Columns

Column Type

Description

authorization_identifier sql_identifier

Name of the user being mapped, or PUBLIC if the mapping is public

foreign_server_catalog sql_identifier

Name of the database that the foreign server used by this mapping is defined in (always the current database)

foreign_server_name sql_identifier

Name of the foreign server used by this mapping

option_name sql_identifier

Name of an option

option_value character_data

Value of the option. This column will show as null unless the current user is the user being mapped, or the mapping is for PUBLIC and the current user is the server owner, or the current user is a superuser. The intent is to protect password information stored as user mapping option.

Prev | Up | Next
37.60. user_defined_types | Home | 37.62. user_mappings


37.62. user_mappings
Prev Up Chapter 37. The Information Schema Home Next

37.62. user_mappings

The view user_mappings contains all user mappings defined in the current database. Only those user mappings are shown where the current user has access to the corresponding foreign server (by way of being the owner or having some privilege).

Table 37.60. user_mappings Columns

Column Type

Description

authorization_identifier sql_identifier

Name of the user being mapped, or PUBLIC if the mapping is public

foreign_server_catalog sql_identifier

Name of the database that the foreign server used by this mapping is defined in (always the current database)

foreign_server_name sql_identifier

Name of the foreign server used by this mapping

Prev | Up | Next
37.61. user_mapping_options | Home | 37.63. view_column_usage


37.63. view_column_usage
Prev Up Chapter 37. The Information Schema Home Next

37.63. view_column_usage

The view view_column_usage identifies all columns that are used in the query expression of a view (the SELECT statement that defines the view). A column is only included if the table that contains the column is owned by a currently enabled role.

Note

Columns of system tables are not included. This should be fixed sometime.

Table 37.61. view_column_usage Columns

Column Type

Description

view_catalog sql_identifier

Name of the database that contains the view (always the current database)

view_schema sql_identifier

Name of the schema that contains the view

view_name sql_identifier

Name of the view

table_catalog sql_identifier

Name of the database that contains the table that contains the column that is used by the view (always the current database)

table_schema sql_identifier

Name of the schema that contains the table that contains the column that is used by the view

table_name sql_identifier

Name of the table that contains the column that is used by the view

column_name sql_identifier

Name of the column that is used by the view

Prev | Up | Next
37.62. user_mappings | Home | 37.64. view_routine_usage


37.64. view_routine_usage
Prev Up Chapter 37. The Information Schema Home Next

37.64. view_routine_usage

The view view_routine_usage identifies all routines (functions and procedures) that are used in the query expression of a view (the SELECT statement that defines the view). A routine is only included if that routine is owned by a currently enabled role.

Table 37.62. view_routine_usage Columns

Column Type

Description

table_catalog sql_identifier

Name of the database containing the view (always the current database)

table_schema sql_identifier

Name of the schema containing the view

table_name sql_identifier

Name of the view

specific_catalog sql_identifier

Name of the database containing the function (always the current database)

specific_schema sql_identifier

Name of the schema containing the function

specific_name sql_identifier

The “specific name” of the function. See Section 37.45 for more information.

Prev | Up | Next
37.63. view_column_usage | Home | 37.65. view_table_usage


37.65. view_table_usage
Prev Up Chapter 37. The Information Schema Home Next

37.65. view_table_usage

The view view_table_usage identifies all tables that are used in the query expression of a view (the SELECT statement that defines the view). A table is only included if that table is owned by a currently enabled role.

Note

System tables are not included. This should be fixed sometime.

Table 37.63. view_table_usage Columns

Column Type

Description

view_catalog sql_identifier

Name of the database that contains the view (always the current database)

view_schema sql_identifier

Name of the schema that contains the view

view_name sql_identifier

Name of the view

table_catalog sql_identifier

Name of the database that contains the table that is used by the view (always the current database)

table_schema sql_identifier

Name of the schema that contains the table that is used by the view

table_name sql_identifier

Name of the table that is used by the view

Prev | Up | Next
37.64. view_routine_usage | Home | 37.66. views


37.66. views
Prev Up Chapter 37. The Information Schema Home Next

37.66. views

The view views contains all views defined in the current database. Only those views are shown that the current user has access to (by way of being the owner or having some privilege).

Table 37.64. views Columns

Column Type

Description

table_catalog sql_identifier

Name of the database that contains the view (always the current database)

table_schema sql_identifier

Name of the schema that contains the view

table_name sql_identifier

Name of the view

view_definition character_data

Query expression defining the view (null if the view is not owned by a currently enabled role)

check_option character_data

CASCADED or LOCAL if the view has a CHECK OPTION defined on it, NONE if not

is_updatable yes_or_no

YES if the view is updatable (allows UPDATE and DELETE), NO if not

is_insertable_into yes_or_no

YES if the view is insertable into (allows INSERT), NO if not

is_trigger_updatable yes_or_no

YES if the view has an INSTEAD OF UPDATE trigger defined on it, NO if not

is_trigger_deletable yes_or_no

YES if the view has an INSTEAD OF DELETE trigger defined on it, NO if not

is_trigger_insertable_into yes_or_no

YES if the view has an INSTEAD OF INSERT trigger defined on it, NO if not

Prev | Up | Next
37.65. view_table_usage | Home | Part V. Server Programming


Chapter 38. Extending SQL
Prev Up Part V. Server Programming Home Next

Chapter 38. Extending SQL

Table of Contents

38.1. How Extensibility Works
38.2. The PostgreSQL Type System
38.2.1. Base Types
38.2.2. Container Types
38.2.3. Domains
38.2.4. Pseudo-Types
38.2.5. Polymorphic Types
38.3. User-Defined Functions
38.4. User-Defined Procedures
38.5. Query Language (SQL) Functions
38.5.1. Arguments for SQL Functions
38.5.2. SQL Functions on Base Types
38.5.3. SQL Functions on Composite Types
38.5.4. SQL Functions with Output Parameters
38.5.5. SQL Procedures with Output Parameters
38.5.6. SQL Functions with Variable Numbers of Arguments
38.5.7. SQL Functions with Default Values for Arguments
38.5.8. SQL Functions as Table Sources
38.5.9. SQL Functions Returning Sets
38.5.10. SQL Functions Returning TABLE
38.5.11. Polymorphic SQL Functions
38.5.12. SQL Functions with Collations
38.6. Function Overloading
38.7. Function Volatility Categories
38.8. Procedural Language Functions
38.9. Internal Functions
38.10. C-Language Functions
38.10.1. Dynamic Loading
38.10.2. Base Types in C-Language Functions
38.10.3. Version 1 Calling Conventions
38.10.4. Writing Code
38.10.5. Compiling and Linking Dynamically-Loaded Functions
38.10.6. Composite-Type Arguments
38.10.7. Returning Rows (Composite Types)
38.10.8. Returning Sets
38.10.9. Polymorphic Arguments and Return Types
38.10.10. Shared Memory and LWLocks
38.10.11. Using C++ for Extensibility
38.11. Function Optimization Information
38.12. User-Defined Aggregates
38.12.1. Moving-Aggregate Mode
38.12.2. Polymorphic and Variadic Aggregates
38.12.3. Ordered-Set Aggregates
38.12.4. Partial Aggregation
38.12.5. Support Functions for Aggregates
38.13. User-Defined Types
38.13.1. TOAST Considerations
38.14. User-Defined Operators
38.15. Operator Optimization Information
38.15.1. COMMUTATOR
38.15.2. NEGATOR
38.15.3. RESTRICT
38.15.4. JOIN
38.15.5. HASHES
38.15.6. MERGES
38.16. Interfacing Extensions to Indexes
38.16.1. Index Methods and Operator Classes
38.16.2. Index Method Strategies
38.16.3. Index Method Support Routines
38.16.4. An Example
38.16.5. Operator Classes and Operator Families
38.16.6. System Dependencies on Operator Classes
38.16.7. Ordering Operators
38.16.8. Special Features of Operator Classes
38.17. Packaging Related Objects into an Extension
38.17.1. Extension Files
38.17.2. Extension Relocatability
38.17.3. Extension Configuration Tables
38.17.4. Extension Updates
38.17.5. Installing Extensions Using Update Scripts
38.17.6. Security Considerations for Extensions
38.17.7. Extension Example
38.18. Extension Building Infrastructure

In the sections that follow, we will discuss how you can extend the PostgreSQL SQL query language by adding:

  • functions (starting in Section 38.3)

  • aggregates (starting in Section 38.12)

  • data types (starting in Section 38.13)

  • operators (starting in Section 38.14)

  • operator classes for indexes (starting in Section 38.16)

  • packages of related objects (starting in Section 38.17)

Prev | Up | Next
Part V. Server Programming | Home | 38.1. How Extensibility Works


38.1. How Extensibility Works
Prev Up Chapter 38. Extending SQL Home Next

38.1. How Extensibility Works

PostgreSQL is extensible because its operation is catalog-driven. If you are familiar with standard relational database systems, you know that they store information about databases, tables, columns, etc., in what are commonly known as system catalogs. (Some systems call this the data dictionary.) The catalogs appear to the user as tables like any other, but the DBMS stores its internal bookkeeping in them. One key difference between PostgreSQL and standard relational database systems is that PostgreSQL stores much more information in its catalogs: not only information about tables and columns, but also information about data types, functions, access methods, and so on. These tables can be modified by the user, and since PostgreSQL bases its operation on these tables, this means that PostgreSQL can be extended by users. By comparison, conventional database systems can only be extended by changing hardcoded procedures in the source code or by loading modules specially written by the DBMS vendor.

The PostgreSQL server can moreover incorporate user-written code into itself through dynamic loading. That is, the user can specify an object code file (e.g., a shared library) that implements a new type or function, and PostgreSQL will load it as required. Code written in SQL is even more trivial to add to the server. This ability to modify its operation “on the fly” makes PostgreSQL uniquely suited for rapid prototyping of new applications and storage structures.

Prev | Up | Next
Chapter 38. Extending SQL | Home | 38.2. The PostgreSQL Type System


38.2. The PostgreSQL Type System
Prev Up Chapter 38. Extending SQL Home Next

38.2. The PostgreSQL Type System

38.2.1. Base Types
38.2.2. Container Types
38.2.3. Domains
38.2.4. Pseudo-Types
38.2.5. Polymorphic Types

PostgreSQL data types can be divided into base types, container types, domains, and pseudo-types.

38.2.1. Base Types

Base types are those, like integer, that are implemented below the level of the SQL language (typically in a low-level language such as C). They generally correspond to what are often known as abstract data types. PostgreSQL can only operate on such types through functions provided by the user and only understands the behavior of such types to the extent that the user describes them. The built-in base types are described in Chapter 8.

Enumerated (enum) types can be considered as a subcategory of base types. The main difference is that they can be created using just SQL commands, without any low-level programming. Refer to Section 8.7 for more information.

38.2.2. Container Types

PostgreSQL has three kinds of “container” types, which are types that contain multiple values of other types. These are arrays, composites, and ranges.

Arrays can hold multiple values that are all of the same type. An array type is automatically created for each base type, composite type, range type, and domain type. But there are no arrays of arrays. So far as the type system is concerned, multi-dimensional arrays are the same as one-dimensional arrays. Refer to Section 8.15 for more information.

Composite types, or row types, are created whenever the user creates a table. It is also possible to use CREATE TYPE to define a “stand-alone” composite type with no associated table. A composite type is simply a list of types with associated field names. A value of a composite type is a row or record of field values. Refer to Section 8.16 for more information.

A range type can hold two values of the same type, which are the lower and upper bounds of the range. Range types are user-created, although a few built-in ones exist. Refer to Section 8.17 for more information.

38.2.3. Domains

A domain is based on a particular underlying type and for many purposes is interchangeable with its underlying type. However, a domain can have constraints that restrict its valid values to a subset of what the underlying type would allow. Domains are created using the SQL command CREATE DOMAIN. Refer to Section 8.18 for more information.

38.2.4. Pseudo-Types

There are a few “pseudo-types” for special purposes. Pseudo-types cannot appear as columns of tables or components of container types, but they can be used to declare the argument and result types of functions. This provides a mechanism within the type system to identify special classes of functions. Table 8.27 lists the existing pseudo-types.

38.2.5. Polymorphic Types

Some pseudo-types of special interest are the polymorphic types, which are used to declare polymorphic functions. This powerful feature allows a single function definition to operate on many different data types, with the specific data type(s) being determined by the data types actually passed to it in a particular call. The polymorphic types are shown in Table 38.1. Some examples of their use appear in Section 38.5.11.

Table 38.1. Polymorphic Types

Name | Family | Description
anyelement | Simple | Indicates that a function accepts any data type
anyarray | Simple | Indicates that a function accepts any array data type
anynonarray | Simple | Indicates that a function accepts any non-array data type
anyenum | Simple | Indicates that a function accepts any enum data type (see Section 8.7)
anyrange | Simple | Indicates that a function accepts any range data type (see Section 8.17)
anymultirange | Simple | Indicates that a function accepts any multirange data type (see Section 8.17)
anycompatible | Common | Indicates that a function accepts any data type, with automatic promotion of multiple arguments to a common data type
anycompatiblearray | Common | Indicates that a function accepts any array data type, with automatic promotion of multiple arguments to a common data type
anycompatiblenonarray | Common | Indicates that a function accepts any non-array data type, with automatic promotion of multiple arguments to a common data type
anycompatiblerange | Common | Indicates that a function accepts any range data type, with automatic promotion of multiple arguments to a common data type
anycompatiblemultirange | Common | Indicates that a function accepts any multirange data type, with automatic promotion of multiple arguments to a common data type

Polymorphic arguments and results are tied to each other and are resolved to specific data types when a query calling a polymorphic function is parsed. When there is more than one polymorphic argument, the actual data types of the input values must match up as described below. If the function's result type is polymorphic, or it has output parameters of polymorphic types, the types of those results are deduced from the actual types of the polymorphic inputs as described below.

For the “simple” family of polymorphic types, the matching and deduction rules work like this:

Each position (either argument or return value) declared as anyelement is allowed to have any specific actual data type, but in any given call they must all be the same actual type. Each position declared as anyarray can have any array data type, but similarly they must all be the same type. And similarly, positions declared as anyrange must all be the same range type. Likewise for anymultirange.

Furthermore, if there are positions declared anyarray and others declared anyelement, the actual array type in the anyarray positions must be an array whose elements are the same type appearing in the anyelement positions. anynonarray is treated exactly the same as anyelement, but adds the additional constraint that the actual type must not be an array type. anyenum is treated exactly the same as anyelement, but adds the additional constraint that the actual type must be an enum type.

Similarly, if there are positions declared anyrange and others declared anyelement or anyarray, the actual range type in the anyrange positions must be a range whose subtype is the same type appearing in the anyelement positions and the same as the element type of the anyarray positions. If there are positions declared anymultirange, their actual multirange type must contain ranges matching parameters declared anyrange and base elements matching parameters declared anyelement and anyarray.

Thus, when more than one argument position is declared with a polymorphic type, the net effect is that only certain combinations of actual argument types are allowed. For example, a function declared as equal(anyelement, anyelement) will take any two input values, so long as they are of the same data type.

When the return value of a function is declared as a polymorphic type, there must be at least one argument position that is also polymorphic, and the actual data type(s) supplied for the polymorphic arguments determine the actual result type for that call. For example, if there were not already an array subscripting mechanism, one could define a function that implements subscripting as subscript(anyarray, integer) returns anyelement. This declaration constrains the actual first argument to be an array type, and allows the parser to infer the correct result type from the actual first argument's type. Another example is that a function declared as f(anyarray) returns anyenum will only accept arrays of enum types.

In most cases, the parser can infer the actual data type for a polymorphic result type from arguments that are of a different polymorphic type in the same family; for example anyarray can be deduced from anyelement or vice versa. An exception is that a polymorphic result of type anyrange requires an argument of type anyrange; it cannot be deduced from anyarray or anyelement arguments. This is because there could be multiple range types with the same subtype.

Note that anynonarray and anyenum do not represent separate type variables; they are the same type as anyelement, just with an additional constraint. For example, declaring a function as f(anyelement, anyenum) is equivalent to declaring it as f(anyenum, anyenum): both actual arguments have to be the same enum type.

For the “common” family of polymorphic types, the matching and deduction rules work approximately the same as for the “simple” family, with one major difference: the actual types of the arguments need not be identical, so long as they can be implicitly cast to a single common type. The common type is selected following the same rules as for UNION and related constructs (see Section 10.5). Selection of the common type considers the actual types of anycompatible and anycompatiblenonarray inputs, the array element types of anycompatiblearray inputs, the range subtypes of anycompatiblerange inputs, and the multirange subtypes of anycompatiblemultirange inputs. If anycompatiblenonarray is present then the common type is required to be a non-array type. Once a common type is identified, arguments in anycompatible and anycompatiblenonarray positions are automatically cast to that type, and arguments in anycompatiblearray positions are automatically cast to the array type for that type.

Since there is no way to select a range type knowing only its subtype, use of anycompatiblerange and/or anycompatiblemultirange requires that all arguments declared with that type have the same actual range and/or multirange type, and that that type's subtype agree with the selected common type, so that no casting of the range values is required. As with anyrange and anymultirange, use of anycompatiblerange and anymultirange as a function result type requires that there be an anycompatiblerange or anycompatiblemultirange argument.

Notice that there is no anycompatibleenum type. Such a type would not be very useful, since there normally are not any implicit casts to enum types, meaning that there would be no way to resolve a common type for dissimilar enum inputs.

The “simple” and “common” polymorphic families represent two independent sets of type variables. Consider for example

CREATE FUNCTION myfunc(a anyelement, b anyelement,
                       c anycompatible, d anycompatible)
RETURNS anycompatible AS ...

In an actual call of this function, the first two inputs must have exactly the same type. The last two inputs must be promotable to a common type, but this type need not have anything to do with the type of the first two inputs. The result will have the common type of the last two inputs.

A variadic function (one taking a variable number of arguments, as in Section 38.5.6) can be polymorphic: this is accomplished by declaring its last parameter as VARIADIC anyarray or VARIADIC anycompatiblearray. For purposes of argument matching and determining the actual result type, such a function behaves the same as if you had written the appropriate number of anynonarray or anycompatiblenonarray parameters.

Prev | Up | Next
38.1. How Extensibility Works | Home | 38.3. User-Defined Functions


38.17. Packaging Related Objects into an Extension
Prev Up Chapter 38. Extending SQL Home Next

38.17. Packaging Related Objects into an Extension

38.17.1. Extension Files
38.17.2. Extension Relocatability
38.17.3. Extension Configuration Tables
38.17.4. Extension Updates
38.17.5. Installing Extensions Using Update Scripts
38.17.6. Security Considerations for Extensions
38.17.7. Extension Example

A useful extension to PostgreSQL typically includes multiple SQL objects; for example, a new data type will require new functions, new operators, and probably new index operator classes. It is helpful to collect all these objects into a single package to simplify database management. PostgreSQL calls such a package an extension. To define an extension, you need at least a script file that contains the SQL commands to create the extension's objects, and a control file that specifies a few basic properties of the extension itself. If the extension includes C code, there will typically also be a shared library file into which the C code has been built. Once you have these files, a simple CREATE EXTENSION command loads the objects into your database.

The main advantage of using an extension, rather than just running the SQL script to load a bunch of “loose” objects into your database, is that PostgreSQL will then understand that the objects of the extension go together. You can drop all the objects with a single DROP EXTENSION command (no need to maintain a separate “uninstall” script). Even more useful, pg_dump knows that it should not dump the individual member objects of the extension — it will just include a CREATE EXTENSION command in dumps, instead. This vastly simplifies migration to a new version of the extension that might contain more or different objects than the old version. Note however that you must have the extension's control, script, and other files available when loading such a dump into a new database.

PostgreSQL will not let you drop an individual object contained in an extension, except by dropping the whole extension. Also, while you can change the definition of an extension member object (for example, via CREATE OR REPLACE FUNCTION for a function), bear in mind that the modified definition will not be dumped by pg_dump. Such a change is usually only sensible if you concurrently make the same change in the extension's script file. (But there are special provisions for tables containing configuration data; see Section 38.17.3.) In production situations, it's generally better to create an extension update script to perform changes to extension member objects.

The extension script may set privileges on objects that are part of the extension, using GRANT and REVOKE statements. The final set of privileges for each object (if any are set) will be stored in the pg_init_privs system catalog. When pg_dump is used, the CREATE EXTENSION command will be included in the dump, followed by the set of GRANT and REVOKE statements necessary to set the privileges on the objects to what they were at the time the dump was taken.

PostgreSQL does not currently support extension scripts issuing CREATE POLICY or SECURITY LABEL statements. These are expected to be set after the extension has been created. All RLS policies and security labels on extension objects will be included in dumps created by pg_dump.

The extension mechanism also has provisions for packaging modification scripts that adjust the definitions of the SQL objects contained in an extension. For example, if version 1.1 of an extension adds one function and changes the body of another function compared to 1.0, the extension author can provide an update script that makes just those two changes. The ALTER EXTENSION UPDATE command can then be used to apply these changes and track which version of the extension is actually installed in a given database.

The kinds of SQL objects that can be members of an extension are shown in the description of ALTER EXTENSION. Notably, objects that are database-cluster-wide, such as databases, roles, and tablespaces, cannot be extension members since an extension is only known within one database. (Although an extension script is not prohibited from creating such objects, if it does so they will not be tracked as part of the extension.) Also notice that while a table can be a member of an extension, its subsidiary objects such as indexes are not directly considered members of the extension. Another important point is that schemas can belong to extensions, but not vice versa: an extension as such has an unqualified name and does not exist “within” any schema. The extension's member objects, however, will belong to schemas whenever appropriate for their object types. It may or may not be appropriate for an extension to own the schema(s) its member objects are within.

If an extension's script creates any temporary objects (such as temp tables), those objects are treated as extension members for the remainder of the current session, but are automatically dropped at session end, as any temporary object would be. This is an exception to the rule that extension member objects cannot be dropped without dropping the whole extension.

38.17.1. Extension Files

The CREATE EXTENSION command relies on a control file for each extension, which must be named the same as the extension with a suffix of .control, and must be placed in the installation's SHAREDIR/extension directory. There must also be at least one SQL script file, which follows the naming pattern extension--version.sql (for example, foo--1.0.sql for version 1.0 of extension foo). By default, the script file(s) are also placed in the SHAREDIR/extension directory; but the control file can specify a different directory for the script file(s).

The file format for an extension control file is the same as for the postgresql.conf file, namely a list of parameter_name = value assignments, one per line. Blank lines and comments introduced by # are allowed. Be sure to quote any value that is not a single word or number.

A control file can set the following parameters:

directory (string)

The directory containing the extension's SQL script file(s). Unless an absolute path is given, the name is relative to the installation's SHAREDIR directory. The default behavior is equivalent to specifying directory = 'extension'.

default_version (string)

The default version of the extension (the one that will be installed if no version is specified in CREATE EXTENSION). Although this can be omitted, that will result in CREATE EXTENSION failing if no VERSION option appears, so you generally don't want to do that.

comment (string)

A comment (any string) about the extension. The comment is applied when initially creating an extension, but not during extension updates (since that might override user-added comments). Alternatively, the extension's comment can be set by writing a COMMENT command in the script file.

encoding (string)

The character set encoding used by the script file(s). This should be specified if the script files contain any non-ASCII characters. Otherwise the files will be assumed to be in the database encoding.

module_pathname (string)

The value of this parameter will be substituted for each occurrence of MODULE_PATHNAME in the script file(s). If it is not set, no substitution is made. Typically, this is set to $libdir/shared_library_name and then MODULE_PATHNAME is used in CREATE FUNCTION commands for C-language functions, so that the script files do not need to hard-wire the name of the shared library.

requires (string)

A list of names of extensions that this extension depends on, for example requires = 'foo, bar'. Those extensions must be installed before this one can be installed.

superuser (boolean)

If this parameter is true (which is the default), only superusers can create the extension or update it to a new version (but see also trusted, below). If it is set to false, just the privileges required to execute the commands in the installation or update script are required. This should normally be set to true if any of the script commands require superuser privileges. (Such commands would fail anyway, but it's more user-friendly to give the error up front.)

trusted (boolean)

This parameter, if set to true (which is not the default), allows some non-superusers to install an extension that has superuser set to true. Specifically, installation will be permitted for anyone who has CREATE privilege on the current database. When the user executing CREATE EXTENSION is not a superuser but is allowed to install by virtue of this parameter, then the installation or update script is run as the bootstrap superuser, not as the calling user. This parameter is irrelevant if superuser is false. Generally, this should not be set true for extensions that could allow access to otherwise-superuser-only abilities, such as file system access. Also, marking an extension trusted requires significant extra effort to write the extension's installation and update script(s) securely; see Section 38.17.6.

relocatable (boolean)

An extension is relocatable if it is possible to move its contained objects into a different schema after initial creation of the extension. The default is false, i.e., the extension is not relocatable. See Section 38.17.2 for more information.

schema (string)

This parameter can only be set for non-relocatable extensions. It forces the extension to be loaded into exactly the named schema and not any other. The schema parameter is consulted only when initially creating an extension, not during extension updates. See Section 38.17.2 for more information.

In addition to the primary control file extension.control, an extension can have secondary control files named in the style extension--version.control. If supplied, these must be located in the script file directory. Secondary control files follow the same format as the primary control file. Any parameters set in a secondary control file override the primary control file when installing or updating to that version of the extension. However, the parameters directory and default_version cannot be set in a secondary control file.

An extension's SQL script files can contain any SQL commands, except for transaction control commands (BEGIN, COMMIT, etc) and commands that cannot be executed inside a transaction block (such as VACUUM). This is because the script files are implicitly executed within a transaction block.

An extension's SQL script files can also contain lines beginning with \echo, which will be ignored (treated as comments) by the extension mechanism. This provision is commonly used to throw an error if the script file is fed to psql rather than being loaded via CREATE EXTENSION (see example script in Section 38.17.7). Without that, users might accidentally load the extension's contents as “loose” objects rather than as an extension, a state of affairs that's a bit tedious to recover from.

If the extension script contains the string @extowner@, that string is replaced with the (suitably quoted) name of the user calling CREATE EXTENSION or ALTER EXTENSION. Typically this feature is used by extensions that are marked trusted to assign ownership of selected objects to the calling user rather than the bootstrap superuser. (One should be careful about doing so, however. For example, assigning ownership of a C-language function to a non-superuser would create a privilege escalation path for that user.)

While the script files can contain any characters allowed by the specified encoding, control files should contain only plain ASCII, because there is no way for PostgreSQL to know what encoding a control file is in. In practice this is only an issue if you want to use non-ASCII characters in the extension's comment. Recommended practice in that case is to not use the control file comment parameter, but instead use COMMENT ON EXTENSION within a script file to set the comment.

38.17.2. Extension Relocatability

Users often wish to load the objects contained in an extension into a different schema than the extension's author had in mind. There are three supported levels of relocatability:

  • A fully relocatable extension can be moved into another schema at any time, even after it's been loaded into a database. This is done with the ALTER EXTENSION SET SCHEMA command, which automatically renames all the member objects into the new schema. Normally, this is only possible if the extension contains no internal assumptions about what schema any of its objects are in. Also, the extension's objects must all be in one schema to begin with (ignoring objects that do not belong to any schema, such as procedural languages). Mark a fully relocatable extension by setting relocatable = true in its control file.

  • An extension might be relocatable during installation but not afterwards. This is typically the case if the extension's script file needs to reference the target schema explicitly, for example in setting search_path properties for SQL functions. For such an extension, set relocatable = false in its control file, and use @extschema@ to refer to the target schema in the script file. All occurrences of this string will be replaced by the actual target schema's name before the script is executed. The user can set the target schema using the SCHEMA option of CREATE EXTENSION.

  • If the extension does not support relocation at all, set relocatable = false in its control file, and also set schema to the name of the intended target schema. This will prevent use of the SCHEMA option of CREATE EXTENSION, unless it specifies the same schema named in the control file. This choice is typically necessary if the extension contains internal assumptions about schema names that can't be replaced by uses of @extschema@. The @extschema@ substitution mechanism is available in this case too, although it is of limited use since the schema name is determined by the control file.

In all cases, the script file will be executed with search_path initially set to point to the target schema; that is, CREATE EXTENSION does the equivalent of this:

SET LOCAL search_path TO @extschema@, pg_temp;

This allows the objects created by the script file to go into the target schema. The script file can change search_path if it wishes, but that is generally undesirable. search_path is restored to its previous setting upon completion of CREATE EXTENSION.

The target schema is determined by the schema parameter in the control file if that is given, otherwise by the SCHEMA option of CREATE EXTENSION if that is given, otherwise the current default object creation schema (the first one in the caller's search_path). When the control file schema parameter is used, the target schema will be created if it doesn't already exist, but in the other two cases it must already exist.

If any prerequisite extensions are listed in requires in the control file, their target schemas are added to the initial setting of search_path, following the new extension's target schema. This allows their objects to be visible to the new extension's script file.

For security, pg_temp is automatically appended to the end of search_path in all cases.

Although a non-relocatable extension can contain objects spread across multiple schemas, it is usually desirable to place all the objects meant for external use into a single schema, which is considered the extension's target schema. Such an arrangement works conveniently with the default setting of search_path during creation of dependent extensions.

38.17.3. Extension Configuration Tables

Some extensions include configuration tables, which contain data that might be added or changed by the user after installation of the extension. Ordinarily, if a table is part of an extension, neither the table's definition nor its content will be dumped by pg_dump. But that behavior is undesirable for a configuration table; any data changes made by the user need to be included in dumps, or the extension will behave differently after a dump and reload.

To solve this problem, an extension's script file can mark a table or a sequence it has created as a configuration relation, which will cause pg_dump to include the table's or the sequence's contents (not its definition) in dumps. To do that, call the function pg_extension_config_dump(regclass, text) after creating the table or the sequence, for example

CREATE TABLE my_config (key text, value text);
CREATE SEQUENCE my_config_seq;

SELECT pg_catalog.pg_extension_config_dump('my_config', '');
SELECT pg_catalog.pg_extension_config_dump('my_config_seq', '');

Any number of tables or sequences can be marked this way. Sequences associated with serial or bigserial columns can be marked as well.

When the second argument of pg_extension_config_dump is an empty string, the entire contents of the table are dumped by pg_dump. This is usually only correct if the table is initially empty as created by the extension script. If there is a mixture of initial data and user-provided data in the table, the second argument of pg_extension_config_dump provides a WHERE condition that selects the data to be dumped. For example, you might do

CREATE TABLE my_config (key text, value text, standard_entry boolean);

SELECT pg_catalog.pg_extension_config_dump('my_config', 'WHERE NOT standard_entry');

and then make sure that standard_entry is true only in the rows created by the extension's script.

For sequences, the second argument of pg_extension_config_dump has no effect.

More complicated situations, such as initially-provided rows that might be modified by users, can be handled by creating triggers on the configuration table to ensure that modified rows are marked correctly.

You can alter the filter condition associated with a configuration table by calling pg_extension_config_dump again. (This would typically be useful in an extension update script.) The only way to mark a table as no longer a configuration table is to dissociate it from the extension with ALTER EXTENSION ... DROP TABLE.

Note that foreign key relationships between these tables will dictate the order in which the tables are dumped out by pg_dump. Specifically, pg_dump will attempt to dump the referenced-by table before the referencing table. As the foreign key relationships are set up at CREATE EXTENSION time (prior to data being loaded into the tables) circular dependencies are not supported. When circular dependencies exist, the data will still be dumped out but the dump will not be able to be restored directly and user intervention will be required.

Sequences associated with serial or bigserial columns need to be directly marked to dump their state. Marking their parent relation is not enough for this purpose.

38.17.4. Extension Updates

One advantage of the extension mechanism is that it provides convenient ways to manage updates to the SQL commands that define an extension's objects. This is done by associating a version name or number with each released version of the extension's installation script. In addition, if you want users to be able to update their databases dynamically from one version to the next, you should provide update scripts that make the necessary changes to go from one version to the next. Update scripts have names following the pattern extension--old_version--target_version.sql (for example, foo--1.0--1.1.sql contains the commands to modify version 1.0 of extension foo into version 1.1).

Given that a suitable update script is available, the command ALTER EXTENSION UPDATE will update an installed extension to the specified new version. The update script is run in the same environment that CREATE EXTENSION provides for installation scripts: in particular, search_path is set up in the same way, and any new objects created by the script are automatically added to the extension. Also, if the script chooses to drop extension member objects, they are automatically dissociated from the extension.

If an extension has secondary control files, the control parameters that are used for an update script are those associated with the script's target (new) version.

ALTER EXTENSION is able to execute sequences of update script files to achieve a requested update. For example, if only foo--1.0--1.1.sql and foo--1.1--2.0.sql are available, ALTER EXTENSION will apply them in sequence if an update to version 2.0 is requested when 1.0 is currently installed.

PostgreSQL doesn't assume anything about the properties of version names: for example, it does not know whether 1.1 follows 1.0. It just matches up the available version names and follows the path that requires applying the fewest update scripts. (A version name can actually be any string that doesn't contain -- or leading or trailing -.)

Sometimes it is useful to provide “downgrade” scripts, for example foo--1.1--1.0.sql to allow reverting the changes associated with version 1.1. If you do that, be careful of the possibility that a downgrade script might unexpectedly get applied because it yields a shorter path. The risky case is where there is a “fast path” update script that jumps ahead several versions as well as a downgrade script to the fast path's start point. It might take fewer steps to apply the downgrade and then the fast path than to move ahead one version at a time. If the downgrade script drops any irreplaceable objects, this will yield undesirable results.

To check for unexpected update paths, use this command:

SELECT * FROM pg_extension_update_paths('extension_name');

This shows each pair of distinct known version names for the specified extension, together with the update path sequence that would be taken to get from the source version to the target version, or NULL if there is no available update path. The path is shown in textual form with -- separators. You can use regexp_split_to_array(path,'--') if you prefer an array format.

38.17.5. Installing Extensions Using Update Scripts

An extension that has been around for awhile will probably exist in several versions, for which the author will need to write update scripts. For example, if you have released a foo extension in versions 1.0, 1.1, and 1.2, there should be update scripts foo--1.0--1.1.sql and foo--1.1--1.2.sql. Before PostgreSQL 10, it was necessary to also create new script files foo--1.1.sql and foo--1.2.sql that directly build the newer extension versions, or else the newer versions could not be installed directly, only by installing 1.0 and then updating. That was tedious and duplicative, but now it's unnecessary, because CREATE EXTENSION can follow update chains automatically. For example, if only the script files foo--1.0.sql, foo--1.0--1.1.sql, and foo--1.1--1.2.sql are available then a request to install version 1.2 is honored by running those three scripts in sequence. The processing is the same as if you'd first installed 1.0 and then updated to 1.2. (As with ALTER EXTENSION UPDATE, if multiple pathways are available then the shortest is preferred.) Arranging an extension's script files in this style can reduce the amount of maintenance effort needed to produce small updates.

If you use secondary (version-specific) control files with an extension maintained in this style, keep in mind that each version needs a control file even if it has no stand-alone installation script, as that control file will determine how the implicit update to that version is performed. For example, if foo--1.0.control specifies requires = 'bar' but foo's other control files do not, the extension's dependency on bar will be dropped when updating from 1.0 to another version.

38.17.6. Security Considerations for Extensions

Widely-distributed extensions should assume little about the database they occupy. Therefore, it's appropriate to write functions provided by an extension in a secure style that cannot be compromised by search-path-based attacks.

An extension that has the superuser property set to true must also consider security hazards for the actions taken within its installation and update scripts. It is not terribly difficult for a malicious user to create trojan-horse objects that will compromise later execution of a carelessly-written extension script, allowing that user to acquire superuser privileges.

If an extension is marked trusted, then its installation schema can be selected by the installing user, who might intentionally use an insecure schema in hopes of gaining superuser privileges. Therefore, a trusted extension is extremely exposed from a security standpoint, and all its script commands must be carefully examined to ensure that no compromise is possible.

Advice about writing functions securely is provided in Section 38.17.6.1 below, and advice about writing installation scripts securely is provided in Section 38.17.6.2.

38.17.6.1. Security Considerations for Extension Functions

SQL-language and PL-language functions provided by extensions are at risk of search-path-based attacks when they are executed, since parsing of these functions occurs at execution time not creation time.

The CREATE FUNCTION reference page contains advice about writing SECURITY DEFINER functions safely. It's good practice to apply those techniques for any function provided by an extension, since the function might be called by a high-privilege user.

If you cannot set the search_path to contain only secure schemas, assume that each unqualified name could resolve to an object that a malicious user has defined. Beware of constructs that depend on search_path implicitly; for example, IN and CASE expression WHEN always select an operator using the search path. In their place, use OPERATOR(schema.=) ANY and CASE WHEN expression.

A general-purpose extension usually should not assume that it's been installed into a secure schema, which means that even schema-qualified references to its own objects are not entirely risk-free. For example, if the extension has defined a function myschema.myfunc(bigint) then a call such as myschema.myfunc(42) could be captured by a hostile function myschema.myfunc(integer). Be careful that the data types of function and operator parameters exactly match the declared argument types, using explicit casts where necessary.

38.17.6.2. Security Considerations for Extension Scripts

An extension installation or update script should be written to guard against search-path-based attacks occurring when the script executes. If an object reference in the script can be made to resolve to some other object than the script author intended, then a compromise might occur immediately, or later when the mis-defined extension object is used.

DDL commands such as CREATE FUNCTION and CREATE OPERATOR CLASS are generally secure, but beware of any command having a general-purpose expression as a component. For example, CREATE VIEW needs to be vetted, as does a DEFAULT expression in CREATE FUNCTION.

Sometimes an extension script might need to execute general-purpose SQL, for example to make catalog adjustments that aren't possible via DDL. Be careful to execute such commands with a secure search_path; do not trust the path provided by CREATE/ALTER EXTENSION to be secure. Best practice is to temporarily set search_path to 'pg_catalog, pg_temp' and insert references to the extension's installation schema explicitly where needed. (This practice might also be helpful for creating views.) Examples can be found in the contrib modules in the PostgreSQL source code distribution.

Cross-extension references are extremely difficult to make fully secure, partially because of uncertainty about which schema the other extension is in. The hazards are reduced if both extensions are installed in the same schema, because then a hostile object cannot be placed ahead of the referenced extension in the installation-time search_path. However, no mechanism currently exists to require that. For now, best practice is to not mark an extension trusted if it depends on another one, unless that other one is always installed in pg_catalog.

Do not use CREATE OR REPLACE FUNCTION, except in an update script that must change the definition of a function that is known to be an extension member already. (Likewise for other OR REPLACE options.) Using OR REPLACE unnecessarily not only has a risk of accidentally overwriting someone else's function, but it creates a security hazard since the overwritten function would still be owned by its original owner, who could modify it.

38.17.7. Extension Example

Here is a complete example of an SQL-only extension, a two-element composite type that can store any type of value in its slots, which are named “k” and “v”. Non-text values are automatically coerced to text for storage.

The script file pair--1.0.sql looks like this:

-- complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "CREATE EXTENSION pair" to load this file. \quit

CREATE TYPE pair AS ( k text, v text );

CREATE FUNCTION pair(text, text)
RETURNS pair LANGUAGE SQL AS 'SELECT ROW($1, $2)::@extschema@.pair;';

CREATE OPERATOR ~> (LEFTARG = text, RIGHTARG = text, FUNCTION = pair);

-- "SET search_path" is easy to get right, but qualified names perform better.
CREATE FUNCTION lower(pair)
RETURNS pair LANGUAGE SQL
AS 'SELECT ROW(lower($1.k), lower($1.v))::@extschema@.pair;'
SET search_path = pg_temp;

CREATE FUNCTION pair_concat(pair, pair)
RETURNS pair LANGUAGE SQL
AS 'SELECT ROW($1.k OPERATOR(pg_catalog.||) $2.k,
               $1.v OPERATOR(pg_catalog.||) $2.v)::@extschema@.pair;';

The control file pair.control looks like this:

# pair extension
comment = 'A key/value pair data type'
default_version = '1.0'
# cannot be relocatable because of use of @extschema@
relocatable = false

While you hardly need a makefile to install these two files into the correct directory, you could use a Makefile containing this:

EXTENSION = pair
DATA = pair--1.0.sql

PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

This makefile relies on PGXS, which is described in Section 38.18. The command make install will install the control and script files into the correct directory as reported by pg_config.

Once the files are installed, use the CREATE EXTENSION command to load the objects into any particular database.

Prev | Up | Next
38.16. Interfacing Extensions to Indexes | Home | 38.18. Extension Building Infrastructure


38.18. Extension Building Infrastructure
Prev Up Chapter 38. Extending SQL Home Next

38.18. Extension Building Infrastructure

If you are thinking about distributing your PostgreSQL extension modules, setting up a portable build system for them can be fairly difficult. Therefore the PostgreSQL installation provides a build infrastructure for extensions, called PGXS, so that simple extension modules can be built simply against an already installed server. PGXS is mainly intended for extensions that include C code, although it can be used for pure-SQL extensions too. Note that PGXS is not intended to be a universal build system framework that can be used to build any software interfacing to PostgreSQL; it simply automates common build rules for simple server extension modules. For more complicated packages, you might need to write your own build system.

To use the PGXS infrastructure for your extension, you must write a simple makefile. In the makefile, you need to set some variables and include the global PGXS makefile. Here is an example that builds an extension module named isbn_issn, consisting of a shared library containing some C code, an extension control file, an SQL script, an include file (only needed if other modules might need to access the extension functions without going via SQL), and a documentation text file:

MODULES = isbn_issn
EXTENSION = isbn_issn
DATA = isbn_issn--1.0.sql
DOCS = README.isbn_issn
HEADERS_isbn_issn = isbn_issn.h

PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

The last three lines should always be the same. Earlier in the file, you assign variables or add custom make rules.

Set one of these three variables to specify what is built:

MODULES

list of shared-library objects to be built from source files with same stem (do not include library suffixes in this list)

MODULE_big

a shared library to build from multiple source files (list object files in OBJS)

PROGRAM

an executable program to build (list object files in OBJS)

The following variables can also be set:

EXTENSION

extension name(s); for each name you must provide an extension.control file, which will be installed into prefix/share/extension

MODULEDIR

subdirectory of prefix/share into which DATA and DOCS files should be installed (if not set, default is extension if EXTENSION is set, or contrib if not)

DATA

random files to install into prefix/share/$MODULEDIR

DATA_built

random files to install into prefix/share/$MODULEDIR, which need to be built first

DATA_TSEARCH

random files to install under prefix/share/tsearch_data

DOCS

random files to install under prefix/doc/$MODULEDIR

HEADERS
HEADERS_built

Files to (optionally build and) install under prefix/include/server/$MODULEDIR/$MODULE_big.

Unlike DATA_built, files in HEADERS_built are not removed by the clean target; if you want them removed, also add them to EXTRA_CLEAN or add your own rules to do it.

HEADERS_$MODULE
HEADERS_built_$MODULE

Files to install (after building if specified) under prefix/include/server/$MODULEDIR/$MODULE, where $MODULE must be a module name used in MODULES or MODULE_big.

Unlike DATA_built, files in HEADERS_built_$MODULE are not removed by the clean target; if you want them removed, also add them to EXTRA_CLEAN or add your own rules to do it.

It is legal to use both variables for the same module, or any combination, unless you have two module names in the MODULES list that differ only by the presence of a prefix built_, which would cause ambiguity. In that (hopefully unlikely) case, you should use only the HEADERS_built_$MODULE variables.

SCRIPTS

script files (not binaries) to install into prefix/bin

SCRIPTS_built

script files (not binaries) to install into prefix/bin, which need to be built first

REGRESS

list of regression test cases (without suffix), see below

REGRESS_OPTS

additional switches to pass to pg_regress

ISOLATION

list of isolation test cases, see below for more details

ISOLATION_OPTS

additional switches to pass to pg_isolation_regress

TAP_TESTS

switch defining if TAP tests need to be run, see below

NO_INSTALL

don't define an install target, useful for test modules that don't need their build products to be installed

NO_INSTALLCHECK

don't define an installcheck target, useful e.g., if tests require special configuration, or don't use pg_regress

EXTRA_CLEAN

extra files to remove in make clean

PG_CPPFLAGS

will be prepended to CPPFLAGS

PG_CFLAGS

will be appended to CFLAGS

PG_CXXFLAGS

will be appended to CXXFLAGS

PG_LDFLAGS

will be prepended to LDFLAGS

PG_LIBS

will be added to PROGRAM link line

SHLIB_LINK

will be added to MODULE_big link line

PG_CONFIG

path to pg_config program for the PostgreSQL installation to build against (typically just pg_config to use the first one in your PATH)

Put this makefile as Makefile in the directory which holds your extension. Then you can do make to compile, and then make install to install your module. By default, the extension is compiled and installed for the PostgreSQL installation that corresponds to the first pg_config program found in your PATH. You can use a different installation by setting PG_CONFIG to point to its pg_config program, either within the makefile or on the make command line.

You can also run make in a directory outside the source tree of your extension, if you want to keep the build directory separate. This procedure is also called a VPATH build. Here's how:

mkdir build_dir
cd build_dir
make -f /path/to/extension/source/tree/Makefile
make -f /path/to/extension/source/tree/Makefile install

Alternatively, you can set up a directory for a VPATH build in a similar way to how it is done for the core code. One way to do this is using the core script config/prep_buildtree. Once this has been done you can build by setting the make variable VPATH like this:

make VPATH=/path/to/extension/source/tree
make VPATH=/path/to/extension/source/tree install

This procedure can work with a greater variety of directory layouts.

The scripts listed in the REGRESS variable are used for regression testing of your module, which can be invoked by make installcheck after doing make install. For this to work you must have a running PostgreSQL server. The script files listed in REGRESS must appear in a subdirectory named sql/ in your extension's directory. These files must have extension .sql, which must not be included in the REGRESS list in the makefile. For each test there should also be a file containing the expected output in a subdirectory named expected/, with the same stem and extension .out. make installcheck executes each test script with psql, and compares the resulting output to the matching expected file. Any differences will be written to the file regression.diffs in diff -c format. Note that trying to run a test that is missing its expected file will be reported as “trouble”, so make sure you have all expected files.

The scripts listed in the ISOLATION variable are used for tests stressing behavior of concurrent session with your module, which can be invoked by make installcheck after doing make install. For this to work you must have a running PostgreSQL server. The script files listed in ISOLATION must appear in a subdirectory named specs/ in your extension's directory. These files must have extension .spec, which must not be included in the ISOLATION list in the makefile. For each test there should also be a file containing the expected output in a subdirectory named expected/, with the same stem and extension .out. make installcheck executes each test script, and compares the resulting output to the matching expected file. Any differences will be written to the file output_iso/regression.diffs in diff -c format. Note that trying to run a test that is missing its expected file will be reported as “trouble”, so make sure you have all expected files.

TAP_TESTS enables the use of TAP tests. Data from each run is present in a subdirectory named tmp_check/. See also Section 33.4 for more details.

Tip

The easiest way to create the expected files is to create empty files, then do a test run (which will of course report differences). Inspect the actual result files found in the results/ directory (for tests in REGRESS), or output_iso/results/ directory (for tests in ISOLATION), then copy them to expected/ if they match what you expect from the test.

Prev | Up | Next
38.17. Packaging Related Objects into an Extension | Home | Chapter 39. Triggers


Chapter 40. Event Triggers
Prev Up Part V. Server Programming Home Next

Chapter 40. Event Triggers

Table of Contents

40.1. Overview of Event Trigger Behavior
40.2. Event Trigger Firing Matrix
40.3. Writing Event Trigger Functions in C
40.4. A Complete Event Trigger Example
40.5. A Table Rewrite Event Trigger Example

To supplement the trigger mechanism discussed in Chapter 39, PostgreSQL also provides event triggers. Unlike regular triggers, which are attached to a single table and capture only DML events, event triggers are global to a particular database and are capable of capturing DDL events.

Like regular triggers, event triggers can be written in any procedural language that includes event trigger support, or in C, but not in plain SQL.

Prev | Up | Next
39.4. A Complete Trigger Example | Home | 40.1. Overview of Event Trigger Behavior


40.1. Overview of Event Trigger Behavior
Prev Up Chapter 40. Event Triggers Home Next

40.1. Overview of Event Trigger Behavior

An event trigger fires whenever the event with which it is associated occurs in the database in which it is defined. Currently, the only supported events are ddl_command_start, ddl_command_end, table_rewrite and sql_drop. Support for additional events may be added in future releases.

The ddl_command_start event occurs just before the execution of a CREATE, ALTER, DROP, SECURITY LABEL, COMMENT, GRANT or REVOKE command. No check whether the affected object exists or doesn't exist is performed before the event trigger fires. As an exception, however, this event does not occur for DDL commands targeting shared objects — databases, roles, and tablespaces — or for commands targeting event triggers themselves. The event trigger mechanism does not support these object types. ddl_command_start also occurs just before the execution of a SELECT INTO command, since this is equivalent to CREATE TABLE AS.

The ddl_command_end event occurs just after the execution of this same set of commands. To obtain more details on the DDL operations that took place, use the set-returning function pg_event_trigger_ddl_commands() from the ddl_command_end event trigger code (see Section 9.29). Note that the trigger fires after the actions have taken place (but before the transaction commits), and thus the system catalogs can be read as already changed.

The sql_drop event occurs just before the ddl_command_end event trigger for any operation that drops database objects. To list the objects that have been dropped, use the set-returning function pg_event_trigger_dropped_objects() from the sql_drop event trigger code (see Section 9.29). Note that the trigger is executed after the objects have been deleted from the system catalogs, so it's not possible to look them up anymore.

The table_rewrite event occurs just before a table is rewritten by some actions of the commands ALTER TABLE and ALTER TYPE. While other control statements are available to rewrite a table, like CLUSTER and VACUUM, the table_rewrite event is not triggered by them.

Event triggers (like other functions) cannot be executed in an aborted transaction. Thus, if a DDL command fails with an error, any associated ddl_command_end triggers will not be executed. Conversely, if a ddl_command_start trigger fails with an error, no further event triggers will fire, and no attempt will be made to execute the command itself. Similarly, if a ddl_command_end trigger fails with an error, the effects of the DDL statement will be rolled back, just as they would be in any other case where the containing transaction aborts.

For a complete list of commands supported by the event trigger mechanism, see Section 40.2.

Event triggers are created using the command CREATE EVENT TRIGGER. In order to create an event trigger, you must first create a function with the special return type event_trigger. This function need not (and may not) return a value; the return type serves merely as a signal that the function is to be invoked as an event trigger.

If more than one event trigger is defined for a particular event, they will fire in alphabetical order by trigger name.

A trigger definition can also specify a WHEN condition so that, for example, a ddl_command_start trigger can be fired only for particular commands which the user wishes to intercept. A common use of such triggers is to restrict the range of DDL operations which users may perform.

Prev | Up | Next
Chapter 40. Event Triggers | Home | 40.2. Event Trigger Firing Matrix


40.2. Event Trigger Firing Matrix
Prev Up Chapter 40. Event Triggers Home Next

40.2. Event Trigger Firing Matrix

Table 40.1 lists all commands for which event triggers are supported.

Table 40.1. Event Trigger Support by Command Tag

Command Tag | ddl_​command_​start | ddl_​command_​end | sql_​drop | table_​rewrite | Notes
ALTER AGGREGATE | X | X | - | - |
ALTER COLLATION | X | X | - | - |
ALTER CONVERSION | X | X | - | - |
ALTER DOMAIN | X | X | - | - |
ALTER DEFAULT PRIVILEGES | X | X | - | - |
ALTER EXTENSION | X | X | - | - |
ALTER FOREIGN DATA WRAPPER | X | X | - | - |
ALTER FOREIGN TABLE | X | X | X | - |
ALTER FUNCTION | X | X | - | - |
ALTER LANGUAGE | X | X | - | - |
ALTER LARGE OBJECT | X | X | - | - |
ALTER MATERIALIZED VIEW | X | X | - | - |
ALTER OPERATOR | X | X | - | - |
ALTER OPERATOR CLASS | X | X | - | - |
ALTER OPERATOR FAMILY | X | X | - | - |
ALTER POLICY | X | X | - | - |
ALTER PROCEDURE | X | X | - | - |
ALTER PUBLICATION | X | X | - | - |
ALTER ROUTINE | X | X | - | - |
ALTER SCHEMA | X | X | - | - |
ALTER SEQUENCE | X | X | - | - |
ALTER SERVER | X | X | - | - |
ALTER STATISTICS | X | X | - | - |
ALTER SUBSCRIPTION | X | X | - | - |
ALTER TABLE | X | X | X | X |
ALTER TEXT SEARCH CONFIGURATION | X | X | - | - |
ALTER TEXT SEARCH DICTIONARY | X | X | - | - |
ALTER TEXT SEARCH PARSER | X | X | - | - |
ALTER TEXT SEARCH TEMPLATE | X | X | - | - |
ALTER TRIGGER | X | X | - | - |
ALTER TYPE | X | X | - | X |
ALTER USER MAPPING | X | X | - | - |
ALTER VIEW | X | X | - | - |
COMMENT | X | X | - | - | Only for local objects
CREATE ACCESS METHOD | X | X | - | - |
CREATE AGGREGATE | X | X | - | - |
CREATE CAST | X | X | - | - |
CREATE COLLATION | X | X | - | - |
CREATE CONVERSION | X | X | - | - |
CREATE DOMAIN | X | X | - | - |
CREATE EXTENSION | X | X | - | - |
CREATE FOREIGN DATA WRAPPER | X | X | - | - |
CREATE FOREIGN TABLE | X | X | - | - |
CREATE FUNCTION | X | X | - | - |
CREATE INDEX | X | X | - | - |
CREATE LANGUAGE | X | X | - | - |
CREATE MATERIALIZED VIEW | X | X | - | - |
CREATE OPERATOR | X | X | - | - |
CREATE OPERATOR CLASS | X | X | - | - |
CREATE OPERATOR FAMILY | X | X | - | - |
CREATE POLICY | X | X | - | - |
CREATE PROCEDURE | X | X | - | - |
CREATE PUBLICATION | X | X | - | - |
CREATE RULE | X | X | - | - |
CREATE SCHEMA | X | X | - | - |
CREATE SEQUENCE | X | X | - | - |
CREATE SERVER | X | X | - | - |
CREATE STATISTICS | X | X | - | - |
CREATE SUBSCRIPTION | X | X | - | - |
CREATE TABLE | X | X | - | - |
CREATE TABLE AS | X | X | - | - |
CREATE TEXT SEARCH CONFIGURATION | X | X | - | - |
CREATE TEXT SEARCH DICTIONARY | X | X | - | - |
CREATE TEXT SEARCH PARSER | X | X | - | - |
CREATE TEXT SEARCH TEMPLATE | X | X | - | - |
CREATE TRIGGER | X | X | - | - |
CREATE TYPE | X | X | - | - |
CREATE USER MAPPING | X | X | - | - |
CREATE VIEW | X | X | - | - |
DROP ACCESS METHOD | X | X | X | - |
DROP AGGREGATE | X | X | X | - |
DROP CAST | X | X | X | - |
DROP COLLATION | X | X | X | - |
DROP CONVERSION | X | X | X | - |
DROP DOMAIN | X | X | X | - |
DROP EXTENSION | X | X | X | - |
DROP FOREIGN DATA WRAPPER | X | X | X | - |
DROP FOREIGN TABLE | X | X | X | - |
DROP FUNCTION | X | X | X | - |
DROP INDEX | X | X | X | - |
DROP LANGUAGE | X | X | X | - |
DROP MATERIALIZED VIEW | X | X | X | - |
DROP OPERATOR | X | X | X | - |
DROP OPERATOR CLASS | X | X | X | - |
DROP OPERATOR FAMILY | X | X | X | - |
DROP OWNED | X | X | X | - |
DROP POLICY | X | X | X | - |
DROP PROCEDURE | X | X | X | - |
DROP PUBLICATION | X | X | X | - |
DROP ROUTINE | X | X | X | - |
DROP RULE | X | X | X | - |
DROP SCHEMA | X | X | X | - |
DROP SEQUENCE | X | X | X | - |
DROP SERVER | X | X | X | - |
DROP STATISTICS | X | X | X | - |
DROP SUBSCRIPTION | X | X | X | - |
DROP TABLE | X | X | X | - |
DROP TEXT SEARCH CONFIGURATION | X | X | X | - |
DROP TEXT SEARCH DICTIONARY | X | X | X | - |
DROP TEXT SEARCH PARSER | X | X | X | - |
DROP TEXT SEARCH TEMPLATE | X | X | X | - |
DROP TRIGGER | X | X | X | - |
DROP TYPE | X | X | X | - |
DROP USER MAPPING | X | X | X | - |
DROP VIEW | X | X | X | - |
GRANT | X | X | - | - | Only for local objects
IMPORT FOREIGN SCHEMA | X | X | - | - |
REFRESH MATERIALIZED VIEW | X | X | - | - |
REVOKE | X | X | - | - | Only for local objects
SECURITY LABEL | X | X | - | - | Only for local objects
SELECT INTO | X | X | - | - |

Prev | Up | Next
40.1. Overview of Event Trigger Behavior | Home | 40.3. Writing Event Trigger Functions in C


40.3. Writing Event Trigger Functions in C
Prev Up Chapter 40. Event Triggers Home Next

40.3. Writing Event Trigger Functions in C

This section describes the low-level details of the interface to an event trigger function. This information is only needed when writing event trigger functions in C. If you are using a higher-level language then these details are handled for you. In most cases you should consider using a procedural language before writing your event triggers in C. The documentation of each procedural language explains how to write an event trigger in that language.

Event trigger functions must use the “version 1” function manager interface.

When a function is called by the event trigger manager, it is not passed any normal arguments, but it is passed a “context” pointer pointing to a EventTriggerData structure. C functions can check whether they were called from the event trigger manager or not by executing the macro:

CALLED_AS_EVENT_TRIGGER(fcinfo)

which expands to:

((fcinfo)->context != NULL && IsA((fcinfo)->context, EventTriggerData))

If this returns true, then it is safe to cast fcinfo->context to type EventTriggerData * and make use of the pointed-to EventTriggerData structure. The function must not alter the EventTriggerData structure or any of the data it points to.

struct EventTriggerData is defined in commands/event_trigger.h:

typedef struct EventTriggerData
{
    NodeTag     type;
    const char *event;      /* event name */
    Node       *parsetree;  /* parse tree */
    CommandTag  tag;        /* command tag */
} EventTriggerData;

where the members are defined as follows:

type

Always T_EventTriggerData.

event

Describes the event for which the function is called, one of "ddl_command_start", "ddl_command_end", "sql_drop", "table_rewrite". See Section 40.1 for the meaning of these events.

parsetree

A pointer to the parse tree of the command. Check the PostgreSQL source code for details. The parse tree structure is subject to change without notice.

tag

The command tag associated with the event for which the event trigger is run, for example "CREATE FUNCTION".

An event trigger function must return a NULL pointer (not an SQL null value, that is, do not set isNull true).

Prev | Up | Next
40.2. Event Trigger Firing Matrix | Home | 40.4. A Complete Event Trigger Example


40.4. A Complete Event Trigger Example
Prev Up Chapter 40. Event Triggers Home Next

40.4. A Complete Event Trigger Example

Here is a very simple example of an event trigger function written in C. (Examples of triggers written in procedural languages can be found in the documentation of the procedural languages.)

The function noddl raises an exception each time it is called. The event trigger definition associated the function with the ddl_command_start event. The effect is that all DDL commands (with the exceptions mentioned in Section 40.1) are prevented from running.

This is the source code of the trigger function:

#include "postgres.h"
#include "commands/event_trigger.h"

PG_MODULE_MAGIC;

PG_FUNCTION_INFO_V1(noddl);

Datum
noddl(PG_FUNCTION_ARGS)
{
    EventTriggerData *trigdata;

    if (!CALLED_AS_EVENT_TRIGGER(fcinfo))  /* internal error */
        elog(ERROR, "not fired by event trigger manager");

    trigdata = (EventTriggerData *) fcinfo->context;

    ereport(ERROR,
        (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
                 errmsg("command \"%s\" denied", trigdata->tag)));

    PG_RETURN_NULL();
}

After you have compiled the source code (see Section 38.10.5), declare the function and the triggers:

CREATE FUNCTION noddl() RETURNS event_trigger
    AS 'noddl' LANGUAGE C;

CREATE EVENT TRIGGER noddl ON ddl_command_start
    EXECUTE FUNCTION noddl();

Now you can test the operation of the trigger:

=# \dy
                     List of event triggers
 Name  |       Event       | Owner | Enabled | Function | Tags
-------+-------------------+-------+---------+----------+------
 noddl | ddl_command_start | dim   | enabled | noddl    |
(1 row)

=# CREATE TABLE foo(id serial);
ERROR:  command "CREATE TABLE" denied

In this situation, in order to be able to run some DDL commands when you need to do so, you have to either drop the event trigger or disable it. It can be convenient to disable the trigger for only the duration of a transaction:

BEGIN;
ALTER EVENT TRIGGER noddl DISABLE;
CREATE TABLE foo (id serial);
ALTER EVENT TRIGGER noddl ENABLE;
COMMIT;

(Recall that DDL commands on event triggers themselves are not affected by event triggers.)

Prev | Up | Next
40.3. Writing Event Trigger Functions in C | Home | 40.5. A Table Rewrite Event Trigger Example


40.5. A Table Rewrite Event Trigger Example
Prev Up Chapter 40. Event Triggers Home Next

40.5. A Table Rewrite Event Trigger Example

Thanks to the table_rewrite event, it is possible to implement a table rewriting policy only allowing the rewrite in maintenance windows.

Here's an example implementing such a policy.

CREATE OR REPLACE FUNCTION no_rewrite()
 RETURNS event_trigger
 LANGUAGE plpgsql AS
$$
---
--- Implement local Table Rewriting policy:
---   public.foo is not allowed rewriting, ever
---   other tables are only allowed rewriting between 1am and 6am
---   unless they have more than 100 blocks
---
DECLARE
  table_oid oid := pg_event_trigger_table_rewrite_oid();
  current_hour integer := extract('hour' from current_time);
  pages integer;
  max_pages integer := 100;
BEGIN
  IF pg_event_trigger_table_rewrite_oid() = 'public.foo'::regclass
  THEN
        RAISE EXCEPTION 'you''re not allowed to rewrite the table %',
                        table_oid::regclass;
  END IF;

  SELECT INTO pages relpages FROM pg_class WHERE oid = table_oid;
  IF pages > max_pages
  THEN
        RAISE EXCEPTION 'rewrites only allowed for table with less than % pages',
                        max_pages;
  END IF;

  IF current_hour NOT BETWEEN 1 AND 6
  THEN
        RAISE EXCEPTION 'rewrites only allowed between 1am and 6am';
  END IF;
END;
$$;

CREATE EVENT TRIGGER no_rewrite_allowed
                  ON table_rewrite
   EXECUTE FUNCTION no_rewrite();

Prev | Up | Next
40.4. A Complete Event Trigger Example | Home | Chapter 41. The Rule System


41.1. The Query Tree
Prev Up Chapter 41. The Rule System Home Next

41.1. The Query Tree

To understand how the rule system works it is necessary to know when it is invoked and what its input and results are.

The rule system is located between the parser and the planner. It takes the output of the parser, one query tree, and the user-defined rewrite rules, which are also query trees with some extra information, and creates zero or more query trees as result. So its input and output are always things the parser itself could have produced and thus, anything it sees is basically representable as an SQL statement.

Now what is a query tree? It is an internal representation of an SQL statement where the single parts that it is built from are stored separately. These query trees can be shown in the server log if you set the configuration parameters debug_print_parse, debug_print_rewritten, or debug_print_plan. The rule actions are also stored as query trees, in the system catalog pg_rewrite. They are not formatted like the log output, but they contain exactly the same information.

Reading a raw query tree requires some experience. But since SQL representations of query trees are sufficient to understand the rule system, this chapter will not teach how to read them.

When reading the SQL representations of the query trees in this chapter it is necessary to be able to identify the parts the statement is broken into when it is in the query tree structure. The parts of a query tree are

the command type

This is a simple value telling which command (SELECT, INSERT, UPDATE, DELETE) produced the query tree.

the range table

The range table is a list of relations that are used in the query. In a SELECT statement these are the relations given after the FROM key word.

Every range table entry identifies a table or view and tells by which name it is called in the other parts of the query. In the query tree, the range table entries are referenced by number rather than by name, so here it doesn't matter if there are duplicate names as it would in an SQL statement. This can happen after the range tables of rules have been merged in. The examples in this chapter will not have this situation.

the result relation

This is an index into the range table that identifies the relation where the results of the query go.

SELECT queries don't have a result relation. (The special case of SELECT INTO is mostly identical to CREATE TABLE followed by INSERT ... SELECT, and is not discussed separately here.)

For INSERT, UPDATE, and DELETE commands, the result relation is the table (or view!) where the changes are to take effect.

the target list

The target list is a list of expressions that define the result of the query. In the case of a SELECT, these expressions are the ones that build the final output of the query. They correspond to the expressions between the key words SELECT and FROM. (* is just an abbreviation for all the column names of a relation. It is expanded by the parser into the individual columns, so the rule system never sees it.)

DELETE commands don't need a normal target list because they don't produce any result. Instead, the planner adds a special CTID entry to the empty target list, to allow the executor to find the row to be deleted. (CTID is added when the result relation is an ordinary table. If it is a view, a whole-row variable is added instead, by the rule system, as described in Section 41.2.4.)

For INSERT commands, the target list describes the new rows that should go into the result relation. It consists of the expressions in the VALUES clause or the ones from the SELECT clause in INSERT ... SELECT. The first step of the rewrite process adds target list entries for any columns that were not assigned to by the original command but have defaults. Any remaining columns (with neither a given value nor a default) will be filled in by the planner with a constant null expression.

For UPDATE commands, the target list describes the new rows that should replace the old ones. In the rule system, it contains just the expressions from the SET column = expression part of the command. The planner will handle missing columns by inserting expressions that copy the values from the old row into the new one. Just as for DELETE, a CTID or whole-row variable is added so that the executor can identify the old row to be updated.

Every entry in the target list contains an expression that can be a constant value, a variable pointing to a column of one of the relations in the range table, a parameter, or an expression tree made of function calls, constants, variables, operators, etc.

the qualification

The query's qualification is an expression much like one of those contained in the target list entries. The result value of this expression is a Boolean that tells whether the operation (INSERT, UPDATE, DELETE, or SELECT) for the final result row should be executed or not. It corresponds to the WHERE clause of an SQL statement.

the join tree

The query's join tree shows the structure of the FROM clause. For a simple query like SELECT ... FROM a, b, c, the join tree is just a list of the FROM items, because we are allowed to join them in any order. But when JOIN expressions, particularly outer joins, are used, we have to join in the order shown by the joins. In that case, the join tree shows the structure of the JOIN expressions. The restrictions associated with particular JOIN clauses (from ON or USING expressions) are stored as qualification expressions attached to those join-tree nodes. It turns out to be convenient to store the top-level WHERE expression as a qualification attached to the top-level join-tree item, too. So really the join tree represents both the FROM and WHERE clauses of a SELECT.

the others

The other parts of the query tree like the ORDER BY clause aren't of interest here. The rule system substitutes some entries there while applying rules, but that doesn't have much to do with the fundamentals of the rule system.

Prev | Up | Next
Chapter 41. The Rule System | Home | 41.2. Views and the Rule System


Chapter 43. PL/pgSQL — SQL Procedural Language
Prev Up Part V. Server Programming Home Next

Chapter 43. PL/pgSQL — SQL Procedural Language

Table of Contents

43.1. Overview
43.1.1. Advantages of Using PL/pgSQL
43.1.2. Supported Argument and Result Data Types
43.2. Structure of PL/pgSQL
43.3. Declarations
43.3.1. Declaring Function Parameters
43.3.2. ALIAS
43.3.3. Copying Types
43.3.4. Row Types
43.3.5. Record Types
43.3.6. Collation of PL/pgSQL Variables
43.4. Expressions
43.5. Basic Statements
43.5.1. Assignment
43.5.2. Executing SQL Commands
43.5.3. Executing a Command with a Single-Row Result
43.5.4. Executing Dynamic Commands
43.5.5. Obtaining the Result Status
43.5.6. Doing Nothing At All
43.6. Control Structures
43.6.1. Returning from a Function
43.6.2. Returning from a Procedure
43.6.3. Calling a Procedure
43.6.4. Conditionals
43.6.5. Simple Loops
43.6.6. Looping through Query Results
43.6.7. Looping through Arrays
43.6.8. Trapping Errors
43.6.9. Obtaining Execution Location Information
43.7. Cursors
43.7.1. Declaring Cursor Variables
43.7.2. Opening Cursors
43.7.3. Using Cursors
43.7.4. Looping through a Cursor's Result
43.8. Transaction Management
43.9. Errors and Messages
43.9.1. Reporting Errors and Messages
43.9.2. Checking Assertions
43.10. Trigger Functions
43.10.1. Triggers on Data Changes
43.10.2. Triggers on Events
43.11. PL/pgSQL under the Hood
43.11.1. Variable Substitution
43.11.2. Plan Caching
43.12. Tips for Developing in PL/pgSQL
43.12.1. Handling of Quotation Marks
43.12.2. Additional Compile-Time and Run-Time Checks
43.13. Porting from Oracle PL/SQL
43.13.1. Porting Examples
43.13.2. Other Things to Watch For
43.13.3. Appendix
Prev | Up | Next
42.1. Installing Procedural Languages | Home | 43.1. Overview


43.1. Overview
Prev Up Chapter 43. PL/pgSQL — SQL Procedural Language Home Next

43.1. Overview

43.1.1. Advantages of Using PL/pgSQL
43.1.2. Supported Argument and Result Data Types

PL/pgSQL is a loadable procedural language for the PostgreSQL database system. The design goals of PL/pgSQL were to create a loadable procedural language that

  • can be used to create functions, procedures, and triggers,

  • adds control structures to the SQL language,

  • can perform complex computations,

  • inherits all user-defined types, functions, procedures, and operators,

  • can be defined to be trusted by the server,

  • is easy to use.

Functions created with PL/pgSQL can be used anywhere that built-in functions could be used. For example, it is possible to create complex conditional computation functions and later use them to define operators or use them in index expressions.

In PostgreSQL 9.0 and later, PL/pgSQL is installed by default. However it is still a loadable module, so especially security-conscious administrators could choose to remove it.

43.1.1. Advantages of Using PL/pgSQL

SQL is the language PostgreSQL and most other relational databases use as query language. It's portable and easy to learn. But every SQL statement must be executed individually by the database server.

That means that your client application must send each query to the database server, wait for it to be processed, receive and process the results, do some computation, then send further queries to the server. All this incurs interprocess communication and will also incur network overhead if your client is on a different machine than the database server.

With PL/pgSQL you can group a block of computation and a series of queries inside the database server, thus having the power of a procedural language and the ease of use of SQL, but with considerable savings of client/server communication overhead.

  • Extra round trips between client and server are eliminated

  • Intermediate results that the client does not need do not have to be marshaled or transferred between server and client

  • Multiple rounds of query parsing can be avoided

This can result in a considerable performance increase as compared to an application that does not use stored functions.

Also, with PL/pgSQL you can use all the data types, operators and functions of SQL.

43.1.2. Supported Argument and Result Data Types

Functions written in PL/pgSQL can accept as arguments any scalar or array data type supported by the server, and they can return a result of any of these types. They can also accept or return any composite type (row type) specified by name. It is also possible to declare a PL/pgSQL function as accepting record, which means that any composite type will do as input, or as returning record, which means that the result is a row type whose columns are determined by specification in the calling query, as discussed in Section 7.2.1.4.

PL/pgSQL functions can be declared to accept a variable number of arguments by using the VARIADIC marker. This works exactly the same way as for SQL functions, as discussed in Section 38.5.6.

PL/pgSQL functions can also be declared to accept and return the polymorphic types described in Section 38.2.5, thus allowing the actual data types handled by the function to vary from call to call. Examples appear in Section 43.3.1.

PL/pgSQL functions can also be declared to return a “set” (or table) of any data type that can be returned as a single instance. Such a function generates its output by executing RETURN NEXT for each desired element of the result set, or by using RETURN QUERY to output the result of evaluating a query.

Finally, a PL/pgSQL function can be declared to return void if it has no useful return value. (Alternatively, it could be written as a procedure in that case.)

PL/pgSQL functions can also be declared with output parameters in place of an explicit specification of the return type. This does not add any fundamental capability to the language, but it is often convenient, especially for returning multiple values. The RETURNS TABLE notation can also be used in place of RETURNS SETOF.

Specific examples appear in Section 43.3.1 and Section 43.6.1.

Prev | Up | Next
Chapter 43. PL/pgSQL — SQL Procedural Language | Home | 43.2. Structure of PL/pgSQL


43.2. Structure of PL/pgSQL
Prev Up Chapter 43. PL/pgSQL — SQL Procedural Language Home Next

43.2. Structure of PL/pgSQL

Functions written in PL/pgSQL are defined to the server by executing CREATE FUNCTION commands. Such a command would normally look like, say,

CREATE FUNCTION somefunc(integer, text) RETURNS integer
AS 'function body text'
LANGUAGE plpgsql;

The function body is simply a string literal so far as CREATE FUNCTION is concerned. It is often helpful to use dollar quoting (see Section 4.1.2.4) to write the function body, rather than the normal single quote syntax. Without dollar quoting, any single quotes or backslashes in the function body must be escaped by doubling them. Almost all the examples in this chapter use dollar-quoted literals for their function bodies.

PL/pgSQL is a block-structured language. The complete text of a function body must be a block. A block is defined as:

[ <<label>> ]
[ DECLARE
    declarations ]
BEGIN
    statements
END [ label ];

Each declaration and each statement within a block is terminated by a semicolon. A block that appears within another block must have a semicolon after END, as shown above; however the final END that concludes a function body does not require a semicolon.

Tip

A common mistake is to write a semicolon immediately after BEGIN. This is incorrect and will result in a syntax error.

A label is only needed if you want to identify the block for use in an EXIT statement, or to qualify the names of the variables declared in the block. If a label is given after END, it must match the label at the block's beginning.

All key words are case-insensitive. Identifiers are implicitly converted to lower case unless double-quoted, just as they are in ordinary SQL commands.

Comments work the same way in PL/pgSQL code as in ordinary SQL. A double dash (--) starts a comment that extends to the end of the line. A /* starts a block comment that extends to the matching occurrence of */. Block comments nest.

Any statement in the statement section of a block can be a subblock. Subblocks can be used for logical grouping or to localize variables to a small group of statements. Variables declared in a subblock mask any similarly-named variables of outer blocks for the duration of the subblock; but you can access the outer variables anyway if you qualify their names with their block's label. For example:

CREATE FUNCTION somefunc() RETURNS integer AS $$
<< outerblock >>
DECLARE
    quantity integer := 30;
BEGIN
    RAISE NOTICE 'Quantity here is %', quantity;  -- Prints 30
    quantity := 50;
    --
    -- Create a subblock
    --
    DECLARE
        quantity integer := 80;
    BEGIN
        RAISE NOTICE 'Quantity here is %', quantity;  -- Prints 80
        RAISE NOTICE 'Outer quantity here is %', outerblock.quantity;  -- Prints 50
    END;

    RAISE NOTICE 'Quantity here is %', quantity;  -- Prints 50

    RETURN quantity;
END;
$$ LANGUAGE plpgsql;

Note

There is actually a hidden “outer block” surrounding the body of any PL/pgSQL function. This block provides the declarations of the function's parameters (if any), as well as some special variables such as FOUND (see Section 43.5.5). The outer block is labeled with the function's name, meaning that parameters and special variables can be qualified with the function's name.

It is important not to confuse the use of BEGIN/END for grouping statements in PL/pgSQL with the similarly-named SQL commands for transaction control. PL/pgSQL's BEGIN/END are only for grouping; they do not start or end a transaction. See Section 43.8 for information on managing transactions in PL/pgSQL. Also, a block containing an EXCEPTION clause effectively forms a subtransaction that can be rolled back without affecting the outer transaction. For more about that see Section 43.6.8.

Prev | Up | Next
43.1. Overview | Home | 43.3. Declarations


43.3. Declarations
Prev Up Chapter 43. PL/pgSQL — SQL Procedural Language Home Next

43.3. Declarations

43.3.1. Declaring Function Parameters
43.3.2. ALIAS
43.3.3. Copying Types
43.3.4. Row Types
43.3.5. Record Types
43.3.6. Collation of PL/pgSQL Variables

All variables used in a block must be declared in the declarations section of the block. (The only exceptions are that the loop variable of a FOR loop iterating over a range of integer values is automatically declared as an integer variable, and likewise the loop variable of a FOR loop iterating over a cursor's result is automatically declared as a record variable.)

PL/pgSQL variables can have any SQL data type, such as integer, varchar, and char.

Here are some examples of variable declarations:

user_id integer;
quantity numeric(5);
url varchar;
myrow tablename%ROWTYPE;
myfield tablename.columnname%TYPE;
arow RECORD;

The general syntax of a variable declaration is:

name [ CONSTANT ] type [ COLLATE collation_name ] [ NOT NULL ] [ { DEFAULT | := | = } expression ];

The DEFAULT clause, if given, specifies the initial value assigned to the variable when the block is entered. If the DEFAULT clause is not given then the variable is initialized to the SQL null value. The CONSTANT option prevents the variable from being assigned to after initialization, so that its value will remain constant for the duration of the block. The COLLATE option specifies a collation to use for the variable (see Section 43.3.6). If NOT NULL is specified, an assignment of a null value results in a run-time error. All variables declared as NOT NULL must have a nonnull default value specified. Equal (=) can be used instead of PL/SQL-compliant :=.

A variable's default value is evaluated and assigned to the variable each time the block is entered (not just once per function call). So, for example, assigning now() to a variable of type timestamp causes the variable to have the time of the current function call, not the time when the function was precompiled.

Examples:

quantity integer DEFAULT 32;
url varchar := 'http://mysite.com';
user_id CONSTANT integer := 10;

43.3.1. Declaring Function Parameters

Parameters passed to functions are named with the identifiers $1, $2, etc. Optionally, aliases can be declared for $n parameter names for increased readability. Either the alias or the numeric identifier can then be used to refer to the parameter value.

There are two ways to create an alias. The preferred way is to give a name to the parameter in the CREATE FUNCTION command, for example:

CREATE FUNCTION sales_tax(subtotal real) RETURNS real AS $$
BEGIN
    RETURN subtotal * 0.06;
END;
$$ LANGUAGE plpgsql;

The other way is to explicitly declare an alias, using the declaration syntax

name ALIAS FOR $n;

The same example in this style looks like:

CREATE FUNCTION sales_tax(real) RETURNS real AS $$
DECLARE
    subtotal ALIAS FOR $1;
BEGIN
    RETURN subtotal * 0.06;
END;
$$ LANGUAGE plpgsql;

Note

These two examples are not perfectly equivalent. In the first case, subtotal could be referenced as sales_tax.subtotal, but in the second case it could not. (Had we attached a label to the inner block, subtotal could be qualified with that label, instead.)

Some more examples:

CREATE FUNCTION instr(varchar, integer) RETURNS integer AS $$
DECLARE
    v_string ALIAS FOR $1;
    index ALIAS FOR $2;
BEGIN
    -- some computations using v_string and index here
END;
$$ LANGUAGE plpgsql;

CREATE FUNCTION concat_selected_fields(in_t sometablename) RETURNS text AS $$
BEGIN
    RETURN in_t.f1 || in_t.f3 || in_t.f5 || in_t.f7;
END;
$$ LANGUAGE plpgsql;

When a PL/pgSQL function is declared with output parameters, the output parameters are given $n names and optional aliases in just the same way as the normal input parameters. An output parameter is effectively a variable that starts out NULL; it should be assigned to during the execution of the function. The final value of the parameter is what is returned. For instance, the sales-tax example could also be done this way:

CREATE FUNCTION sales_tax(subtotal real, OUT tax real) AS $$
BEGIN
    tax := subtotal * 0.06;
END;
$$ LANGUAGE plpgsql;

Notice that we omitted RETURNS real — we could have included it, but it would be redundant.

To call a function with OUT parameters, omit the output parameter(s) in the function call:

SELECT sales_tax(100.00);

Output parameters are most useful when returning multiple values. A trivial example is:

CREATE FUNCTION sum_n_product(x int, y int, OUT sum int, OUT prod int) AS $$
BEGIN
    sum := x + y;
    prod := x * y;
END;
$$ LANGUAGE plpgsql;

SELECT * FROM sum_n_product(2, 4);
 sum | prod
-----+------
   6 |    8

As discussed in Section 38.5.4, this effectively creates an anonymous record type for the function's results. If a RETURNS clause is given, it must say RETURNS record.

This also works with procedures, for example:

CREATE PROCEDURE sum_n_product(x int, y int, OUT sum int, OUT prod int) AS $$
BEGIN
    sum := x + y;
    prod := x * y;
END;
$$ LANGUAGE plpgsql;

In a call to a procedure, all the parameters must be specified. For output parameters, NULL may be specified when calling the procedure from plain SQL:

CALL sum_n_product(2, 4, NULL, NULL);
 sum | prod
-----+------
   6 |    8

However, when calling a procedure from PL/pgSQL, you should instead write a variable for any output parameter; the variable will receive the result of the call. See Section 43.6.3 for details.

Another way to declare a PL/pgSQL function is with RETURNS TABLE, for example:

CREATE FUNCTION extended_sales(p_itemno int)
RETURNS TABLE(quantity int, total numeric) AS $$
BEGIN
    RETURN QUERY SELECT s.quantity, s.quantity * s.price FROM sales AS s
                 WHERE s.itemno = p_itemno;
END;
$$ LANGUAGE plpgsql;

This is exactly equivalent to declaring one or more OUT parameters and specifying RETURNS SETOF sometype.

When the return type of a PL/pgSQL function is declared as a polymorphic type (see Section 38.2.5), a special parameter $0 is created. Its data type is the actual return type of the function, as deduced from the actual input types. This allows the function to access its actual return type as shown in Section 43.3.3. $0 is initialized to null and can be modified by the function, so it can be used to hold the return value if desired, though that is not required. $0 can also be given an alias. For example, this function works on any data type that has a + operator:

CREATE FUNCTION add_three_values(v1 anyelement, v2 anyelement, v3 anyelement)
RETURNS anyelement AS $$
DECLARE
    result ALIAS FOR $0;
BEGIN
    result := v1 + v2 + v3;
    RETURN result;
END;
$$ LANGUAGE plpgsql;

The same effect can be obtained by declaring one or more output parameters as polymorphic types. In this case the special $0 parameter is not used; the output parameters themselves serve the same purpose. For example:

CREATE FUNCTION add_three_values(v1 anyelement, v2 anyelement, v3 anyelement,
                                 OUT sum anyelement)
AS $$
BEGIN
    sum := v1 + v2 + v3;
END;
$$ LANGUAGE plpgsql;

In practice it might be more useful to declare a polymorphic function using the anycompatible family of types, so that automatic promotion of the input arguments to a common type will occur. For example:

CREATE FUNCTION add_three_values(v1 anycompatible, v2 anycompatible, v3 anycompatible)
RETURNS anycompatible AS $$
BEGIN
    RETURN v1 + v2 + v3;
END;
$$ LANGUAGE plpgsql;

With this example, a call such as

SELECT add_three_values(1, 2, 4.7);

will work, automatically promoting the integer inputs to numeric. The function using anyelement would require you to cast the three inputs to the same type manually.

43.3.2. ALIAS

newname ALIAS FOR oldname;

The ALIAS syntax is more general than is suggested in the previous section: you can declare an alias for any variable, not just function parameters. The main practical use for this is to assign a different name for variables with predetermined names, such as NEW or OLD within a trigger function.

Examples:

DECLARE
  prior ALIAS FOR old;
  updated ALIAS FOR new;

Since ALIAS creates two different ways to name the same object, unrestricted use can be confusing. It's best to use it only for the purpose of overriding predetermined names.

43.3.3. Copying Types

variable%TYPE

%TYPE provides the data type of a variable or table column. You can use this to declare variables that will hold database values. For example, let's say you have a column named user_id in your users table. To declare a variable with the same data type as users.user_id you write:

user_id users.user_id%TYPE;

By using %TYPE you don't need to know the data type of the structure you are referencing, and most importantly, if the data type of the referenced item changes in the future (for instance: you change the type of user_id from integer to real), you might not need to change your function definition.

%TYPE is particularly valuable in polymorphic functions, since the data types needed for internal variables can change from one call to the next. Appropriate variables can be created by applying %TYPE to the function's arguments or result placeholders.

43.3.4. Row Types

name table_name%ROWTYPE;
name composite_type_name;

A variable of a composite type is called a row variable (or row-type variable). Such a variable can hold a whole row of a SELECT or FOR query result, so long as that query's column set matches the declared type of the variable. The individual fields of the row value are accessed using the usual dot notation, for example rowvar.field.

A row variable can be declared to have the same type as the rows of an existing table or view, by using the table_name%ROWTYPE notation; or it can be declared by giving a composite type's name. (Since every table has an associated composite type of the same name, it actually does not matter in PostgreSQL whether you write %ROWTYPE or not. But the form with %ROWTYPE is more portable.)

Parameters to a function can be composite types (complete table rows). In that case, the corresponding identifier $n will be a row variable, and fields can be selected from it, for example $1.user_id.

Here is an example of using composite types. table1 and table2 are existing tables having at least the mentioned fields:

CREATE FUNCTION merge_fields(t_row table1) RETURNS text AS $$
DECLARE
    t2_row table2%ROWTYPE;
BEGIN
    SELECT * INTO t2_row FROM table2 WHERE ... ;
    RETURN t_row.f1 || t2_row.f3 || t_row.f5 || t2_row.f7;
END;
$$ LANGUAGE plpgsql;

SELECT merge_fields(t.*) FROM table1 t WHERE ... ;

43.3.5. Record Types

name RECORD;

Record variables are similar to row-type variables, but they have no predefined structure. They take on the actual row structure of the row they are assigned during a SELECT or FOR command. The substructure of a record variable can change each time it is assigned to. A consequence of this is that until a record variable is first assigned to, it has no substructure, and any attempt to access a field in it will draw a run-time error.

Note that RECORD is not a true data type, only a placeholder. One should also realize that when a PL/pgSQL function is declared to return type record, this is not quite the same concept as a record variable, even though such a function might use a record variable to hold its result. In both cases the actual row structure is unknown when the function is written, but for a function returning record the actual structure is determined when the calling query is parsed, whereas a record variable can change its row structure on-the-fly.

43.3.6. Collation of PL/pgSQL Variables

When a PL/pgSQL function has one or more parameters of collatable data types, a collation is identified for each function call depending on the collations assigned to the actual arguments, as described in Section 24.2. If a collation is successfully identified (i.e., there are no conflicts of implicit collations among the arguments) then all the collatable parameters are treated as having that collation implicitly. This will affect the behavior of collation-sensitive operations within the function. For example, consider

CREATE FUNCTION less_than(a text, b text) RETURNS boolean AS $$
BEGIN
    RETURN a < b;
END;
$$ LANGUAGE plpgsql;

SELECT less_than(text_field_1, text_field_2) FROM table1;
SELECT less_than(text_field_1, text_field_2 COLLATE "C") FROM table1;

The first use of less_than will use the common collation of text_field_1 and text_field_2 for the comparison, while the second use will use C collation.

Furthermore, the identified collation is also assumed as the collation of any local variables that are of collatable types. Thus this function would not work any differently if it were written as

CREATE FUNCTION less_than(a text, b text) RETURNS boolean AS $$
DECLARE
    local_a text := a;
    local_b text := b;
BEGIN
    RETURN local_a < local_b;
END;
$$ LANGUAGE plpgsql;

If there are no parameters of collatable data types, or no common collation can be identified for them, then parameters and local variables use the default collation of their data type (which is usually the database's default collation, but could be different for variables of domain types).

A local variable of a collatable data type can have a different collation associated with it by including the COLLATE option in its declaration, for example

DECLARE
    local_a text COLLATE "en_US";

This option overrides the collation that would otherwise be given to the variable according to the rules above.

Also, of course explicit COLLATE clauses can be written inside a function if it is desired to force a particular collation to be used in a particular operation. For example,

CREATE FUNCTION less_than_c(a text, b text) RETURNS boolean AS $$
BEGIN
    RETURN a < b COLLATE "C";
END;
$$ LANGUAGE plpgsql;

This overrides the collations associated with the table columns, parameters, or local variables used in the expression, just as would happen in a plain SQL command.

Prev | Up | Next
43.2. Structure of PL/pgSQL | Home | 43.4. Expressions


43.4. Expressions
Prev Up Chapter 43. PL/pgSQL — SQL Procedural Language Home Next

43.4. Expressions

All expressions used in PL/pgSQL statements are processed using the server's main SQL executor. For example, when you write a PL/pgSQL statement like

IF expression THEN ...

PL/pgSQL will evaluate the expression by feeding a query like

SELECT expression

to the main SQL engine. While forming the SELECT command, any occurrences of PL/pgSQL variable names are replaced by query parameters, as discussed in detail in Section 43.11.1. This allows the query plan for the SELECT to be prepared just once and then reused for subsequent evaluations with different values of the variables. Thus, what really happens on first use of an expression is essentially a PREPARE command. For example, if we have declared two integer variables x and y, and we write

IF x < y THEN ...

what happens behind the scenes is equivalent to

PREPARE statement_name(integer, integer) AS SELECT $1 < $2;

and then this prepared statement is EXECUTEd for each execution of the IF statement, with the current values of the PL/pgSQL variables supplied as parameter values. Normally these details are not important to a PL/pgSQL user, but they are useful to know when trying to diagnose a problem. More information appears in Section 43.11.2.

Since an expression is converted to a SELECT command, it can contain the same clauses that an ordinary SELECT would, except that it cannot include a top-level UNION, INTERSECT, or EXCEPT clause. Thus for example one could test whether a table is non-empty with

IF count(*) > 0 FROM my_table THEN ...

since the expression between IF and THEN is parsed as though it were SELECT count(*) > 0 FROM my_table. The SELECT must produce a single column, and not more than one row. (If it produces no rows, the result is taken as NULL.)

Prev | Up | Next
43.3. Declarations | Home | 43.5. Basic Statements


43.5. Basic Statements
Prev Up Chapter 43. PL/pgSQL — SQL Procedural Language Home Next

43.5. Basic Statements

43.5.1. Assignment
43.5.2. Executing SQL Commands
43.5.3. Executing a Command with a Single-Row Result
43.5.4. Executing Dynamic Commands
43.5.5. Obtaining the Result Status
43.5.6. Doing Nothing At All

In this section and the following ones, we describe all the statement types that are explicitly understood by PL/pgSQL. Anything not recognized as one of these statement types is presumed to be an SQL command and is sent to the main database engine to execute, as described in Section 43.5.2.

43.5.1. Assignment

An assignment of a value to a PL/pgSQL variable is written as:

variable { := | = } expression;

As explained previously, the expression in such a statement is evaluated by means of an SQL SELECT command sent to the main database engine. The expression must yield a single value (possibly a row value, if the variable is a row or record variable). The target variable can be a simple variable (optionally qualified with a block name), a field of a row or record target, or an element or slice of an array target. Equal (=) can be used instead of PL/SQL-compliant :=.

If the expression's result data type doesn't match the variable's data type, the value will be coerced as though by an assignment cast (see Section 10.4). If no assignment cast is known for the pair of data types involved, the PL/pgSQL interpreter will attempt to convert the result value textually, that is by applying the result type's output function followed by the variable type's input function. Note that this could result in run-time errors generated by the input function, if the string form of the result value is not acceptable to the input function.

Examples:

tax := subtotal * 0.06;
my_record.user_id := 20;
my_array[j] := 20;
my_array[1:3] := array[1,2,3];
complex_array[n].realpart = 12.3;

43.5.2. Executing SQL Commands

In general, any SQL command that does not return rows can be executed within a PL/pgSQL function just by writing the command. For example, you could create and fill a table by writing

CREATE TABLE mytable (id int primary key, data text);
INSERT INTO mytable VALUES (1,'one'), (2,'two');

If the command does return rows (for example SELECT, or INSERT/UPDATE/DELETE with RETURNING), there are two ways to proceed. When the command will return at most one row, or you only care about the first row of output, write the command as usual but add an INTO clause to capture the output, as described in Section 43.5.3. To process all of the output rows, write the command as the data source for a FOR loop, as described in Section 43.6.6.

Usually it is not sufficient just to execute statically-defined SQL commands. Typically you'll want a command to use varying data values, or even to vary in more fundamental ways such as by using different table names at different times. Again, there are two ways to proceed depending on the situation.

PL/pgSQL variable values can be automatically inserted into optimizable SQL commands, which are SELECT, INSERT, UPDATE, DELETE, and certain utility commands that incorporate one of these, such as EXPLAIN and CREATE TABLE ... AS SELECT. In these commands, any PL/pgSQL variable name appearing in the command text is replaced by a query parameter, and then the current value of the variable is provided as the parameter value at run time. This is exactly like the processing described earlier for expressions; for details see Section 43.11.1.

When executing an optimizable SQL command in this way, PL/pgSQL may cache and re-use the execution plan for the command, as discussed in Section 43.11.2.

Non-optimizable SQL commands (also called utility commands) are not capable of accepting query parameters. So automatic substitution of PL/pgSQL variables does not work in such commands. To include non-constant text in a utility command executed from PL/pgSQL, you must build the utility command as a string and then EXECUTE it, as discussed in Section 43.5.4.

EXECUTE must also be used if you want to modify the command in some other way than supplying a data value, for example by changing a table name.

Sometimes it is useful to evaluate an expression or SELECT query but discard the result, for example when calling a function that has side-effects but no useful result value. To do this in PL/pgSQL, use the PERFORM statement:

PERFORM query;

This executes query and discards the result. Write the query the same way you would write an SQL SELECT command, but replace the initial keyword SELECT with PERFORM. For WITH queries, use PERFORM and then place the query in parentheses. (In this case, the query can only return one row.) PL/pgSQL variables will be substituted into the query just as described above, and the plan is cached in the same way. Also, the special variable FOUND is set to true if the query produced at least one row, or false if it produced no rows (see Section 43.5.5).

Note

One might expect that writing SELECT directly would accomplish this result, but at present the only accepted way to do it is PERFORM. An SQL command that can return rows, such as SELECT, will be rejected as an error unless it has an INTO clause as discussed in the next section.

An example:

PERFORM create_mv('cs_session_page_requests_mv', my_query);

43.5.3. Executing a Command with a Single-Row Result

The result of an SQL command yielding a single row (possibly of multiple columns) can be assigned to a record variable, row-type variable, or list of scalar variables. This is done by writing the base SQL command and adding an INTO clause. For example,

SELECT select_expressions INTO [STRICT] target FROM ...;
INSERT ... RETURNING expressions INTO [STRICT] target;
UPDATE ... RETURNING expressions INTO [STRICT] target;
DELETE ... RETURNING expressions INTO [STRICT] target;

where target can be a record variable, a row variable, or a comma-separated list of simple variables and record/row fields. PL/pgSQL variables will be substituted into the rest of the command (that is, everything but the INTO clause) just as described above, and the plan is cached in the same way. This works for SELECT, INSERT/UPDATE/DELETE with RETURNING, and certain utility commands that return row sets, such as EXPLAIN. Except for the INTO clause, the SQL command is the same as it would be written outside PL/pgSQL.

Tip

Note that this interpretation of SELECT with INTO is quite different from PostgreSQL's regular SELECT INTO command, wherein the INTO target is a newly created table. If you want to create a table from a SELECT result inside a PL/pgSQL function, use the syntax CREATE TABLE ... AS SELECT.

If a row variable or a variable list is used as target, the command's result columns must exactly match the structure of the target as to number and data types, or else a run-time error occurs. When a record variable is the target, it automatically configures itself to the row type of the command's result columns.

The INTO clause can appear almost anywhere in the SQL command. Customarily it is written either just before or just after the list of select_expressions in a SELECT command, or at the end of the command for other command types. It is recommended that you follow this convention in case the PL/pgSQL parser becomes stricter in future versions.

If STRICT is not specified in the INTO clause, then target will be set to the first row returned by the command, or to nulls if the command returned no rows. (Note that “the first row” is not well-defined unless you've used ORDER BY.) Any result rows after the first row are discarded. You can check the special FOUND variable (see Section 43.5.5) to determine whether a row was returned:

SELECT * INTO myrec FROM emp WHERE empname = myname;
IF NOT FOUND THEN
    RAISE EXCEPTION 'employee % not found', myname;
END IF;

If the STRICT option is specified, the command must return exactly one row or a run-time error will be reported, either NO_DATA_FOUND (no rows) or TOO_MANY_ROWS (more than one row). You can use an exception block if you wish to catch the error, for example:

BEGIN
    SELECT * INTO STRICT myrec FROM emp WHERE empname = myname;
    EXCEPTION
        WHEN NO_DATA_FOUND THEN
            RAISE EXCEPTION 'employee % not found', myname;
        WHEN TOO_MANY_ROWS THEN
            RAISE EXCEPTION 'employee % not unique', myname;
END;

Successful execution of a command with STRICT always sets FOUND to true.

For INSERT/UPDATE/DELETE with RETURNING, PL/pgSQL reports an error for more than one returned row, even when STRICT is not specified. This is because there is no option such as ORDER BY with which to determine which affected row should be returned.

If print_strict_params is enabled for the function, then when an error is thrown because the requirements of STRICT are not met, the DETAIL part of the error message will include information about the parameters passed to the command. You can change the print_strict_params setting for all functions by setting plpgsql.print_strict_params, though only subsequent function compilations will be affected. You can also enable it on a per-function basis by using a compiler option, for example:

CREATE FUNCTION get_userid(username text) RETURNS int
AS $$
#print_strict_params on
DECLARE
userid int;
BEGIN
    SELECT users.userid INTO STRICT userid
        FROM users WHERE users.username = get_userid.username;
    RETURN userid;
END;
$$ LANGUAGE plpgsql;

On failure, this function might produce an error message such as

ERROR:  query returned no rows
DETAIL:  parameters: $1 = 'nosuchuser'
CONTEXT:  PL/pgSQL function get_userid(text) line 6 at SQL statement

Note

The STRICT option matches the behavior of Oracle PL/SQL's SELECT INTO and related statements.

43.5.4. Executing Dynamic Commands

Oftentimes you will want to generate dynamic commands inside your PL/pgSQL functions, that is, commands that will involve different tables or different data types each time they are executed. PL/pgSQL's normal attempts to cache plans for commands (as discussed in Section 43.11.2) will not work in such scenarios. To handle this sort of problem, the EXECUTE statement is provided:

EXECUTE command-string [ INTO [STRICT] target ] [ USING expression [, ... ] ];

where command-string is an expression yielding a string (of type text) containing the command to be executed. The optional target is a record variable, a row variable, or a comma-separated list of simple variables and record/row fields, into which the results of the command will be stored. The optional USING expressions supply values to be inserted into the command.

No substitution of PL/pgSQL variables is done on the computed command string. Any required variable values must be inserted in the command string as it is constructed; or you can use parameters as described below.

Also, there is no plan caching for commands executed via EXECUTE. Instead, the command is always planned each time the statement is run. Thus the command string can be dynamically created within the function to perform actions on different tables and columns.

The INTO clause specifies where the results of an SQL command returning rows should be assigned. If a row variable or variable list is provided, it must exactly match the structure of the command's results; if a record variable is provided, it will configure itself to match the result structure automatically. If multiple rows are returned, only the first will be assigned to the INTO variable(s). If no rows are returned, NULL is assigned to the INTO variable(s). If no INTO clause is specified, the command results are discarded.

If the STRICT option is given, an error is reported unless the command produces exactly one row.

The command string can use parameter values, which are referenced in the command as $1, $2, etc. These symbols refer to values supplied in the USING clause. This method is often preferable to inserting data values into the command string as text: it avoids run-time overhead of converting the values to text and back, and it is much less prone to SQL-injection attacks since there is no need for quoting or escaping. An example is:

EXECUTE 'SELECT count(*) FROM mytable WHERE inserted_by = $1 AND inserted <= $2'
   INTO c
   USING checked_user, checked_date;

Note that parameter symbols can only be used for data values — if you want to use dynamically determined table or column names, you must insert them into the command string textually. For example, if the preceding query needed to be done against a dynamically selected table, you could do this:

EXECUTE 'SELECT count(*) FROM '
    || quote_ident(tabname)
    || ' WHERE inserted_by = $1 AND inserted <= $2'
   INTO c
   USING checked_user, checked_date;

A cleaner approach is to use format()'s %I specification to insert table or column names with automatic quoting:

EXECUTE format('SELECT count(*) FROM %I '
   'WHERE inserted_by = $1 AND inserted <= $2', tabname)
   INTO c
   USING checked_user, checked_date;

(This example relies on the SQL rule that string literals separated by a newline are implicitly concatenated.)

Another restriction on parameter symbols is that they only work in optimizable SQL commands (SELECT, INSERT, UPDATE, DELETE, and certain commands containing one of these). In other statement types (generically called utility statements), you must insert values textually even if they are just data values.

An EXECUTE with a simple constant command string and some USING parameters, as in the first example above, is functionally equivalent to just writing the command directly in PL/pgSQL and allowing replacement of PL/pgSQL variables to happen automatically. The important difference is that EXECUTE will re-plan the command on each execution, generating a plan that is specific to the current parameter values; whereas PL/pgSQL may otherwise create a generic plan and cache it for re-use. In situations where the best plan depends strongly on the parameter values, it can be helpful to use EXECUTE to positively ensure that a generic plan is not selected.

SELECT INTO is not currently supported within EXECUTE; instead, execute a plain SELECT command and specify INTO as part of the EXECUTE itself.

Note

The PL/pgSQL EXECUTE statement is not related to the EXECUTE SQL statement supported by the PostgreSQL server. The server's EXECUTE statement cannot be used directly within PL/pgSQL functions (and is not needed).

Example 43.1. Quoting Values in Dynamic Queries

When working with dynamic commands you will often have to handle escaping of single quotes. The recommended method for quoting fixed text in your function body is dollar quoting. (If you have legacy code that does not use dollar quoting, please refer to the overview in Section 43.12.1, which can save you some effort when translating said code to a more reasonable scheme.)

Dynamic values require careful handling since they might contain quote characters. An example using format() (this assumes that you are dollar quoting the function body so quote marks need not be doubled):

EXECUTE format('UPDATE tbl SET %I = $1 '
   'WHERE key = $2', colname) USING newvalue, keyvalue;

It is also possible to call the quoting functions directly:

EXECUTE 'UPDATE tbl SET '
        || quote_ident(colname)
        || ' = '
        || quote_literal(newvalue)
        || ' WHERE key = '
        || quote_literal(keyvalue);

This example demonstrates the use of the quote_ident and quote_literal functions (see Section 9.4). For safety, expressions containing column or table identifiers should be passed through quote_ident before insertion in a dynamic query. Expressions containing values that should be literal strings in the constructed command should be passed through quote_literal. These functions take the appropriate steps to return the input text enclosed in double or single quotes respectively, with any embedded special characters properly escaped.

Because quote_literal is labeled STRICT, it will always return null when called with a null argument. In the above example, if newvalue or keyvalue were null, the entire dynamic query string would become null, leading to an error from EXECUTE. You can avoid this problem by using the quote_nullable function, which works the same as quote_literal except that when called with a null argument it returns the string NULL. For example,

EXECUTE 'UPDATE tbl SET '
        || quote_ident(colname)
        || ' = '
        || quote_nullable(newvalue)
        || ' WHERE key = '
        || quote_nullable(keyvalue);

If you are dealing with values that might be null, you should usually use quote_nullable in place of quote_literal.

As always, care must be taken to ensure that null values in a query do not deliver unintended results. For example the WHERE clause

'WHERE key = ' || quote_nullable(keyvalue)

will never succeed if keyvalue is null, because the result of using the equality operator = with a null operand is always null. If you wish null to work like an ordinary key value, you would need to rewrite the above as

'WHERE key IS NOT DISTINCT FROM ' || quote_nullable(keyvalue)

(At present, IS NOT DISTINCT FROM is handled much less efficiently than =, so don't do this unless you must. See Section 9.2 for more information on nulls and IS DISTINCT.)

Note that dollar quoting is only useful for quoting fixed text. It would be a very bad idea to try to write this example as:

EXECUTE 'UPDATE tbl SET '
        || quote_ident(colname)
        || ' = $$'
        || newvalue
        || '$$ WHERE key = '
        || quote_literal(keyvalue);

because it would break if the contents of newvalue happened to contain $$. The same objection would apply to any other dollar-quoting delimiter you might pick. So, to safely quote text that is not known in advance, you must use quote_literal, quote_nullable, or quote_ident, as appropriate.

Dynamic SQL statements can also be safely constructed using the format function (see Section 9.4.1). For example:

EXECUTE format('UPDATE tbl SET %I = %L '
   'WHERE key = %L', colname, newvalue, keyvalue);

%I is equivalent to quote_ident, and %L is equivalent to quote_nullable. The format function can be used in conjunction with the USING clause:

EXECUTE format('UPDATE tbl SET %I = $1 WHERE key = $2', colname)
   USING newvalue, keyvalue;

This form is better because the variables are handled in their native data type format, rather than unconditionally converting them to text and quoting them via %L. It is also more efficient.

A much larger example of a dynamic command and EXECUTE can be seen in Example 43.10, which builds and executes a CREATE FUNCTION command to define a new function.

43.5.5. Obtaining the Result Status

There are several ways to determine the effect of a command. The first method is to use the GET DIAGNOSTICS command, which has the form:

GET [ CURRENT ] DIAGNOSTICS variable { = | := } item [ , ... ];

This command allows retrieval of system status indicators. CURRENT is a noise word (but see also GET STACKED DIAGNOSTICS in Section 43.6.8.1). Each item is a key word identifying a status value to be assigned to the specified variable (which should be of the right data type to receive it). The currently available status items are shown in Table 43.1. Colon-equal (:=) can be used instead of the SQL-standard = token. An example:

GET DIAGNOSTICS integer_var = ROW_COUNT;

Table 43.1. Available Diagnostics Items

Name | Type | Description
ROW_COUNT | bigint | the number of rows processed by the most recent SQL command
PG_CONTEXT | text | line(s) of text describing the current call stack (see Section 43.6.9)

The second method to determine the effects of a command is to check the special variable named FOUND, which is of type boolean. FOUND starts out false within each PL/pgSQL function call. It is set by each of the following types of statements:

  • A SELECT INTO statement sets FOUND true if a row is assigned, false if no row is returned.

  • A PERFORM statement sets FOUND true if it produces (and discards) one or more rows, false if no row is produced.

  • UPDATE, INSERT, and DELETE statements set FOUND true if at least one row is affected, false if no row is affected.

  • A FETCH statement sets FOUND true if it returns a row, false if no row is returned.

  • A MOVE statement sets FOUND true if it successfully repositions the cursor, false otherwise.

  • A FOR or FOREACH statement sets FOUND true if it iterates one or more times, else false. FOUND is set this way when the loop exits; inside the execution of the loop, FOUND is not modified by the loop statement, although it might be changed by the execution of other statements within the loop body.

  • RETURN QUERY and RETURN QUERY EXECUTE statements set FOUND true if the query returns at least one row, false if no row is returned.

Other PL/pgSQL statements do not change the state of FOUND. Note in particular that EXECUTE changes the output of GET DIAGNOSTICS, but does not change FOUND.

FOUND is a local variable within each PL/pgSQL function; any changes to it affect only the current function.

43.5.6. Doing Nothing At All

Sometimes a placeholder statement that does nothing is useful. For example, it can indicate that one arm of an if/then/else chain is deliberately empty. For this purpose, use the NULL statement:

NULL;

For example, the following two fragments of code are equivalent:

BEGIN
    y := x / 0;
EXCEPTION
    WHEN division_by_zero THEN
        NULL;  -- ignore the error
END;

BEGIN
    y := x / 0;
EXCEPTION
    WHEN division_by_zero THEN  -- ignore the error
END;

Which is preferable is a matter of taste.

Note

In Oracle's PL/SQL, empty statement lists are not allowed, and so NULL statements are required for situations such as this. PL/pgSQL allows you to just write nothing, instead.

Prev | Up | Next
43.4. Expressions | Home | 43.6. Control Structures


43.6. Control Structures
Prev Up Chapter 43. PL/pgSQL — SQL Procedural Language Home Next

43.6. Control Structures

43.6.1. Returning from a Function
43.6.2. Returning from a Procedure
43.6.3. Calling a Procedure
43.6.4. Conditionals
43.6.5. Simple Loops
43.6.6. Looping through Query Results
43.6.7. Looping through Arrays
43.6.8. Trapping Errors
43.6.9. Obtaining Execution Location Information

Control structures are probably the most useful (and important) part of PL/pgSQL. With PL/pgSQL's control structures, you can manipulate PostgreSQL data in a very flexible and powerful way.

43.6.1. Returning from a Function

There are two commands available that allow you to return data from a function: RETURN and RETURN NEXT.

43.6.1.1. RETURN

RETURN expression;

RETURN with an expression terminates the function and returns the value of expression to the caller. This form is used for PL/pgSQL functions that do not return a set.

In a function that returns a scalar type, the expression's result will automatically be cast into the function's return type as described for assignments. But to return a composite (row) value, you must write an expression delivering exactly the requested column set. This may require use of explicit casting.

If you declared the function with output parameters, write just RETURN with no expression. The current values of the output parameter variables will be returned.

If you declared the function to return void, a RETURN statement can be used to exit the function early; but do not write an expression following RETURN.

The return value of a function cannot be left undefined. If control reaches the end of the top-level block of the function without hitting a RETURN statement, a run-time error will occur. This restriction does not apply to functions with output parameters and functions returning void, however. In those cases a RETURN statement is automatically executed if the top-level block finishes.

Some examples:

-- functions returning a scalar type
RETURN 1 + 2;
RETURN scalar_var;

-- functions returning a composite type
RETURN composite_type_var;
RETURN (1, 2, 'three'::text);  -- must cast columns to correct types

43.6.1.2. RETURN NEXT and RETURN QUERY

RETURN NEXT expression;
RETURN QUERY query;
RETURN QUERY EXECUTE command-string [ USING expression [, ... ] ];

When a PL/pgSQL function is declared to return SETOF sometype, the procedure to follow is slightly different. In that case, the individual items to return are specified by a sequence of RETURN NEXT or RETURN QUERY commands, and then a final RETURN command with no argument is used to indicate that the function has finished executing. RETURN NEXT can be used with both scalar and composite data types; with a composite result type, an entire “table” of results will be returned. RETURN QUERY appends the results of executing a query to the function's result set. RETURN NEXT and RETURN QUERY can be freely intermixed in a single set-returning function, in which case their results will be concatenated.

RETURN NEXT and RETURN QUERY do not actually return from the function — they simply append zero or more rows to the function's result set. Execution then continues with the next statement in the PL/pgSQL function. As successive RETURN NEXT or RETURN QUERY commands are executed, the result set is built up. A final RETURN, which should have no argument, causes control to exit the function (or you can just let control reach the end of the function).

RETURN QUERY has a variant RETURN QUERY EXECUTE, which specifies the query to be executed dynamically. Parameter expressions can be inserted into the computed query string via USING, in just the same way as in the EXECUTE command.

If you declared the function with output parameters, write just RETURN NEXT with no expression. On each execution, the current values of the output parameter variable(s) will be saved for eventual return as a row of the result. Note that you must declare the function as returning SETOF record when there are multiple output parameters, or SETOF sometype when there is just one output parameter of type sometype, in order to create a set-returning function with output parameters.

Here is an example of a function using RETURN NEXT:

CREATE TABLE foo (fooid INT, foosubid INT, fooname TEXT);
INSERT INTO foo VALUES (1, 2, 'three');
INSERT INTO foo VALUES (4, 5, 'six');

CREATE OR REPLACE FUNCTION get_all_foo() RETURNS SETOF foo AS
$BODY$
DECLARE
    r foo%rowtype;
BEGIN
    FOR r IN
        SELECT * FROM foo WHERE fooid > 0
    LOOP
        -- can do some processing here
        RETURN NEXT r; -- return current row of SELECT
    END LOOP;
    RETURN;
END;
$BODY$
LANGUAGE plpgsql;

SELECT * FROM get_all_foo();

Here is an example of a function using RETURN QUERY:

CREATE FUNCTION get_available_flightid(date) RETURNS SETOF integer AS
$BODY$
BEGIN
    RETURN QUERY SELECT flightid
                   FROM flight
                  WHERE flightdate >= $1
                    AND flightdate < ($1 + 1);

    -- Since execution is not finished, we can check whether rows were returned
    -- and raise exception if not.
    IF NOT FOUND THEN
        RAISE EXCEPTION 'No flight at %.', $1;
    END IF;

    RETURN;
 END;
$BODY$
LANGUAGE plpgsql;

-- Returns available flights or raises exception if there are no
-- available flights.
SELECT * FROM get_available_flightid(CURRENT_DATE);

Note

The current implementation of RETURN NEXT and RETURN QUERY stores the entire result set before returning from the function, as discussed above. That means that if a PL/pgSQL function produces a very large result set, performance might be poor: data will be written to disk to avoid memory exhaustion, but the function itself will not return until the entire result set has been generated. A future version of PL/pgSQL might allow users to define set-returning functions that do not have this limitation. Currently, the point at which data begins being written to disk is controlled by the work_mem configuration variable. Administrators who have sufficient memory to store larger result sets in memory should consider increasing this parameter.

43.6.2. Returning from a Procedure

A procedure does not have a return value. A procedure can therefore end without a RETURN statement. If you wish to use a RETURN statement to exit the code early, write just RETURN with no expression.

If the procedure has output parameters, the final values of the output parameter variables will be returned to the caller.

43.6.3. Calling a Procedure

A PL/pgSQL function, procedure, or DO block can call a procedure using CALL. Output parameters are handled differently from the way that CALL works in plain SQL. Each OUT or INOUT parameter of the procedure must correspond to a variable in the CALL statement, and whatever the procedure returns is assigned back to that variable after it returns. For example:

CREATE PROCEDURE triple(INOUT x int)
LANGUAGE plpgsql
AS $$
BEGIN
    x := x * 3;
END;
$$;

DO $$
DECLARE myvar int := 5;
BEGIN
  CALL triple(myvar);
  RAISE NOTICE 'myvar = %', myvar;  -- prints 15
END;
$$;

The variable corresponding to an output parameter can be a simple variable or a field of a composite-type variable. Currently, it cannot be an element of an array.

43.6.4. Conditionals

IF and CASE statements let you execute alternative commands based on certain conditions. PL/pgSQL has three forms of IF:

  • IF ... THEN ... END IF

  • IF ... THEN ... ELSE ... END IF

  • IF ... THEN ... ELSIF ... THEN ... ELSE ... END IF

and two forms of CASE:

  • CASE ... WHEN ... THEN ... ELSE ... END CASE

  • CASE WHEN ... THEN ... ELSE ... END CASE

43.6.4.1. IF-THEN

IF boolean-expression THEN
    statements
END IF;

IF-THEN statements are the simplest form of IF. The statements between THEN and END IF will be executed if the condition is true. Otherwise, they are skipped.

Example:

IF v_user_id <> 0 THEN
    UPDATE users SET email = v_email WHERE user_id = v_user_id;
END IF;

43.6.4.2. IF-THEN-ELSE

IF boolean-expression THEN
    statements
ELSE
    statements
END IF;

IF-THEN-ELSE statements add to IF-THEN by letting you specify an alternative set of statements that should be executed if the condition is not true. (Note this includes the case where the condition evaluates to NULL.)

Examples:

IF parentid IS NULL OR parentid = ''
THEN
    RETURN fullname;
ELSE
    RETURN hp_true_filename(parentid) || '/' || fullname;
END IF;

IF v_count > 0 THEN
    INSERT INTO users_count (count) VALUES (v_count);
    RETURN 't';
ELSE
    RETURN 'f';
END IF;

43.6.4.3. IF-THEN-ELSIF

IF boolean-expression THEN
    statements
[ ELSIF boolean-expression THEN
    statements
[ ELSIF boolean-expression THEN
    statements
    ...
]
]
[ ELSE
    statements ]
END IF;

Sometimes there are more than just two alternatives. IF-THEN-ELSIF provides a convenient method of checking several alternatives in turn. The IF conditions are tested successively until the first one that is true is found. Then the associated statement(s) are executed, after which control passes to the next statement after END IF. (Any subsequent IF conditions are not tested.) If none of the IF conditions is true, then the ELSE block (if any) is executed.

Here is an example:

IF number = 0 THEN
    result := 'zero';
ELSIF number > 0 THEN
    result := 'positive';
ELSIF number < 0 THEN
    result := 'negative';
ELSE
    -- hmm, the only other possibility is that number is null
    result := 'NULL';
END IF;

The key word ELSIF can also be spelled ELSEIF.

An alternative way of accomplishing the same task is to nest IF-THEN-ELSE statements, as in the following example:

IF demo_row.sex = 'm' THEN
    pretty_sex := 'man';
ELSE
    IF demo_row.sex = 'f' THEN
        pretty_sex := 'woman';
    END IF;
END IF;

However, this method requires writing a matching END IF for each IF, so it is much more cumbersome than using ELSIF when there are many alternatives.

43.6.4.4. Simple CASE

CASE search-expression
    WHEN expression [, expression [ ... ]] THEN
      statements
  [ WHEN expression [, expression [ ... ]] THEN
      statements
    ... ]
  [ ELSE
      statements ]
END CASE;

The simple form of CASE provides conditional execution based on equality of operands. The search-expression is evaluated (once) and successively compared to each expression in the WHEN clauses. If a match is found, then the corresponding statements are executed, and then control passes to the next statement after END CASE. (Subsequent WHEN expressions are not evaluated.) If no match is found, the ELSE statements are executed; but if ELSE is not present, then a CASE_NOT_FOUND exception is raised.

Here is a simple example:

CASE x
    WHEN 1, 2 THEN
        msg := 'one or two';
    ELSE
        msg := 'other value than one or two';
END CASE;

43.6.4.5. Searched CASE

CASE
    WHEN boolean-expression THEN
      statements
  [ WHEN boolean-expression THEN
      statements
    ... ]
  [ ELSE
      statements ]
END CASE;

The searched form of CASE provides conditional execution based on truth of Boolean expressions. Each WHEN clause's boolean-expression is evaluated in turn, until one is found that yields true. Then the corresponding statements are executed, and then control passes to the next statement after END CASE. (Subsequent WHEN expressions are not evaluated.) If no true result is found, the ELSE statements are executed; but if ELSE is not present, then a CASE_NOT_FOUND exception is raised.

Here is an example:

CASE
    WHEN x BETWEEN 0 AND 10 THEN
        msg := 'value is between zero and ten';
    WHEN x BETWEEN 11 AND 20 THEN
        msg := 'value is between eleven and twenty';
END CASE;

This form of CASE is entirely equivalent to IF-THEN-ELSIF, except for the rule that reaching an omitted ELSE clause results in an error rather than doing nothing.

43.6.5. Simple Loops

With the LOOP, EXIT, CONTINUE, WHILE, FOR, and FOREACH statements, you can arrange for your PL/pgSQL function to repeat a series of commands.

43.6.5.1. LOOP

[ <<label>> ]
LOOP
    statements
END LOOP [ label ];

LOOP defines an unconditional loop that is repeated indefinitely until terminated by an EXIT or RETURN statement. The optional label can be used by EXIT and CONTINUE statements within nested loops to specify which loop those statements refer to.

43.6.5.2. EXIT

EXIT [ label ] [ WHEN boolean-expression ];

If no label is given, the innermost loop is terminated and the statement following END LOOP is executed next. If label is given, it must be the label of the current or some outer level of nested loop or block. Then the named loop or block is terminated and control continues with the statement after the loop's/block's corresponding END.

If WHEN is specified, the loop exit occurs only if boolean-expression is true. Otherwise, control passes to the statement after EXIT.

EXIT can be used with all types of loops; it is not limited to use with unconditional loops.

When used with a BEGIN block, EXIT passes control to the next statement after the end of the block. Note that a label must be used for this purpose; an unlabeled EXIT is never considered to match a BEGIN block. (This is a change from pre-8.4 releases of PostgreSQL, which would allow an unlabeled EXIT to match a BEGIN block.)

Examples:

LOOP
    -- some computations
    IF count > 0 THEN
        EXIT;  -- exit loop
    END IF;
END LOOP;

LOOP
    -- some computations
    EXIT WHEN count > 0;  -- same result as previous example
END LOOP;

<<ablock>>
BEGIN
    -- some computations
    IF stocks > 100000 THEN
        EXIT ablock;  -- causes exit from the BEGIN block
    END IF;
    -- computations here will be skipped when stocks > 100000
END;

43.6.5.3. CONTINUE

CONTINUE [ label ] [ WHEN boolean-expression ];

If no label is given, the next iteration of the innermost loop is begun. That is, all statements remaining in the loop body are skipped, and control returns to the loop control expression (if any) to determine whether another loop iteration is needed. If label is present, it specifies the label of the loop whose execution will be continued.

If WHEN is specified, the next iteration of the loop is begun only if boolean-expression is true. Otherwise, control passes to the statement after CONTINUE.

CONTINUE can be used with all types of loops; it is not limited to use with unconditional loops.

Examples:

LOOP
    -- some computations
    EXIT WHEN count > 100;
    CONTINUE WHEN count < 50;
    -- some computations for count IN [50 .. 100]
END LOOP;

43.6.5.4. WHILE

[ <<label>> ]
WHILE boolean-expression LOOP
    statements
END LOOP [ label ];

The WHILE statement repeats a sequence of statements so long as the boolean-expression evaluates to true. The expression is checked just before each entry to the loop body.

For example:

WHILE amount_owed > 0 AND gift_certificate_balance > 0 LOOP
    -- some computations here
END LOOP;

WHILE NOT done LOOP
    -- some computations here
END LOOP;

43.6.5.5. FOR (Integer Variant)

[ <<label>> ]
FOR name IN [ REVERSE ] expression .. expression [ BY expression ] LOOP
    statements
END LOOP [ label ];

This form of FOR creates a loop that iterates over a range of integer values. The variable name is automatically defined as type integer and exists only inside the loop (any existing definition of the variable name is ignored within the loop). The two expressions giving the lower and upper bound of the range are evaluated once when entering the loop. If the BY clause isn't specified the iteration step is 1, otherwise it's the value specified in the BY clause, which again is evaluated once on loop entry. If REVERSE is specified then the step value is subtracted, rather than added, after each iteration.

Some examples of integer FOR loops:

FOR i IN 1..10 LOOP
    -- i will take on the values 1,2,3,4,5,6,7,8,9,10 within the loop
END LOOP;

FOR i IN REVERSE 10..1 LOOP
    -- i will take on the values 10,9,8,7,6,5,4,3,2,1 within the loop
END LOOP;

FOR i IN REVERSE 10..1 BY 2 LOOP
    -- i will take on the values 10,8,6,4,2 within the loop
END LOOP;

If the lower bound is greater than the upper bound (or less than, in the REVERSE case), the loop body is not executed at all. No error is raised.

If a label is attached to the FOR loop then the integer loop variable can be referenced with a qualified name, using that label.

43.6.6. Looping through Query Results

Using a different type of FOR loop, you can iterate through the results of a query and manipulate that data accordingly. The syntax is:

[ <<label>> ]
FOR target IN query LOOP
    statements
END LOOP [ label ];

The target is a record variable, row variable, or comma-separated list of scalar variables. The target is successively assigned each row resulting from the query and the loop body is executed for each row. Here is an example:

CREATE FUNCTION refresh_mviews() RETURNS integer AS $$
DECLARE
    mviews RECORD;
BEGIN
    RAISE NOTICE 'Refreshing all materialized views...';

    FOR mviews IN
       SELECT n.nspname AS mv_schema,
              c.relname AS mv_name,
              pg_catalog.pg_get_userbyid(c.relowner) AS owner
         FROM pg_catalog.pg_class c
    LEFT JOIN pg_catalog.pg_namespace n ON (n.oid = c.relnamespace)
        WHERE c.relkind = 'm'
     ORDER BY 1
    LOOP

        -- Now "mviews" has one record with information about the materialized view

        RAISE NOTICE 'Refreshing materialized view %.% (owner: %)...',
                     quote_ident(mviews.mv_schema),
                     quote_ident(mviews.mv_name),
                     quote_ident(mviews.owner);
        EXECUTE format('REFRESH MATERIALIZED VIEW %I.%I', mviews.mv_schema, mviews.mv_name);
    END LOOP;

    RAISE NOTICE 'Done refreshing materialized views.';
    RETURN 1;
END;
$$ LANGUAGE plpgsql;

If the loop is terminated by an EXIT statement, the last assigned row value is still accessible after the loop.

The query used in this type of FOR statement can be any SQL command that returns rows to the caller: SELECT is the most common case, but you can also use INSERT, UPDATE, or DELETE with a RETURNING clause. Some utility commands such as EXPLAIN will work too.

PL/pgSQL variables are replaced by query parameters, and the query plan is cached for possible re-use, as discussed in detail in Section 43.11.1 and Section 43.11.2.

The FOR-IN-EXECUTE statement is another way to iterate over rows:

[ <<label>> ]
FOR target IN EXECUTE text_expression [ USING expression [, ... ] ] LOOP
    statements
END LOOP [ label ];

This is like the previous form, except that the source query is specified as a string expression, which is evaluated and replanned on each entry to the FOR loop. This allows the programmer to choose the speed of a preplanned query or the flexibility of a dynamic query, just as with a plain EXECUTE statement. As with EXECUTE, parameter values can be inserted into the dynamic command via USING.

Another way to specify the query whose results should be iterated through is to declare it as a cursor. This is described in Section 43.7.4.

43.6.7. Looping through Arrays

The FOREACH loop is much like a FOR loop, but instead of iterating through the rows returned by an SQL query, it iterates through the elements of an array value. (In general, FOREACH is meant for looping through components of a composite-valued expression; variants for looping through composites besides arrays may be added in future.) The FOREACH statement to loop over an array is:

[ <<label>> ]
FOREACH target [ SLICE number ] IN ARRAY expression LOOP
    statements
END LOOP [ label ];

Without SLICE, or if SLICE 0 is specified, the loop iterates through individual elements of the array produced by evaluating the expression. The target variable is assigned each element value in sequence, and the loop body is executed for each element. Here is an example of looping through the elements of an integer array:

CREATE FUNCTION sum(int[]) RETURNS int8 AS $$
DECLARE
  s int8 := 0;
  x int;
BEGIN
  FOREACH x IN ARRAY $1
  LOOP
    s := s + x;
  END LOOP;
  RETURN s;
END;
$$ LANGUAGE plpgsql;

The elements are visited in storage order, regardless of the number of array dimensions. Although the target is usually just a single variable, it can be a list of variables when looping through an array of composite values (records). In that case, for each array element, the variables are assigned from successive columns of the composite value.

With a positive SLICE value, FOREACH iterates through slices of the array rather than single elements. The SLICE value must be an integer constant not larger than the number of dimensions of the array. The target variable must be an array, and it receives successive slices of the array value, where each slice is of the number of dimensions specified by SLICE. Here is an example of iterating through one-dimensional slices:

CREATE FUNCTION scan_rows(int[]) RETURNS void AS $$
DECLARE
  x int[];
BEGIN
  FOREACH x SLICE 1 IN ARRAY $1
  LOOP
    RAISE NOTICE 'row = %', x;
  END LOOP;
END;
$$ LANGUAGE plpgsql;

SELECT scan_rows(ARRAY[[1,2,3],[4,5,6],[7,8,9],[10,11,12]]);

NOTICE:  row = {1,2,3}
NOTICE:  row = {4,5,6}
NOTICE:  row = {7,8,9}
NOTICE:  row = {10,11,12}

43.6.8. Trapping Errors

By default, any error occurring in a PL/pgSQL function aborts execution of the function and the surrounding transaction. You can trap errors and recover from them by using a BEGIN block with an EXCEPTION clause. The syntax is an extension of the normal syntax for a BEGIN block:

[ <<label>> ]
[ DECLARE
    declarations ]
BEGIN
    statements
EXCEPTION
    WHEN condition [ OR condition ... ] THEN
        handler_statements
    [ WHEN condition [ OR condition ... ] THEN
          handler_statements
      ... ]
END;

If no error occurs, this form of block simply executes all the statements, and then control passes to the next statement after END. But if an error occurs within the statements, further processing of the statements is abandoned, and control passes to the EXCEPTION list. The list is searched for the first condition matching the error that occurred. If a match is found, the corresponding handler_statements are executed, and then control passes to the next statement after END. If no match is found, the error propagates out as though the EXCEPTION clause were not there at all: the error can be caught by an enclosing block with EXCEPTION, or if there is none it aborts processing of the function.

The condition names can be any of those shown in Appendix A. A category name matches any error within its category. The special condition name OTHERS matches every error type except QUERY_CANCELED and ASSERT_FAILURE. (It is possible, but often unwise, to trap those two error types by name.) Condition names are not case-sensitive. Also, an error condition can be specified by SQLSTATE code; for example these are equivalent:

WHEN division_by_zero THEN ...
WHEN SQLSTATE '22012' THEN ...

If a new error occurs within the selected handler_statements, it cannot be caught by this EXCEPTION clause, but is propagated out. A surrounding EXCEPTION clause could catch it.

When an error is caught by an EXCEPTION clause, the local variables of the PL/pgSQL function remain as they were when the error occurred, but all changes to persistent database state within the block are rolled back. As an example, consider this fragment:

INSERT INTO mytab(firstname, lastname) VALUES('Tom', 'Jones');
BEGIN
    UPDATE mytab SET firstname = 'Joe' WHERE lastname = 'Jones';
    x := x + 1;
    y := x / 0;
EXCEPTION
    WHEN division_by_zero THEN
        RAISE NOTICE 'caught division_by_zero';
        RETURN x;
END;

When control reaches the assignment to y, it will fail with a division_by_zero error. This will be caught by the EXCEPTION clause. The value returned in the RETURN statement will be the incremented value of x, but the effects of the UPDATE command will have been rolled back. The INSERT command preceding the block is not rolled back, however, so the end result is that the database contains Tom Jones not Joe Jones.

Tip

A block containing an EXCEPTION clause is significantly more expensive to enter and exit than a block without one. Therefore, don't use EXCEPTION without need.

Example 43.2. Exceptions with UPDATE/INSERT

This example uses exception handling to perform either UPDATE or INSERT, as appropriate. It is recommended that applications use INSERT with ON CONFLICT DO UPDATE rather than actually using this pattern. This example serves primarily to illustrate use of PL/pgSQL control flow structures:

CREATE TABLE db (a INT PRIMARY KEY, b TEXT);

CREATE FUNCTION merge_db(key INT, data TEXT) RETURNS VOID AS
$$
BEGIN
    LOOP
        -- first try to update the key
        UPDATE db SET b = data WHERE a = key;
        IF found THEN
            RETURN;
        END IF;
        -- not there, so try to insert the key
        -- if someone else inserts the same key concurrently,
        -- we could get a unique-key failure
        BEGIN
            INSERT INTO db(a,b) VALUES (key, data);
            RETURN;
        EXCEPTION WHEN unique_violation THEN
            -- Do nothing, and loop to try the UPDATE again.
        END;
    END LOOP;
END;
$$
LANGUAGE plpgsql;

SELECT merge_db(1, 'david');
SELECT merge_db(1, 'dennis');

This coding assumes the unique_violation error is caused by the INSERT, and not by, say, an INSERT in a trigger function on the table. It might also misbehave if there is more than one unique index on the table, since it will retry the operation regardless of which index caused the error. More safety could be had by using the features discussed next to check that the trapped error was the one expected.

43.6.8.1. Obtaining Information about an Error

Exception handlers frequently need to identify the specific error that occurred. There are two ways to get information about the current exception in PL/pgSQL: special variables and the GET STACKED DIAGNOSTICS command.

Within an exception handler, the special variable SQLSTATE contains the error code that corresponds to the exception that was raised (refer to Table A.1 for a list of possible error codes). The special variable SQLERRM contains the error message associated with the exception. These variables are undefined outside exception handlers.

Within an exception handler, one may also retrieve information about the current exception by using the GET STACKED DIAGNOSTICS command, which has the form:

GET STACKED DIAGNOSTICS variable { = | := } item [ , ... ];

Each item is a key word identifying a status value to be assigned to the specified variable (which should be of the right data type to receive it). The currently available status items are shown in Table 43.2.

Table 43.2. Error Diagnostics Items

Name | Type | Description
RETURNED_SQLSTATE | text | the SQLSTATE error code of the exception
COLUMN_NAME | text | the name of the column related to exception
CONSTRAINT_NAME | text | the name of the constraint related to exception
PG_DATATYPE_NAME | text | the name of the data type related to exception
MESSAGE_TEXT | text | the text of the exception's primary message
TABLE_NAME | text | the name of the table related to exception
SCHEMA_NAME | text | the name of the schema related to exception
PG_EXCEPTION_DETAIL | text | the text of the exception's detail message, if any
PG_EXCEPTION_HINT | text | the text of the exception's hint message, if any
PG_EXCEPTION_CONTEXT | text | line(s) of text describing the call stack at the time of the exception (see Section 43.6.9)

If the exception did not set a value for an item, an empty string will be returned.

Here is an example:

DECLARE
  text_var1 text;
  text_var2 text;
  text_var3 text;
BEGIN
  -- some processing which might cause an exception
  ...
EXCEPTION WHEN OTHERS THEN
  GET STACKED DIAGNOSTICS text_var1 = MESSAGE_TEXT,
                          text_var2 = PG_EXCEPTION_DETAIL,
                          text_var3 = PG_EXCEPTION_HINT;
END;

43.6.9. Obtaining Execution Location Information

The GET DIAGNOSTICS command, previously described in Section 43.5.5, retrieves information about current execution state (whereas the GET STACKED DIAGNOSTICS command discussed above reports information about the execution state as of a previous error). Its PG_CONTEXT status item is useful for identifying the current execution location. PG_CONTEXT returns a text string with line(s) of text describing the call stack. The first line refers to the current function and currently executing GET DIAGNOSTICS command. The second and any subsequent lines refer to calling functions further up the call stack. For example:

CREATE OR REPLACE FUNCTION outer_func() RETURNS integer AS $$
BEGIN
  RETURN inner_func();
END;
$$ LANGUAGE plpgsql;

CREATE OR REPLACE FUNCTION inner_func() RETURNS integer AS $$
DECLARE
  stack text;
BEGIN
  GET DIAGNOSTICS stack = PG_CONTEXT;
  RAISE NOTICE E'--- Call Stack ---\n%', stack;
  RETURN 1;
END;
$$ LANGUAGE plpgsql;

SELECT outer_func();

NOTICE:  --- Call Stack ---
PL/pgSQL function inner_func() line 5 at GET DIAGNOSTICS
PL/pgSQL function outer_func() line 3 at RETURN
CONTEXT:  PL/pgSQL function outer_func() line 3 at RETURN
 outer_func
 ------------
           1
(1 row)

GET STACKED DIAGNOSTICS ... PG_EXCEPTION_CONTEXT returns the same sort of stack trace, but describing the location at which an error was detected, rather than the current location.

Prev | Up | Next
43.5. Basic Statements | Home | 43.7. Cursors


43.7. Cursors
Prev Up Chapter 43. PL/pgSQL — SQL Procedural Language Home Next

43.7. Cursors

43.7.1. Declaring Cursor Variables
43.7.2. Opening Cursors
43.7.3. Using Cursors
43.7.4. Looping through a Cursor's Result

Rather than executing a whole query at once, it is possible to set up a cursor that encapsulates the query, and then read the query result a few rows at a time. One reason for doing this is to avoid memory overrun when the result contains a large number of rows. (However, PL/pgSQL users do not normally need to worry about that, since FOR loops automatically use a cursor internally to avoid memory problems.) A more interesting usage is to return a reference to a cursor that a function has created, allowing the caller to read the rows. This provides an efficient way to return large row sets from functions.

43.7.1. Declaring Cursor Variables

All access to cursors in PL/pgSQL goes through cursor variables, which are always of the special data type refcursor. One way to create a cursor variable is just to declare it as a variable of type refcursor. Another way is to use the cursor declaration syntax, which in general is:

name [ [ NO ] SCROLL ] CURSOR [ ( arguments ) ] FOR query;

(FOR can be replaced by IS for Oracle compatibility.) If SCROLL is specified, the cursor will be capable of scrolling backward; if NO SCROLL is specified, backward fetches will be rejected; if neither specification appears, it is query-dependent whether backward fetches will be allowed. arguments, if specified, is a comma-separated list of pairs name datatype that define names to be replaced by parameter values in the given query. The actual values to substitute for these names will be specified later, when the cursor is opened.

Some examples:

DECLARE
    curs1 refcursor;
    curs2 CURSOR FOR SELECT * FROM tenk1;
    curs3 CURSOR (key integer) FOR SELECT * FROM tenk1 WHERE unique1 = key;

All three of these variables have the data type refcursor, but the first can be used with any query, while the second has a fully specified query already bound to it, and the last has a parameterized query bound to it. (key will be replaced by an integer parameter value when the cursor is opened.) The variable curs1 is said to be unbound since it is not bound to any particular query.

The SCROLL option cannot be used when the cursor's query uses FOR UPDATE/SHARE. Also, it is best to use NO SCROLL with a query that involves volatile functions. The implementation of SCROLL assumes that re-reading the query's output will give consistent results, which a volatile function might not do.

43.7.2. Opening Cursors

Before a cursor can be used to retrieve rows, it must be opened. (This is the equivalent action to the SQL command DECLARE CURSOR.) PL/pgSQL has three forms of the OPEN statement, two of which use unbound cursor variables while the third uses a bound cursor variable.

Note

Bound cursor variables can also be used without explicitly opening the cursor, via the FOR statement described in Section 43.7.4.

43.7.2.1. OPEN FOR query

OPEN unbound_cursorvar [ [ NO ] SCROLL ] FOR query;

The cursor variable is opened and given the specified query to execute. The cursor cannot be open already, and it must have been declared as an unbound cursor variable (that is, as a simple refcursor variable). The query must be a SELECT, or something else that returns rows (such as EXPLAIN). The query is treated in the same way as other SQL commands in PL/pgSQL: PL/pgSQL variable names are substituted, and the query plan is cached for possible reuse. When a PL/pgSQL variable is substituted into the cursor query, the value that is substituted is the one it has at the time of the OPEN; subsequent changes to the variable will not affect the cursor's behavior. The SCROLL and NO SCROLL options have the same meanings as for a bound cursor.

An example:

OPEN curs1 FOR SELECT * FROM foo WHERE key = mykey;

43.7.2.2. OPEN FOR EXECUTE

OPEN unbound_cursorvar [ [ NO ] SCROLL ] FOR EXECUTE query_string
                                     [ USING expression [, ... ] ];

The cursor variable is opened and given the specified query to execute. The cursor cannot be open already, and it must have been declared as an unbound cursor variable (that is, as a simple refcursor variable). The query is specified as a string expression, in the same way as in the EXECUTE command. As usual, this gives flexibility so the query plan can vary from one run to the next (see Section 43.11.2), and it also means that variable substitution is not done on the command string. As with EXECUTE, parameter values can be inserted into the dynamic command via format() and USING. The SCROLL and NO SCROLL options have the same meanings as for a bound cursor.

An example:

OPEN curs1 FOR EXECUTE format('SELECT * FROM %I WHERE col1 = $1',tabname) USING keyvalue;

In this example, the table name is inserted into the query via format(). The comparison value for col1 is inserted via a USING parameter, so it needs no quoting.

43.7.2.3. Opening a Bound Cursor

OPEN bound_cursorvar [ ( [ argument_name := ] argument_value [, ...] ) ];

This form of OPEN is used to open a cursor variable whose query was bound to it when it was declared. The cursor cannot be open already. A list of actual argument value expressions must appear if and only if the cursor was declared to take arguments. These values will be substituted in the query.

The query plan for a bound cursor is always considered cacheable; there is no equivalent of EXECUTE in this case. Notice that SCROLL and NO SCROLL cannot be specified in OPEN, as the cursor's scrolling behavior was already determined.

Argument values can be passed using either positional or named notation. In positional notation, all arguments are specified in order. In named notation, each argument's name is specified using := to separate it from the argument expression. Similar to calling functions, described in Section 4.3, it is also allowed to mix positional and named notation.

Examples (these use the cursor declaration examples above):

OPEN curs2;
OPEN curs3(42);
OPEN curs3(key := 42);

Because variable substitution is done on a bound cursor's query, there are really two ways to pass values into the cursor: either with an explicit argument to OPEN, or implicitly by referencing a PL/pgSQL variable in the query. However, only variables declared before the bound cursor was declared will be substituted into it. In either case the value to be passed is determined at the time of the OPEN. For example, another way to get the same effect as the curs3 example above is

DECLARE
    key integer;
    curs4 CURSOR FOR SELECT * FROM tenk1 WHERE unique1 = key;
BEGIN
    key := 42;
    OPEN curs4;

43.7.3. Using Cursors

Once a cursor has been opened, it can be manipulated with the statements described here.

These manipulations need not occur in the same function that opened the cursor to begin with. You can return a refcursor value out of a function and let the caller operate on the cursor. (Internally, a refcursor value is simply the string name of a so-called portal containing the active query for the cursor. This name can be passed around, assigned to other refcursor variables, and so on, without disturbing the portal.)

All portals are implicitly closed at transaction end. Therefore a refcursor value is usable to reference an open cursor only until the end of the transaction.

43.7.3.1. FETCH

FETCH [ direction { FROM | IN } ] cursor INTO target;

FETCH retrieves the next row from the cursor into a target, which might be a row variable, a record variable, or a comma-separated list of simple variables, just like SELECT INTO. If there is no next row, the target is set to NULL(s). As with SELECT INTO, the special variable FOUND can be checked to see whether a row was obtained or not.

The direction clause can be any of the variants allowed in the SQL FETCH command except the ones that can fetch more than one row; namely, it can be NEXT, PRIOR, FIRST, LAST, ABSOLUTE count, RELATIVE count, FORWARD, or BACKWARD. Omitting direction is the same as specifying NEXT. In the forms using a count, the count can be any integer-valued expression (unlike the SQL FETCH command, which only allows an integer constant). direction values that require moving backward are likely to fail unless the cursor was declared or opened with the SCROLL option.

cursor must be the name of a refcursor variable that references an open cursor portal.

Examples:

FETCH curs1 INTO rowvar;
FETCH curs2 INTO foo, bar, baz;
FETCH LAST FROM curs3 INTO x, y;
FETCH RELATIVE -2 FROM curs4 INTO x;

43.7.3.2. MOVE

MOVE [ direction { FROM | IN } ] cursor;

MOVE repositions a cursor without retrieving any data. MOVE works exactly like the FETCH command, except it only repositions the cursor and does not return the row moved to. As with SELECT INTO, the special variable FOUND can be checked to see whether there was a next row to move to.

Examples:

MOVE curs1;
MOVE LAST FROM curs3;
MOVE RELATIVE -2 FROM curs4;
MOVE FORWARD 2 FROM curs4;

43.7.3.3. UPDATE/DELETE WHERE CURRENT OF

UPDATE table SET ... WHERE CURRENT OF cursor;
DELETE FROM table WHERE CURRENT OF cursor;

When a cursor is positioned on a table row, that row can be updated or deleted using the cursor to identify the row. There are restrictions on what the cursor's query can be (in particular, no grouping) and it's best to use FOR UPDATE in the cursor. For more information see the DECLARE reference page.

An example:

UPDATE foo SET dataval = myval WHERE CURRENT OF curs1;

43.7.3.4. CLOSE

CLOSE cursor;

CLOSE closes the portal underlying an open cursor. This can be used to release resources earlier than end of transaction, or to free up the cursor variable to be opened again.

An example:

CLOSE curs1;

43.7.3.5. Returning Cursors

PL/pgSQL functions can return cursors to the caller. This is useful to return multiple rows or columns, especially with very large result sets. To do this, the function opens the cursor and returns the cursor name to the caller (or simply opens the cursor using a portal name specified by or otherwise known to the caller). The caller can then fetch rows from the cursor. The cursor can be closed by the caller, or it will be closed automatically when the transaction closes.

The portal name used for a cursor can be specified by the programmer or automatically generated. To specify a portal name, simply assign a string to the refcursor variable before opening it. The string value of the refcursor variable will be used by OPEN as the name of the underlying portal. However, if the refcursor variable is null, OPEN automatically generates a name that does not conflict with any existing portal, and assigns it to the refcursor variable.

Note

A bound cursor variable is initialized to the string value representing its name, so that the portal name is the same as the cursor variable name, unless the programmer overrides it by assignment before opening the cursor. But an unbound cursor variable defaults to the null value initially, so it will receive an automatically-generated unique name, unless overridden.

The following example shows one way a cursor name can be supplied by the caller:

CREATE TABLE test (col text);
INSERT INTO test VALUES ('123');

CREATE FUNCTION reffunc(refcursor) RETURNS refcursor AS '
BEGIN
    OPEN $1 FOR SELECT col FROM test;
    RETURN $1;
END;
' LANGUAGE plpgsql;

BEGIN;
SELECT reffunc('funccursor');
FETCH ALL IN funccursor;
COMMIT;

The following example uses automatic cursor name generation:

CREATE FUNCTION reffunc2() RETURNS refcursor AS '
DECLARE
    ref refcursor;
BEGIN
    OPEN ref FOR SELECT col FROM test;
    RETURN ref;
END;
' LANGUAGE plpgsql;

-- need to be in a transaction to use cursors.
BEGIN;
SELECT reffunc2();

      reffunc2
--------------------
 <unnamed cursor 1>
(1 row)

FETCH ALL IN "<unnamed cursor 1>";
COMMIT;

The following example shows one way to return multiple cursors from a single function:

CREATE FUNCTION myfunc(refcursor, refcursor) RETURNS SETOF refcursor AS $$
BEGIN
    OPEN $1 FOR SELECT * FROM table_1;
    RETURN NEXT $1;
    OPEN $2 FOR SELECT * FROM table_2;
    RETURN NEXT $2;
END;
$$ LANGUAGE plpgsql;

-- need to be in a transaction to use cursors.
BEGIN;

SELECT * FROM myfunc('a', 'b');

FETCH ALL FROM a;
FETCH ALL FROM b;
COMMIT;

43.7.4. Looping through a Cursor's Result

There is a variant of the FOR statement that allows iterating through the rows returned by a cursor. The syntax is:

[ <<label>> ]
FOR recordvar IN bound_cursorvar [ ( [ argument_name := ] argument_value [, ...] ) ] LOOP
    statements
END LOOP [ label ];

The cursor variable must have been bound to some query when it was declared, and it cannot be open already. The FOR statement automatically opens the cursor, and it closes the cursor again when the loop exits. A list of actual argument value expressions must appear if and only if the cursor was declared to take arguments. These values will be substituted in the query, in just the same way as during an OPEN (see Section 43.7.2.3).

The variable recordvar is automatically defined as type record and exists only inside the loop (any existing definition of the variable name is ignored within the loop). Each row returned by the cursor is successively assigned to this record variable and the loop body is executed.

Prev | Up | Next
43.6. Control Structures | Home | 43.8. Transaction Management


43.8. Transaction Management
Prev Up Chapter 43. PL/pgSQL — SQL Procedural Language Home Next

43.8. Transaction Management

In procedures invoked by the CALL command as well as in anonymous code blocks (DO command), it is possible to end transactions using the commands COMMIT and ROLLBACK. A new transaction is started automatically after a transaction is ended using these commands, so there is no separate START TRANSACTION command. (Note that BEGIN and END have different meanings in PL/pgSQL.)

Here is a simple example:

CREATE PROCEDURE transaction_test1()
LANGUAGE plpgsql
AS $$
BEGIN
    FOR i IN 0..9 LOOP
        INSERT INTO test1 (a) VALUES (i);
        IF i % 2 = 0 THEN
            COMMIT;
        ELSE
            ROLLBACK;
        END IF;
    END LOOP;
END;
$$;

CALL transaction_test1();

A new transaction starts out with default transaction characteristics such as transaction isolation level. In cases where transactions are committed in a loop, it might be desirable to start new transactions automatically with the same characteristics as the previous one. The commands COMMIT AND CHAIN and ROLLBACK AND CHAIN accomplish this.

Transaction control is only possible in CALL or DO invocations from the top level or nested CALL or DO invocations without any other intervening command. For example, if the call stack is CALL proc1() → CALL proc2() → CALL proc3(), then the second and third procedures can perform transaction control actions. But if the call stack is CALL proc1() → SELECT func2() → CALL proc3(), then the last procedure cannot do transaction control, because of the SELECT in between.

Special considerations apply to cursor loops. Consider this example:

CREATE PROCEDURE transaction_test2()
LANGUAGE plpgsql
AS $$
DECLARE
    r RECORD;
BEGIN
    FOR r IN SELECT * FROM test2 ORDER BY x LOOP
        INSERT INTO test1 (a) VALUES (r.x);
        COMMIT;
    END LOOP;
END;
$$;

CALL transaction_test2();

Normally, cursors are automatically closed at transaction commit. However, a cursor created as part of a loop like this is automatically converted to a holdable cursor by the first COMMIT or ROLLBACK. That means that the cursor is fully evaluated at the first COMMIT or ROLLBACK rather than row by row. The cursor is still removed automatically after the loop, so this is mostly invisible to the user.

Transaction commands are not allowed in cursor loops driven by commands that are not read-only (for example UPDATE ... RETURNING).

A transaction cannot be ended inside a block with exception handlers.

Prev | Up | Next
43.7. Cursors | Home | 43.9. Errors and Messages


43.9. Errors and Messages
Prev Up Chapter 43. PL/pgSQL — SQL Procedural Language Home Next

43.9. Errors and Messages

43.9.1. Reporting Errors and Messages
43.9.2. Checking Assertions

43.9.1. Reporting Errors and Messages

Use the RAISE statement to report messages and raise errors.

RAISE [ level ] 'format' [, expression [, ... ]] [ USING option = expression [, ... ] ];
RAISE [ level ] condition_name [ USING option = expression [, ... ] ];
RAISE [ level ] SQLSTATE 'sqlstate' [ USING option = expression [, ... ] ];
RAISE [ level ] USING option = expression [, ... ];
RAISE ;

The level option specifies the error severity. Allowed levels are DEBUG, LOG, INFO, NOTICE, WARNING, and EXCEPTION, with EXCEPTION being the default. EXCEPTION raises an error (which normally aborts the current transaction); the other levels only generate messages of different priority levels. Whether messages of a particular priority are reported to the client, written to the server log, or both is controlled by the log_min_messages and client_min_messages configuration variables. See Chapter 20 for more information.

After level if any, you can write a format (which must be a simple string literal, not an expression). The format string specifies the error message text to be reported. The format string can be followed by optional argument expressions to be inserted into the message. Inside the format string, % is replaced by the string representation of the next optional argument's value. Write %% to emit a literal %. The number of arguments must match the number of % placeholders in the format string, or an error is raised during the compilation of the function.

In this example, the value of v_job_id will replace the % in the string:

RAISE NOTICE 'Calling cs_create_job(%)', v_job_id;

You can attach additional information to the error report by writing USING followed by option = expression items. Each expression can be any string-valued expression. The allowed option key words are:

MESSAGE

Sets the error message text. This option can't be used in the form of RAISE that includes a format string before USING.

DETAIL

Supplies an error detail message.

HINT

Supplies a hint message.

ERRCODE

Specifies the error code (SQLSTATE) to report, either by condition name, as shown in Appendix A, or directly as a five-character SQLSTATE code.

COLUMN
CONSTRAINT
DATATYPE
TABLE
SCHEMA

Supplies the name of a related object.

This example will abort the transaction with the given error message and hint:

RAISE EXCEPTION 'Nonexistent ID --> %', user_id
      USING HINT = 'Please check your user ID';

These two examples show equivalent ways of setting the SQLSTATE:

RAISE 'Duplicate user ID: %', user_id USING ERRCODE = 'unique_violation';
RAISE 'Duplicate user ID: %', user_id USING ERRCODE = '23505';

There is a second RAISE syntax in which the main argument is the condition name or SQLSTATE to be reported, for example:

RAISE division_by_zero;
RAISE SQLSTATE '22012';

In this syntax, USING can be used to supply a custom error message, detail, or hint. Another way to do the earlier example is

RAISE unique_violation USING MESSAGE = 'Duplicate user ID: ' || user_id;

Still another variant is to write RAISE USING or RAISE level USING and put everything else into the USING list.

The last variant of RAISE has no parameters at all. This form can only be used inside a BEGIN block's EXCEPTION clause; it causes the error currently being handled to be re-thrown.

Note

Before PostgreSQL 9.1, RAISE without parameters was interpreted as re-throwing the error from the block containing the active exception handler. Thus an EXCEPTION clause nested within that handler could not catch it, even if the RAISE was within the nested EXCEPTION clause's block. This was deemed surprising as well as being incompatible with Oracle's PL/SQL.

If no condition name nor SQLSTATE is specified in a RAISE EXCEPTION command, the default is to use ERRCODE_RAISE_EXCEPTION (P0001). If no message text is specified, the default is to use the condition name or SQLSTATE as message text.

Note

When specifying an error code by SQLSTATE code, you are not limited to the predefined error codes, but can select any error code consisting of five digits and/or upper-case ASCII letters, other than 00000. It is recommended that you avoid throwing error codes that end in three zeroes, because these are category codes and can only be trapped by trapping the whole category.

43.9.2. Checking Assertions

The ASSERT statement is a convenient shorthand for inserting debugging checks into PL/pgSQL functions.

ASSERT condition [ , message ];

The condition is a Boolean expression that is expected to always evaluate to true; if it does, the ASSERT statement does nothing further. If the result is false or null, then an ASSERT_FAILURE exception is raised. (If an error occurs while evaluating the condition, it is reported as a normal error.)

If the optional message is provided, it is an expression whose result (if not null) replaces the default error message text “assertion failed”, should the condition fail. The message expression is not evaluated in the normal case where the assertion succeeds.

Testing of assertions can be enabled or disabled via the configuration parameter plpgsql.check_asserts, which takes a Boolean value; the default is on. If this parameter is off then ASSERT statements do nothing.

Note that ASSERT is meant for detecting program bugs, not for reporting ordinary error conditions. Use the RAISE statement, described above, for that.

Prev | Up | Next
43.8. Transaction Management | Home | 43.10. Trigger Functions


43.10. Trigger Functions
Prev Up Chapter 43. PL/pgSQL — SQL Procedural Language Home Next

43.10. Trigger Functions

43.10.1. Triggers on Data Changes
43.10.2. Triggers on Events

PL/pgSQL can be used to define trigger functions on data changes or database events. A trigger function is created with the CREATE FUNCTION command, declaring it as a function with no arguments and a return type of trigger (for data change triggers) or event_trigger (for database event triggers). Special local variables named TG_something are automatically defined to describe the condition that triggered the call.

43.10.1. Triggers on Data Changes

A data change trigger is declared as a function with no arguments and a return type of trigger. Note that the function must be declared with no arguments even if it expects to receive some arguments specified in CREATE TRIGGER — such arguments are passed via TG_ARGV, as described below.

When a PL/pgSQL function is called as a trigger, several special variables are created automatically in the top-level block. They are:

NEW

Data type RECORD; variable holding the new database row for INSERT/UPDATE operations in row-level triggers. This variable is null in statement-level triggers and for DELETE operations.

OLD

Data type RECORD; variable holding the old database row for UPDATE/DELETE operations in row-level triggers. This variable is null in statement-level triggers and for INSERT operations.

TG_NAME

Data type name; variable that contains the name of the trigger actually fired.

TG_WHEN

Data type text; a string of BEFORE, AFTER, or INSTEAD OF, depending on the trigger's definition.

TG_LEVEL

Data type text; a string of either ROW or STATEMENT depending on the trigger's definition.

TG_OP

Data type text; a string of INSERT, UPDATE, DELETE, or TRUNCATE telling for which operation the trigger was fired.

TG_RELID

Data type oid; the object ID of the table that caused the trigger invocation.

TG_RELNAME

Data type name; the name of the table that caused the trigger invocation. This is now deprecated, and could disappear in a future release. Use TG_TABLE_NAME instead.

TG_TABLE_NAME

Data type name; the name of the table that caused the trigger invocation.

TG_TABLE_SCHEMA

Data type name; the name of the schema of the table that caused the trigger invocation.

TG_NARGS

Data type integer; the number of arguments given to the trigger function in the CREATE TRIGGER statement.

TG_ARGV[]

Data type array of text; the arguments from the CREATE TRIGGER statement. The index counts from 0. Invalid indexes (less than 0 or greater than or equal to tg_nargs) result in a null value.

A trigger function must return either NULL or a record/row value having exactly the structure of the table the trigger was fired for.

Row-level triggers fired BEFORE can return null to signal the trigger manager to skip the rest of the operation for this row (i.e., subsequent triggers are not fired, and the INSERT/UPDATE/DELETE does not occur for this row). If a nonnull value is returned then the operation proceeds with that row value. Returning a row value different from the original value of NEW alters the row that will be inserted or updated. Thus, if the trigger function wants the triggering action to succeed normally without altering the row value, NEW (or a value equal thereto) has to be returned. To alter the row to be stored, it is possible to replace single values directly in NEW and return the modified NEW, or to build a complete new record/row to return. In the case of a before-trigger on DELETE, the returned value has no direct effect, but it has to be nonnull to allow the trigger action to proceed. Note that NEW is null in DELETE triggers, so returning that is usually not sensible. The usual idiom in DELETE triggers is to return OLD.

INSTEAD OF triggers (which are always row-level triggers, and may only be used on views) can return null to signal that they did not perform any updates, and that the rest of the operation for this row should be skipped (i.e., subsequent triggers are not fired, and the row is not counted in the rows-affected status for the surrounding INSERT/UPDATE/DELETE). Otherwise a nonnull value should be returned, to signal that the trigger performed the requested operation. For INSERT and UPDATE operations, the return value should be NEW, which the trigger function may modify to support INSERT RETURNING and UPDATE RETURNING (this will also affect the row value passed to any subsequent triggers, or passed to a special EXCLUDED alias reference within an INSERT statement with an ON CONFLICT DO UPDATE clause). For DELETE operations, the return value should be OLD.

The return value of a row-level trigger fired AFTER or a statement-level trigger fired BEFORE or AFTER is always ignored; it might as well be null. However, any of these types of triggers might still abort the entire operation by raising an error.

Example 43.3 shows an example of a trigger function in PL/pgSQL.

Example 43.3. A PL/pgSQL Trigger Function

This example trigger ensures that any time a row is inserted or updated in the table, the current user name and time are stamped into the row. And it checks that an employee's name is given and that the salary is a positive value.

CREATE TABLE emp (
    empname text,
    salary integer,
    last_date timestamp,
    last_user text
);

CREATE FUNCTION emp_stamp() RETURNS trigger AS $emp_stamp$
    BEGIN
        -- Check that empname and salary are given
        IF NEW.empname IS NULL THEN
            RAISE EXCEPTION 'empname cannot be null';
        END IF;
        IF NEW.salary IS NULL THEN
            RAISE EXCEPTION '% cannot have null salary', NEW.empname;
        END IF;

        -- Who works for us when they must pay for it?
        IF NEW.salary < 0 THEN
            RAISE EXCEPTION '% cannot have a negative salary', NEW.empname;
        END IF;

        -- Remember who changed the payroll when
        NEW.last_date := current_timestamp;
        NEW.last_user := current_user;
        RETURN NEW;
    END;
$emp_stamp$ LANGUAGE plpgsql;

CREATE TRIGGER emp_stamp BEFORE INSERT OR UPDATE ON emp
    FOR EACH ROW EXECUTE FUNCTION emp_stamp();

Another way to log changes to a table involves creating a new table that holds a row for each insert, update, or delete that occurs. This approach can be thought of as auditing changes to a table. Example 43.4 shows an example of an audit trigger function in PL/pgSQL.

Example 43.4. A PL/pgSQL Trigger Function for Auditing

This example trigger ensures that any insert, update or delete of a row in the emp table is recorded (i.e., audited) in the emp_audit table. The current time and user name are stamped into the row, together with the type of operation performed on it.

CREATE TABLE emp (
    empname           text NOT NULL,
    salary            integer
);

CREATE TABLE emp_audit(
    operation         char(1)   NOT NULL,
    stamp             timestamp NOT NULL,
    userid            text      NOT NULL,
    empname           text      NOT NULL,
    salary integer
);

CREATE OR REPLACE FUNCTION process_emp_audit() RETURNS TRIGGER AS $emp_audit$
    BEGIN
        --
        -- Create a row in emp_audit to reflect the operation performed on emp,
        -- making use of the special variable TG_OP to work out the operation.
        --
        IF (TG_OP = 'DELETE') THEN
            INSERT INTO emp_audit SELECT 'D', now(), user, OLD.*;
        ELSIF (TG_OP = 'UPDATE') THEN
            INSERT INTO emp_audit SELECT 'U', now(), user, NEW.*;
        ELSIF (TG_OP = 'INSERT') THEN
            INSERT INTO emp_audit SELECT 'I', now(), user, NEW.*;
        END IF;
        RETURN NULL; -- result is ignored since this is an AFTER trigger
    END;
$emp_audit$ LANGUAGE plpgsql;

CREATE TRIGGER emp_audit
AFTER INSERT OR UPDATE OR DELETE ON emp
    FOR EACH ROW EXECUTE FUNCTION process_emp_audit();

A variation of the previous example uses a view joining the main table to the audit table, to show when each entry was last modified. This approach still records the full audit trail of changes to the table, but also presents a simplified view of the audit trail, showing just the last modified timestamp derived from the audit trail for each entry. Example 43.5 shows an example of an audit trigger on a view in PL/pgSQL.

Example 43.5. A PL/pgSQL View Trigger Function for Auditing

This example uses a trigger on the view to make it updatable, and ensure that any insert, update or delete of a row in the view is recorded (i.e., audited) in the emp_audit table. The current time and user name are recorded, together with the type of operation performed, and the view displays the last modified time of each row.

CREATE TABLE emp (
    empname           text PRIMARY KEY,
    salary            integer
);

CREATE TABLE emp_audit(
    operation         char(1)   NOT NULL,
    userid            text      NOT NULL,
    empname           text      NOT NULL,
    salary            integer,
    stamp             timestamp NOT NULL
);

CREATE VIEW emp_view AS
    SELECT e.empname,
           e.salary,
           max(ea.stamp) AS last_updated
      FROM emp e
      LEFT JOIN emp_audit ea ON ea.empname = e.empname
     GROUP BY 1, 2;

CREATE OR REPLACE FUNCTION update_emp_view() RETURNS TRIGGER AS $$
    BEGIN
        --
        -- Perform the required operation on emp, and create a row in emp_audit
        -- to reflect the change made to emp.
        --
        IF (TG_OP = 'DELETE') THEN
            DELETE FROM emp WHERE empname = OLD.empname;
            IF NOT FOUND THEN RETURN NULL; END IF;

            OLD.last_updated = now();
            INSERT INTO emp_audit VALUES('D', user, OLD.*);
            RETURN OLD;
        ELSIF (TG_OP = 'UPDATE') THEN
            UPDATE emp SET salary = NEW.salary WHERE empname = OLD.empname;
            IF NOT FOUND THEN RETURN NULL; END IF;

            NEW.last_updated = now();
            INSERT INTO emp_audit VALUES('U', user, NEW.*);
            RETURN NEW;
        ELSIF (TG_OP = 'INSERT') THEN
            INSERT INTO emp VALUES(NEW.empname, NEW.salary);

            NEW.last_updated = now();
            INSERT INTO emp_audit VALUES('I', user, NEW.*);
            RETURN NEW;
        END IF;
    END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER emp_audit
INSTEAD OF INSERT OR UPDATE OR DELETE ON emp_view
    FOR EACH ROW EXECUTE FUNCTION update_emp_view();

One use of triggers is to maintain a summary table of another table. The resulting summary can be used in place of the original table for certain queries — often with vastly reduced run times. This technique is commonly used in Data Warehousing, where the tables of measured or observed data (called fact tables) might be extremely large. Example 43.6 shows an example of a trigger function in PL/pgSQL that maintains a summary table for a fact table in a data warehouse.

Example 43.6. A PL/pgSQL Trigger Function for Maintaining a Summary Table

The schema detailed here is partly based on the Grocery Store example from The Data Warehouse Toolkit by Ralph Kimball.

--
-- Main tables - time dimension and sales fact.
--
CREATE TABLE time_dimension (
    time_key                    integer NOT NULL,
    day_of_week                 integer NOT NULL,
    day_of_month                integer NOT NULL,
    month                       integer NOT NULL,
    quarter                     integer NOT NULL,
    year                        integer NOT NULL
);
CREATE UNIQUE INDEX time_dimension_key ON time_dimension(time_key);

CREATE TABLE sales_fact (
    time_key                    integer NOT NULL,
    product_key                 integer NOT NULL,
    store_key                   integer NOT NULL,
    amount_sold                 numeric(12,2) NOT NULL,
    units_sold                  integer NOT NULL,
    amount_cost                 numeric(12,2) NOT NULL
);
CREATE INDEX sales_fact_time ON sales_fact(time_key);

--
-- Summary table - sales by time.
--
CREATE TABLE sales_summary_bytime (
    time_key                    integer NOT NULL,
    amount_sold                 numeric(15,2) NOT NULL,
    units_sold                  numeric(12) NOT NULL,
    amount_cost                 numeric(15,2) NOT NULL
);
CREATE UNIQUE INDEX sales_summary_bytime_key ON sales_summary_bytime(time_key);

--
-- Function and trigger to amend summarized column(s) on UPDATE, INSERT, DELETE.
--
CREATE OR REPLACE FUNCTION maint_sales_summary_bytime() RETURNS TRIGGER
AS $maint_sales_summary_bytime$
    DECLARE
        delta_time_key          integer;
        delta_amount_sold       numeric(15,2);
        delta_units_sold        numeric(12);
        delta_amount_cost       numeric(15,2);
    BEGIN

        -- Work out the increment/decrement amount(s).
        IF (TG_OP = 'DELETE') THEN

            delta_time_key = OLD.time_key;
            delta_amount_sold = -1 * OLD.amount_sold;
            delta_units_sold = -1 * OLD.units_sold;
            delta_amount_cost = -1 * OLD.amount_cost;

        ELSIF (TG_OP = 'UPDATE') THEN

            -- forbid updates that change the time_key -
            -- (probably not too onerous, as DELETE + INSERT is how most
            -- changes will be made).
            IF ( OLD.time_key != NEW.time_key) THEN
                RAISE EXCEPTION 'Update of time_key : % -> % not allowed',
                                                      OLD.time_key, NEW.time_key;
            END IF;

            delta_time_key = OLD.time_key;
            delta_amount_sold = NEW.amount_sold - OLD.amount_sold;
            delta_units_sold = NEW.units_sold - OLD.units_sold;
            delta_amount_cost = NEW.amount_cost - OLD.amount_cost;

        ELSIF (TG_OP = 'INSERT') THEN

            delta_time_key = NEW.time_key;
            delta_amount_sold = NEW.amount_sold;
            delta_units_sold = NEW.units_sold;
            delta_amount_cost = NEW.amount_cost;

        END IF;

        -- Insert or update the summary row with the new values.
        <<insert_update>>
        LOOP
            UPDATE sales_summary_bytime
                SET amount_sold = amount_sold + delta_amount_sold,
                    units_sold = units_sold + delta_units_sold,
                    amount_cost = amount_cost + delta_amount_cost
                WHERE time_key = delta_time_key;

            EXIT insert_update WHEN found;

            BEGIN
                INSERT INTO sales_summary_bytime (
                            time_key,
                            amount_sold,
                            units_sold,
                            amount_cost)
                    VALUES (
                            delta_time_key,
                            delta_amount_sold,
                            delta_units_sold,
                            delta_amount_cost
                           );

                EXIT insert_update;

            EXCEPTION
                WHEN UNIQUE_VIOLATION THEN
                    -- do nothing
            END;
        END LOOP insert_update;

        RETURN NULL;

    END;
$maint_sales_summary_bytime$ LANGUAGE plpgsql;

CREATE TRIGGER maint_sales_summary_bytime
AFTER INSERT OR UPDATE OR DELETE ON sales_fact
    FOR EACH ROW EXECUTE FUNCTION maint_sales_summary_bytime();

INSERT INTO sales_fact VALUES(1,1,1,10,3,15);
INSERT INTO sales_fact VALUES(1,2,1,20,5,35);
INSERT INTO sales_fact VALUES(2,2,1,40,15,135);
INSERT INTO sales_fact VALUES(2,3,1,10,1,13);
SELECT * FROM sales_summary_bytime;
DELETE FROM sales_fact WHERE product_key = 1;
SELECT * FROM sales_summary_bytime;
UPDATE sales_fact SET units_sold = units_sold * 2;
SELECT * FROM sales_summary_bytime;

AFTER triggers can also make use of transition tables to inspect the entire set of rows changed by the triggering statement. The CREATE TRIGGER command assigns names to one or both transition tables, and then the function can refer to those names as though they were read-only temporary tables. Example 43.7 shows an example.

Example 43.7. Auditing with Transition Tables

This example produces the same results as Example 43.4, but instead of using a trigger that fires for every row, it uses a trigger that fires once per statement, after collecting the relevant information in a transition table. This can be significantly faster than the row-trigger approach when the invoking statement has modified many rows. Notice that we must make a separate trigger declaration for each kind of event, since the REFERENCING clauses must be different for each case. But this does not stop us from using a single trigger function if we choose. (In practice, it might be better to use three separate functions and avoid the run-time tests on TG_OP.)

CREATE TABLE emp (
    empname           text NOT NULL,
    salary            integer
);

CREATE TABLE emp_audit(
    operation         char(1)   NOT NULL,
    stamp             timestamp NOT NULL,
    userid            text      NOT NULL,
    empname           text      NOT NULL,
    salary integer
);

CREATE OR REPLACE FUNCTION process_emp_audit() RETURNS TRIGGER AS $emp_audit$
    BEGIN
        --
        -- Create rows in emp_audit to reflect the operations performed on emp,
        -- making use of the special variable TG_OP to work out the operation.
        --
        IF (TG_OP = 'DELETE') THEN
            INSERT INTO emp_audit
                SELECT 'D', now(), user, o.* FROM old_table o;
        ELSIF (TG_OP = 'UPDATE') THEN
            INSERT INTO emp_audit
                SELECT 'U', now(), user, n.* FROM new_table n;
        ELSIF (TG_OP = 'INSERT') THEN
            INSERT INTO emp_audit
                SELECT 'I', now(), user, n.* FROM new_table n;
        END IF;
        RETURN NULL; -- result is ignored since this is an AFTER trigger
    END;
$emp_audit$ LANGUAGE plpgsql;

CREATE TRIGGER emp_audit_ins
    AFTER INSERT ON emp
    REFERENCING NEW TABLE AS new_table
    FOR EACH STATEMENT EXECUTE FUNCTION process_emp_audit();
CREATE TRIGGER emp_audit_upd
    AFTER UPDATE ON emp
    REFERENCING OLD TABLE AS old_table NEW TABLE AS new_table
    FOR EACH STATEMENT EXECUTE FUNCTION process_emp_audit();
CREATE TRIGGER emp_audit_del
    AFTER DELETE ON emp
    REFERENCING OLD TABLE AS old_table
    FOR EACH STATEMENT EXECUTE FUNCTION process_emp_audit();

43.10.2. Triggers on Events

PL/pgSQL can be used to define event triggers. PostgreSQL requires that a function that is to be called as an event trigger must be declared as a function with no arguments and a return type of event_trigger.

When a PL/pgSQL function is called as an event trigger, several special variables are created automatically in the top-level block. They are:

TG_EVENT

Data type text; a string representing the event the trigger is fired for.

TG_TAG

Data type text; variable that contains the command tag for which the trigger is fired.

Example 43.8 shows an example of an event trigger function in PL/pgSQL.

Example 43.8. A PL/pgSQL Event Trigger Function

This example trigger simply raises a NOTICE message each time a supported command is executed.

CREATE OR REPLACE FUNCTION snitch() RETURNS event_trigger AS $$
BEGIN
    RAISE NOTICE 'snitch: % %', tg_event, tg_tag;
END;
$$ LANGUAGE plpgsql;

CREATE EVENT TRIGGER snitch ON ddl_command_start EXECUTE FUNCTION snitch();

Prev | Up | Next
43.9. Errors and Messages | Home | 43.11. PL/pgSQL under the Hood


43.11. PL/pgSQL under the Hood
Prev Up Chapter 43. PL/pgSQL — SQL Procedural Language Home Next

43.11. PL/pgSQL under the Hood

43.11.1. Variable Substitution
43.11.2. Plan Caching

This section discusses some implementation details that are frequently important for PL/pgSQL users to know.

43.11.1. Variable Substitution

SQL statements and expressions within a PL/pgSQL function can refer to variables and parameters of the function. Behind the scenes, PL/pgSQL substitutes query parameters for such references. Query parameters will only be substituted in places where they are syntactically permissible. As an extreme case, consider this example of poor programming style:

INSERT INTO foo (foo) VALUES (foo(foo));

The first occurrence of foo must syntactically be a table name, so it will not be substituted, even if the function has a variable named foo. The second occurrence must be the name of a column of that table, so it will not be substituted either. Likewise the third occurrence must be a function name, so it also will not be substituted for. Only the last occurrence is a candidate to be a reference to a variable of the PL/pgSQL function.

Another way to understand this is that variable substitution can only insert data values into an SQL command; it cannot dynamically change which database objects are referenced by the command. (If you want to do that, you must build a command string dynamically, as explained in Section 43.5.4.)

Since the names of variables are syntactically no different from the names of table columns, there can be ambiguity in statements that also refer to tables: is a given name meant to refer to a table column, or a variable? Let's change the previous example to

INSERT INTO dest (col) SELECT foo + bar FROM src;

Here, dest and src must be table names, and col must be a column of dest, but foo and bar might reasonably be either variables of the function or columns of src.

By default, PL/pgSQL will report an error if a name in an SQL statement could refer to either a variable or a table column. You can fix such a problem by renaming the variable or column, or by qualifying the ambiguous reference, or by telling PL/pgSQL which interpretation to prefer.

The simplest solution is to rename the variable or column. A common coding rule is to use a different naming convention for PL/pgSQL variables than you use for column names. For example, if you consistently name function variables v_something while none of your column names start with v_, no conflicts will occur.

Alternatively you can qualify ambiguous references to make them clear. In the above example, src.foo would be an unambiguous reference to the table column. To create an unambiguous reference to a variable, declare it in a labeled block and use the block's label (see Section 43.2). For example,

<<block>>
DECLARE
    foo int;
BEGIN
    foo := ...;
    INSERT INTO dest (col) SELECT block.foo + bar FROM src;

Here block.foo means the variable even if there is a column foo in src. Function parameters, as well as special variables such as FOUND, can be qualified by the function's name, because they are implicitly declared in an outer block labeled with the function's name.

Sometimes it is impractical to fix all the ambiguous references in a large body of PL/pgSQL code. In such cases you can specify that PL/pgSQL should resolve ambiguous references as the variable (which is compatible with PL/pgSQL's behavior before PostgreSQL 9.0), or as the table column (which is compatible with some other systems such as Oracle).

To change this behavior on a system-wide basis, set the configuration parameter plpgsql.variable_conflict to one of error, use_variable, or use_column (where error is the factory default). This parameter affects subsequent compilations of statements in PL/pgSQL functions, but not statements already compiled in the current session. Because changing this setting can cause unexpected changes in the behavior of PL/pgSQL functions, it can only be changed by a superuser.

You can also set the behavior on a function-by-function basis, by inserting one of these special commands at the start of the function text:

#variable_conflict error
#variable_conflict use_variable
#variable_conflict use_column

These commands affect only the function they are written in, and override the setting of plpgsql.variable_conflict. An example is

CREATE FUNCTION stamp_user(id int, comment text) RETURNS void AS $$
    #variable_conflict use_variable
    DECLARE
        curtime timestamp := now();
    BEGIN
        UPDATE users SET last_modified = curtime, comment = comment
          WHERE users.id = id;
    END;
$$ LANGUAGE plpgsql;

In the UPDATE command, curtime, comment, and id will refer to the function's variable and parameters whether or not users has columns of those names. Notice that we had to qualify the reference to users.id in the WHERE clause to make it refer to the table column. But we did not have to qualify the reference to comment as a target in the UPDATE list, because syntactically that must be a column of users. We could write the same function without depending on the variable_conflict setting in this way:

CREATE FUNCTION stamp_user(id int, comment text) RETURNS void AS $$
    <<fn>>
    DECLARE
        curtime timestamp := now();
    BEGIN
        UPDATE users SET last_modified = fn.curtime, comment = stamp_user.comment
          WHERE users.id = stamp_user.id;
    END;
$$ LANGUAGE plpgsql;

Variable substitution does not happen in a command string given to EXECUTE or one of its variants. If you need to insert a varying value into such a command, do so as part of constructing the string value, or use USING, as illustrated in Section 43.5.4.

Variable substitution currently works only in SELECT, INSERT, UPDATE, DELETE, and commands containing one of these (such as EXPLAIN and CREATE TABLE ... AS SELECT), because the main SQL engine allows query parameters only in these commands. To use a non-constant name or value in other statement types (generically called utility statements), you must construct the utility statement as a string and EXECUTE it.

43.11.2. Plan Caching

The PL/pgSQL interpreter parses the function's source text and produces an internal binary instruction tree the first time the function is called (within each session). The instruction tree fully translates the PL/pgSQL statement structure, but individual SQL expressions and SQL commands used in the function are not translated immediately.

As each expression and SQL command is first executed in the function, the PL/pgSQL interpreter parses and analyzes the command to create a prepared statement, using the SPI manager's SPI_prepare function. Subsequent visits to that expression or command reuse the prepared statement. Thus, a function with conditional code paths that are seldom visited will never incur the overhead of analyzing those commands that are never executed within the current session. A disadvantage is that errors in a specific expression or command cannot be detected until that part of the function is reached in execution. (Trivial syntax errors will be detected during the initial parsing pass, but anything deeper will not be detected until execution.)

PL/pgSQL (or more precisely, the SPI manager) can furthermore attempt to cache the execution plan associated with any particular prepared statement. If a cached plan is not used, then a fresh execution plan is generated on each visit to the statement, and the current parameter values (that is, PL/pgSQL variable values) can be used to optimize the selected plan. If the statement has no parameters, or is executed many times, the SPI manager will consider creating a generic plan that is not dependent on specific parameter values, and caching that for re-use. Typically this will happen only if the execution plan is not very sensitive to the values of the PL/pgSQL variables referenced in it. If it is, generating a plan each time is a net win. See PREPARE for more information about the behavior of prepared statements.

Because PL/pgSQL saves prepared statements and sometimes execution plans in this way, SQL commands that appear directly in a PL/pgSQL function must refer to the same tables and columns on every execution; that is, you cannot use a parameter as the name of a table or column in an SQL command. To get around this restriction, you can construct dynamic commands using the PL/pgSQL EXECUTE statement — at the price of performing new parse analysis and constructing a new execution plan on every execution.

The mutable nature of record variables presents another problem in this connection. When fields of a record variable are used in expressions or statements, the data types of the fields must not change from one call of the function to the next, since each expression will be analyzed using the data type that is present when the expression is first reached. EXECUTE can be used to get around this problem when necessary.

If the same function is used as a trigger for more than one table, PL/pgSQL prepares and caches statements independently for each such table — that is, there is a cache for each trigger function and table combination, not just for each function. This alleviates some of the problems with varying data types; for instance, a trigger function will be able to work successfully with a column named key even if it happens to have different types in different tables.

Likewise, functions having polymorphic argument types have a separate statement cache for each combination of actual argument types they have been invoked for, so that data type differences do not cause unexpected failures.

Statement caching can sometimes have surprising effects on the interpretation of time-sensitive values. For example there is a difference between what these two functions do:

CREATE FUNCTION logfunc1(logtxt text) RETURNS void AS $$
    BEGIN
        INSERT INTO logtable VALUES (logtxt, 'now');
    END;
$$ LANGUAGE plpgsql;

and:

CREATE FUNCTION logfunc2(logtxt text) RETURNS void AS $$
    DECLARE
        curtime timestamp;
    BEGIN
        curtime := 'now';
        INSERT INTO logtable VALUES (logtxt, curtime);
    END;
$$ LANGUAGE plpgsql;

In the case of logfunc1, the PostgreSQL main parser knows when analyzing the INSERT that the string 'now' should be interpreted as timestamp, because the target column of logtable is of that type. Thus, 'now' will be converted to a timestamp constant when the INSERT is analyzed, and then used in all invocations of logfunc1 during the lifetime of the session. Needless to say, this isn't what the programmer wanted. A better idea is to use the now() or current_timestamp function.

In the case of logfunc2, the PostgreSQL main parser does not know what type 'now' should become and therefore it returns a data value of type text containing the string now. During the ensuing assignment to the local variable curtime, the PL/pgSQL interpreter casts this string to the timestamp type by calling the textout and timestamp_in functions for the conversion. So, the computed time stamp is updated on each execution as the programmer expects. Even though this happens to work as expected, it's not terribly efficient, so use of the now() function would still be a better idea.

Prev | Up | Next
43.10. Trigger Functions | Home | 43.12. Tips for Developing in PL/pgSQL


43.12. Tips for Developing in PL/pgSQL
Prev Up Chapter 43. PL/pgSQL — SQL Procedural Language Home Next

43.12. Tips for Developing in PL/pgSQL

43.12.1. Handling of Quotation Marks
43.12.2. Additional Compile-Time and Run-Time Checks

One good way to develop in PL/pgSQL is to use the text editor of your choice to create your functions, and in another window, use psql to load and test those functions. If you are doing it this way, it is a good idea to write the function using CREATE OR REPLACE FUNCTION. That way you can just reload the file to update the function definition. For example:

CREATE OR REPLACE FUNCTION testfunc(integer) RETURNS integer AS $$
          ....
$$ LANGUAGE plpgsql;

While running psql, you can load or reload such a function definition file with:

\i filename.sql

and then immediately issue SQL commands to test the function.

Another good way to develop in PL/pgSQL is with a GUI database access tool that facilitates development in a procedural language. One example of such a tool is pgAdmin, although others exist. These tools often provide convenient features such as escaping single quotes and making it easier to recreate and debug functions.

43.12.1. Handling of Quotation Marks

The code of a PL/pgSQL function is specified in CREATE FUNCTION as a string literal. If you write the string literal in the ordinary way with surrounding single quotes, then any single quotes inside the function body must be doubled; likewise any backslashes must be doubled (assuming escape string syntax is used). Doubling quotes is at best tedious, and in more complicated cases the code can become downright incomprehensible, because you can easily find yourself needing half a dozen or more adjacent quote marks. It's recommended that you instead write the function body as a “dollar-quoted” string literal (see Section 4.1.2.4). In the dollar-quoting approach, you never double any quote marks, but instead take care to choose a different dollar-quoting delimiter for each level of nesting you need. For example, you might write the CREATE FUNCTION command as:

CREATE OR REPLACE FUNCTION testfunc(integer) RETURNS integer AS $PROC$
          ....
$PROC$ LANGUAGE plpgsql;

Within this, you might use quote marks for simple literal strings in SQL commands and $$ to delimit fragments of SQL commands that you are assembling as strings. If you need to quote text that includes $$, you could use $Q$, and so on.

The following chart shows what you have to do when writing quote marks without dollar quoting. It might be useful when translating pre-dollar quoting code into something more comprehensible.

1 quotation mark

To begin and end the function body, for example:

CREATE FUNCTION foo() RETURNS integer AS '
          ....
' LANGUAGE plpgsql;

Anywhere within a single-quoted function body, quote marks must appear in pairs.

2 quotation marks

For string literals inside the function body, for example:

a_output := ''Blah'';
SELECT * FROM users WHERE f_name=''foobar'';

In the dollar-quoting approach, you'd just write:

a_output := 'Blah';
SELECT * FROM users WHERE f_name='foobar';

which is exactly what the PL/pgSQL parser would see in either case.

4 quotation marks

When you need a single quotation mark in a string constant inside the function body, for example:

a_output := a_output || '' AND name LIKE ''''foobar'''' AND xyz''

The value actually appended to a_output would be: AND name LIKE 'foobar' AND xyz.

In the dollar-quoting approach, you'd write:

a_output := a_output || $$ AND name LIKE 'foobar' AND xyz$$

being careful that any dollar-quote delimiters around this are not just $$.

6 quotation marks

When a single quotation mark in a string inside the function body is adjacent to the end of that string constant, for example:

a_output := a_output || '' AND name LIKE ''''foobar''''''

The value appended to a_output would then be: AND name LIKE 'foobar'.

In the dollar-quoting approach, this becomes:

a_output := a_output || $$ AND name LIKE 'foobar'$$

10 quotation marks

When you want two single quotation marks in a string constant (which accounts for 8 quotation marks) and this is adjacent to the end of that string constant (2 more). You will probably only need that if you are writing a function that generates other functions, as in Example 43.10. For example:

a_output := a_output || '' if v_'' ||
    referrer_keys.kind || '' like ''''''''''
    || referrer_keys.key_string || ''''''''''
    then return ''''''  || referrer_keys.referrer_type
    || ''''''; end if;'';

The value of a_output would then be:

if v_... like ''...'' then return ''...''; end if;

In the dollar-quoting approach, this becomes:

a_output := a_output || $$ if v_$$ || referrer_keys.kind || $$ like '$$
    || referrer_keys.key_string || $$'
    then return '$$  || referrer_keys.referrer_type
    || $$'; end if;$$;

where we assume we only need to put single quote marks into a_output, because it will be re-quoted before use.

43.12.2. Additional Compile-Time and Run-Time Checks

To aid the user in finding instances of simple but common problems before they cause harm, PL/pgSQL provides additional checks. When enabled, depending on the configuration, they can be used to emit either a WARNING or an ERROR during the compilation of a function. A function which has received a WARNING can be executed without producing further messages, so you are advised to test in a separate development environment.

Setting plpgsql.extra_warnings, or plpgsql.extra_errors, as appropriate, to "all" is encouraged in development and/or testing environments.

These additional checks are enabled through the configuration variables plpgsql.extra_warnings for warnings and plpgsql.extra_errors for errors. Both can be set either to a comma-separated list of checks, "none" or "all". The default is "none". Currently the list of available checks includes:

shadowed_variables

Checks if a declaration shadows a previously defined variable.

strict_multi_assignment

Some PL/PgSQL commands allow assigning values to more than one variable at a time, such as SELECT INTO. Typically, the number of target variables and the number of source variables should match, though PL/PgSQL will use NULL for missing values and extra variables are ignored. Enabling this check will cause PL/PgSQL to throw a WARNING or ERROR whenever the number of target variables and the number of source variables are different.

too_many_rows

Enabling this check will cause PL/PgSQL to check if a given query returns more than one row when an INTO clause is used. As an INTO statement will only ever use one row, having a query return multiple rows is generally either inefficient and/or nondeterministic and therefore is likely an error.

The following example shows the effect of plpgsql.extra_warnings set to shadowed_variables:

SET plpgsql.extra_warnings TO 'shadowed_variables';

CREATE FUNCTION foo(f1 int) RETURNS int AS $$
DECLARE
f1 int;
BEGIN
RETURN f1;
END;
$$ LANGUAGE plpgsql;
WARNING:  variable "f1" shadows a previously defined variable
LINE 3: f1 int;
        ^
CREATE FUNCTION

The below example shows the effects of setting plpgsql.extra_warnings to strict_multi_assignment:

SET plpgsql.extra_warnings TO 'strict_multi_assignment';

CREATE OR REPLACE FUNCTION public.foo()
 RETURNS void
 LANGUAGE plpgsql
AS $$
DECLARE
  x int;
  y int;
BEGIN
  SELECT 1 INTO x, y;
  SELECT 1, 2 INTO x, y;
  SELECT 1, 2, 3 INTO x, y;
END;
$$;

SELECT foo();
WARNING:  number of source and target fields in assignment does not match
DETAIL:  strict_multi_assignment check of extra_warnings is active.
HINT:  Make sure the query returns the exact list of columns.
WARNING:  number of source and target fields in assignment does not match
DETAIL:  strict_multi_assignment check of extra_warnings is active.
HINT:  Make sure the query returns the exact list of columns.

 foo
-----

(1 row)

Prev | Up | Next
43.11. PL/pgSQL under the Hood | Home | 43.13. Porting from Oracle PL/SQL


43.13. Porting from Oracle PL/SQL
Prev Up Chapter 43. PL/pgSQL — SQL Procedural Language Home Next

43.13. Porting from Oracle PL/SQL

43.13.1. Porting Examples
43.13.2. Other Things to Watch For
43.13.3. Appendix

This section explains differences between PostgreSQL's PL/pgSQL language and Oracle's PL/SQL language, to help developers who port applications from Oracle® to PostgreSQL.

PL/pgSQL is similar to PL/SQL in many aspects. It is a block-structured, imperative language, and all variables have to be declared. Assignments, loops, and conditionals are similar. The main differences you should keep in mind when porting from PL/SQL to PL/pgSQL are:

  • If a name used in an SQL command could be either a column name of a table used in the command or a reference to a variable of the function, PL/SQL treats it as a column name. By default, PL/pgSQL will throw an error complaining that the name is ambiguous. You can specify plpgsql.variable_conflict = use_column to change this behavior to match PL/SQL, as explained in Section 43.11.1. It's often best to avoid such ambiguities in the first place, but if you have to port a large amount of code that depends on this behavior, setting variable_conflict may be the best solution.

  • In PostgreSQL the function body must be written as a string literal. Therefore you need to use dollar quoting or escape single quotes in the function body. (See Section 43.12.1.)

  • Data type names often need translation. For example, in Oracle string values are commonly declared as being of type varchar2, which is a non-SQL-standard type. In PostgreSQL, use type varchar or text instead. Similarly, replace type number with numeric, or use some other numeric data type if there's a more appropriate one.

  • Instead of packages, use schemas to organize your functions into groups.

  • Since there are no packages, there are no package-level variables either. This is somewhat annoying. You can keep per-session state in temporary tables instead.

  • Integer FOR loops with REVERSE work differently: PL/SQL counts down from the second number to the first, while PL/pgSQL counts down from the first number to the second, requiring the loop bounds to be swapped when porting. This incompatibility is unfortunate but is unlikely to be changed. (See Section 43.6.5.5.)

  • FOR loops over queries (other than cursors) also work differently: the target variable(s) must have been declared, whereas PL/SQL always declares them implicitly. An advantage of this is that the variable values are still accessible after the loop exits.

  • There are various notational differences for the use of cursor variables.

43.13.1. Porting Examples

Example 43.9 shows how to port a simple function from PL/SQL to PL/pgSQL.

Example 43.9. Porting a Simple Function from PL/SQL to PL/pgSQL

Here is an Oracle PL/SQL function:

CREATE OR REPLACE FUNCTION cs_fmt_browser_version(v_name varchar2,
                                                  v_version varchar2)
RETURN varchar2 IS
BEGIN
    IF v_version IS NULL THEN
        RETURN v_name;
    END IF;
    RETURN v_name || '/' || v_version;
END;
/
show errors;

Let's go through this function and see the differences compared to PL/pgSQL:

  • The type name varchar2 has to be changed to varchar or text. In the examples in this section, we'll use varchar, but text is often a better choice if you do not need specific string length limits.

  • The RETURN key word in the function prototype (not the function body) becomes RETURNS in PostgreSQL. Also, IS becomes AS, and you need to add a LANGUAGE clause because PL/pgSQL is not the only possible function language.

  • In PostgreSQL, the function body is considered to be a string literal, so you need to use quote marks or dollar quotes around it. This substitutes for the terminating / in the Oracle approach.

  • The show errors command does not exist in PostgreSQL, and is not needed since errors are reported automatically.

This is how this function would look when ported to PostgreSQL:

CREATE OR REPLACE FUNCTION cs_fmt_browser_version(v_name varchar,
                                                  v_version varchar)
RETURNS varchar AS $$
BEGIN
    IF v_version IS NULL THEN
        RETURN v_name;
    END IF;
    RETURN v_name || '/' || v_version;
END;
$$ LANGUAGE plpgsql;

Example 43.10 shows how to port a function that creates another function and how to handle the ensuing quoting problems.

Example 43.10. Porting a Function that Creates Another Function from PL/SQL to PL/pgSQL

The following procedure grabs rows from a SELECT statement and builds a large function with the results in IF statements, for the sake of efficiency.

This is the Oracle version:

CREATE OR REPLACE PROCEDURE cs_update_referrer_type_proc IS
    CURSOR referrer_keys IS
        SELECT * FROM cs_referrer_keys
        ORDER BY try_order;
    func_cmd VARCHAR(4000);
BEGIN
    func_cmd := 'CREATE OR REPLACE FUNCTION cs_find_referrer_type(v_host IN VARCHAR2,
                 v_domain IN VARCHAR2, v_url IN VARCHAR2) RETURN VARCHAR2 IS BEGIN';

    FOR referrer_key IN referrer_keys LOOP
        func_cmd := func_cmd ||
          ' IF v_' || referrer_key.kind
          || ' LIKE ''' || referrer_key.key_string
          || ''' THEN RETURN ''' || referrer_key.referrer_type
          || '''; END IF;';
    END LOOP;

    func_cmd := func_cmd || ' RETURN NULL; END;';

    EXECUTE IMMEDIATE func_cmd;
END;
/
show errors;

Here is how this function would end up in PostgreSQL:

CREATE OR REPLACE PROCEDURE cs_update_referrer_type_proc() AS $func$
DECLARE
    referrer_keys CURSOR IS
        SELECT * FROM cs_referrer_keys
        ORDER BY try_order;
    func_body text;
    func_cmd text;
BEGIN
    func_body := 'BEGIN';

    FOR referrer_key IN referrer_keys LOOP
        func_body := func_body ||
          ' IF v_' || referrer_key.kind
          || ' LIKE ' || quote_literal(referrer_key.key_string)
          || ' THEN RETURN ' || quote_literal(referrer_key.referrer_type)
          || '; END IF;' ;
    END LOOP;

    func_body := func_body || ' RETURN NULL; END;';

    func_cmd :=
      'CREATE OR REPLACE FUNCTION cs_find_referrer_type(v_host varchar,
                                                        v_domain varchar,
                                                        v_url varchar)
        RETURNS varchar AS '
      || quote_literal(func_body)
      || ' LANGUAGE plpgsql;' ;

    EXECUTE func_cmd;
END;
$func$ LANGUAGE plpgsql;

Notice how the body of the function is built separately and passed through quote_literal to double any quote marks in it. This technique is needed because we cannot safely use dollar quoting for defining the new function: we do not know for sure what strings will be interpolated from the referrer_key.key_string field. (We are assuming here that referrer_key.kind can be trusted to always be host, domain, or url, but referrer_key.key_string might be anything, in particular it might contain dollar signs.) This function is actually an improvement on the Oracle original, because it will not generate broken code when referrer_key.key_string or referrer_key.referrer_type contain quote marks.

Example 43.11 shows how to port a function with OUT parameters and string manipulation. PostgreSQL does not have a built-in instr function, but you can create one using a combination of other functions. In Section 43.13.3 there is a PL/pgSQL implementation of instr that you can use to make your porting easier.

Example 43.11. Porting a Procedure With String Manipulation and OUT Parameters from PL/SQL to PL/pgSQL

The following Oracle PL/SQL procedure is used to parse a URL and return several elements (host, path, and query).

This is the Oracle version:

CREATE OR REPLACE PROCEDURE cs_parse_url(
    v_url IN VARCHAR2,
    v_host OUT VARCHAR2,  -- This will be passed back
    v_path OUT VARCHAR2,  -- This one too
    v_query OUT VARCHAR2) -- And this one
IS
    a_pos1 INTEGER;
    a_pos2 INTEGER;
BEGIN
    v_host := NULL;
    v_path := NULL;
    v_query := NULL;
    a_pos1 := instr(v_url, '//');

    IF a_pos1 = 0 THEN
        RETURN;
    END IF;
    a_pos2 := instr(v_url, '/', a_pos1 + 2);
    IF a_pos2 = 0 THEN
        v_host := substr(v_url, a_pos1 + 2);
        v_path := '/';
        RETURN;
    END IF;

    v_host := substr(v_url, a_pos1 + 2, a_pos2 - a_pos1 - 2);
    a_pos1 := instr(v_url, '?', a_pos2 + 1);

    IF a_pos1 = 0 THEN
        v_path := substr(v_url, a_pos2);
        RETURN;
    END IF;

    v_path := substr(v_url, a_pos2, a_pos1 - a_pos2);
    v_query := substr(v_url, a_pos1 + 1);
END;
/
show errors;

Here is a possible translation into PL/pgSQL:

CREATE OR REPLACE FUNCTION cs_parse_url(
    v_url IN VARCHAR,
    v_host OUT VARCHAR,  -- This will be passed back
    v_path OUT VARCHAR,  -- This one too
    v_query OUT VARCHAR) -- And this one
AS $$
DECLARE
    a_pos1 INTEGER;
    a_pos2 INTEGER;
BEGIN
    v_host := NULL;
    v_path := NULL;
    v_query := NULL;
    a_pos1 := instr(v_url, '//');

    IF a_pos1 = 0 THEN
        RETURN;
    END IF;
    a_pos2 := instr(v_url, '/', a_pos1 + 2);
    IF a_pos2 = 0 THEN
        v_host := substr(v_url, a_pos1 + 2);
        v_path := '/';
        RETURN;
    END IF;

    v_host := substr(v_url, a_pos1 + 2, a_pos2 - a_pos1 - 2);
    a_pos1 := instr(v_url, '?', a_pos2 + 1);

    IF a_pos1 = 0 THEN
        v_path := substr(v_url, a_pos2);
        RETURN;
    END IF;

    v_path := substr(v_url, a_pos2, a_pos1 - a_pos2);
    v_query := substr(v_url, a_pos1 + 1);
END;
$$ LANGUAGE plpgsql;

This function could be used like this:

SELECT * FROM cs_parse_url('http://foobar.com/query.cgi?baz');

Example 43.12 shows how to port a procedure that uses numerous features that are specific to Oracle.

Example 43.12. Porting a Procedure from PL/SQL to PL/pgSQL

The Oracle version:

CREATE OR REPLACE PROCEDURE cs_create_job(v_job_id IN INTEGER) IS
    a_running_job_count INTEGER;
BEGIN
    LOCK TABLE cs_jobs IN EXCLUSIVE MODE;

    SELECT count(*) INTO a_running_job_count FROM cs_jobs WHERE end_stamp IS NULL;

    IF a_running_job_count > 0 THEN
        COMMIT; -- free lock
        raise_application_error(-20000,
                 'Unable to create a new job: a job is currently running.');
    END IF;

    DELETE FROM cs_active_job;
    INSERT INTO cs_active_job(job_id) VALUES (v_job_id);

    BEGIN
        INSERT INTO cs_jobs (job_id, start_stamp) VALUES (v_job_id, now());
    EXCEPTION
        WHEN dup_val_on_index THEN NULL; -- don't worry if it already exists
    END;
    COMMIT;
END;
/
show errors

This is how we could port this procedure to PL/pgSQL:

CREATE OR REPLACE PROCEDURE cs_create_job(v_job_id integer) AS $$
DECLARE
    a_running_job_count integer;
BEGIN
    LOCK TABLE cs_jobs IN EXCLUSIVE MODE;

    SELECT count(*) INTO a_running_job_count FROM cs_jobs WHERE end_stamp IS NULL;

    IF a_running_job_count > 0 THEN
        COMMIT; -- free lock
        RAISE EXCEPTION 'Unable to create a new job: a job is currently running'; -- (1)
    END IF;

    DELETE FROM cs_active_job;
    INSERT INTO cs_active_job(job_id) VALUES (v_job_id);

    BEGIN
        INSERT INTO cs_jobs (job_id, start_stamp) VALUES (v_job_id, now());
    EXCEPTION
        WHEN unique_violation THEN -- (2)
            -- don't worry if it already exists
    END;
    COMMIT;
END;
$$ LANGUAGE plpgsql;

(1) | The syntax of RAISE is considerably different from Oracle's statement, although the basic case RAISE exception_name works similarly.
(2) | The exception names supported by PL/pgSQL are different from Oracle's. The set of built-in exception names is much larger (see Appendix A). There is not currently a way to declare user-defined exception names, although you can throw user-chosen SQLSTATE values instead.

43.13.2. Other Things to Watch For

This section explains a few other things to watch for when porting Oracle PL/SQL functions to PostgreSQL.

43.13.2.1. Implicit Rollback after Exceptions

In PL/pgSQL, when an exception is caught by an EXCEPTION clause, all database changes since the block's BEGIN are automatically rolled back. That is, the behavior is equivalent to what you'd get in Oracle with:

BEGIN
    SAVEPOINT s1;
    ... code here ...
EXCEPTION
    WHEN ... THEN
        ROLLBACK TO s1;
        ... code here ...
    WHEN ... THEN
        ROLLBACK TO s1;
        ... code here ...
END;

If you are translating an Oracle procedure that uses SAVEPOINT and ROLLBACK TO in this style, your task is easy: just omit the SAVEPOINT and ROLLBACK TO. If you have a procedure that uses SAVEPOINT and ROLLBACK TO in a different way then some actual thought will be required.

43.13.2.2. EXECUTE

The PL/pgSQL version of EXECUTE works similarly to the PL/SQL version, but you have to remember to use quote_literal and quote_ident as described in Section 43.5.4. Constructs of the type EXECUTE 'SELECT * FROM $1'; will not work reliably unless you use these functions.

43.13.2.3. Optimizing PL/pgSQL Functions

PostgreSQL gives you two function creation modifiers to optimize execution: “volatility” (whether the function always returns the same result when given the same arguments) and “strictness” (whether the function returns null if any argument is null). Consult the CREATE FUNCTION reference page for details.

When making use of these optimization attributes, your CREATE FUNCTION statement might look something like this:

CREATE FUNCTION foo(...) RETURNS integer AS $$
...
$$ LANGUAGE plpgsql STRICT IMMUTABLE;

43.13.3. Appendix

This section contains the code for a set of Oracle-compatible instr functions that you can use to simplify your porting efforts.

--
-- instr functions that mimic Oracle's counterpart
-- Syntax: instr(string1, string2 [, n [, m]])
-- where [] denotes optional parameters.
--
-- Search string1, beginning at the nth character, for the mth occurrence
-- of string2.  If n is negative, search backwards, starting at the abs(n)'th
-- character from the end of string1.
-- If n is not passed, assume 1 (search starts at first character).
-- If m is not passed, assume 1 (find first occurrence).
-- Returns starting index of string2 in string1, or 0 if string2 is not found.
--

CREATE FUNCTION instr(varchar, varchar) RETURNS integer AS $$
BEGIN
    RETURN instr($1, $2, 1);
END;
$$ LANGUAGE plpgsql STRICT IMMUTABLE;

CREATE FUNCTION instr(string varchar, string_to_search_for varchar,
                      beg_index integer)
RETURNS integer AS $$
DECLARE
    pos integer NOT NULL DEFAULT 0;
    temp_str varchar;
    beg integer;
    length integer;
    ss_length integer;
BEGIN
    IF beg_index > 0 THEN
        temp_str := substring(string FROM beg_index);
        pos := position(string_to_search_for IN temp_str);

        IF pos = 0 THEN
            RETURN 0;
        ELSE
            RETURN pos + beg_index - 1;
        END IF;
    ELSIF beg_index < 0 THEN
        ss_length := char_length(string_to_search_for);
        length := char_length(string);
        beg := length + 1 + beg_index;

        WHILE beg > 0 LOOP
            temp_str := substring(string FROM beg FOR ss_length);
            IF string_to_search_for = temp_str THEN
                RETURN beg;
            END IF;

            beg := beg - 1;
        END LOOP;

        RETURN 0;
    ELSE
        RETURN 0;
    END IF;
END;
$$ LANGUAGE plpgsql STRICT IMMUTABLE;

CREATE FUNCTION instr(string varchar, string_to_search_for varchar,
                      beg_index integer, occur_index integer)
RETURNS integer AS $$
DECLARE
    pos integer NOT NULL DEFAULT 0;
    occur_number integer NOT NULL DEFAULT 0;
    temp_str varchar;
    beg integer;
    i integer;
    length integer;
    ss_length integer;
BEGIN
    IF occur_index <= 0 THEN
        RAISE 'argument ''%'' is out of range', occur_index
          USING ERRCODE = '22003';
    END IF;

    IF beg_index > 0 THEN
        beg := beg_index - 1;
        FOR i IN 1..occur_index LOOP
            temp_str := substring(string FROM beg + 1);
            pos := position(string_to_search_for IN temp_str);
            IF pos = 0 THEN
                RETURN 0;
            END IF;
            beg := beg + pos;
        END LOOP;

        RETURN beg;
    ELSIF beg_index < 0 THEN
        ss_length := char_length(string_to_search_for);
        length := char_length(string);
        beg := length + 1 + beg_index;

        WHILE beg > 0 LOOP
            temp_str := substring(string FROM beg FOR ss_length);
            IF string_to_search_for = temp_str THEN
                occur_number := occur_number + 1;
                IF occur_number = occur_index THEN
                    RETURN beg;
                END IF;
            END IF;

            beg := beg - 1;
        END LOOP;

        RETURN 0;
    ELSE
        RETURN 0;
    END IF;
END;
$$ LANGUAGE plpgsql STRICT IMMUTABLE;

Prev | Up | Next
43.12. Tips for Developing in PL/pgSQL | Home | Chapter 44. PL/Tcl — Tcl Procedural Language


Chapter 44. PL/Tcl — Tcl Procedural Language
Prev Up Part V. Server Programming Home Next

Chapter 44. PL/Tcl — Tcl Procedural Language

Table of Contents

44.1. Overview
44.2. PL/Tcl Functions and Arguments
44.3. Data Values in PL/Tcl
44.4. Global Data in PL/Tcl
44.5. Database Access from PL/Tcl
44.6. Trigger Functions in PL/Tcl
44.7. Event Trigger Functions in PL/Tcl
44.8. Error Handling in PL/Tcl
44.9. Explicit Subtransactions in PL/Tcl
44.10. Transaction Management
44.11. PL/Tcl Configuration
44.12. Tcl Procedure Names

PL/Tcl is a loadable procedural language for the PostgreSQL database system that enables the Tcl language to be used to write PostgreSQL functions and procedures.

Prev | Up | Next
43.13. Porting from Oracle PL/SQL | Home | 44.1. Overview


44.1. Overview
Prev Up Chapter 44. PL/Tcl — Tcl Procedural Language Home Next

44.1. Overview

PL/Tcl offers most of the capabilities a function writer has in the C language, with a few restrictions, and with the addition of the powerful string processing libraries that are available for Tcl.

One compelling good restriction is that everything is executed from within the safety of the context of a Tcl interpreter. In addition to the limited command set of safe Tcl, only a few commands are available to access the database via SPI and to raise messages via elog(). PL/Tcl provides no way to access internals of the database server or to gain OS-level access under the permissions of the PostgreSQL server process, as a C function can do. Thus, unprivileged database users can be trusted to use this language; it does not give them unlimited authority.

The other notable implementation restriction is that Tcl functions cannot be used to create input/output functions for new data types.

Sometimes it is desirable to write Tcl functions that are not restricted to safe Tcl. For example, one might want a Tcl function that sends email. To handle these cases, there is a variant of PL/Tcl called PL/TclU (for untrusted Tcl). This is exactly the same language except that a full Tcl interpreter is used. If PL/TclU is used, it must be installed as an untrusted procedural language so that only database superusers can create functions in it. The writer of a PL/TclU function must take care that the function cannot be used to do anything unwanted, since it will be able to do anything that could be done by a user logged in as the database administrator.

The shared object code for the PL/Tcl and PL/TclU call handlers is automatically built and installed in the PostgreSQL library directory if Tcl support is specified in the configuration step of the installation procedure. To install PL/Tcl and/or PL/TclU in a particular database, use the CREATE EXTENSION command, for example CREATE EXTENSION pltcl or CREATE EXTENSION pltclu.

Prev | Up | Next
Chapter 44. PL/Tcl — Tcl Procedural Language | Home | 44.2. PL/Tcl Functions and Arguments


44.2. PL/Tcl Functions and Arguments
Prev Up Chapter 44. PL/Tcl — Tcl Procedural Language Home Next

44.2. PL/Tcl Functions and Arguments

To create a function in the PL/Tcl language, use the standard CREATE FUNCTION syntax:

CREATE FUNCTION funcname (argument-types) RETURNS return-type AS $$
    # PL/Tcl function body
$$ LANGUAGE pltcl;

PL/TclU is the same, except that the language has to be specified as pltclu.

The body of the function is simply a piece of Tcl script. When the function is called, the argument values are passed to the Tcl script as variables named 1 ... n. The result is returned from the Tcl code in the usual way, with a return statement. In a procedure, the return value from the Tcl code is ignored.

For example, a function returning the greater of two integer values could be defined as:

CREATE FUNCTION tcl_max(integer, integer) RETURNS integer AS $$
    if {$1 > $2} {return $1}
    return $2
$$ LANGUAGE pltcl STRICT;

Note the clause STRICT, which saves us from having to think about null input values: if a null value is passed, the function will not be called at all, but will just return a null result automatically.

In a nonstrict function, if the actual value of an argument is null, the corresponding $n variable will be set to an empty string. To detect whether a particular argument is null, use the function argisnull. For example, suppose that we wanted tcl_max with one null and one nonnull argument to return the nonnull argument, rather than null:

CREATE FUNCTION tcl_max(integer, integer) RETURNS integer AS $$
    if {[argisnull 1]} {
        if {[argisnull 2]} { return_null }
        return $2
    }
    if {[argisnull 2]} { return $1 }
    if {$1 > $2} {return $1}
    return $2
$$ LANGUAGE pltcl;

As shown above, to return a null value from a PL/Tcl function, execute return_null. This can be done whether the function is strict or not.

Composite-type arguments are passed to the function as Tcl arrays. The element names of the array are the attribute names of the composite type. If an attribute in the passed row has the null value, it will not appear in the array. Here is an example:

CREATE TABLE employee (
    name text,
    salary integer,
    age integer
);

CREATE FUNCTION overpaid(employee) RETURNS boolean AS $$
    if {200000.0 < $1(salary)} {
        return "t"
    }
    if {$1(age) < 30 && 100000.0 < $1(salary)} {
        return "t"
    }
    return "f"
$$ LANGUAGE pltcl;

PL/Tcl functions can return composite-type results, too. To do this, the Tcl code must return a list of column name/value pairs matching the expected result type. Any column names omitted from the list are returned as nulls, and an error is raised if there are unexpected column names. Here is an example:

CREATE FUNCTION square_cube(in int, out squared int, out cubed int) AS $$
    return [list squared [expr {$1 * $1}] cubed [expr {$1 * $1 * $1}]]
$$ LANGUAGE pltcl;

Output arguments of procedures are returned in the same way, for example:

CREATE PROCEDURE tcl_triple(INOUT a integer, INOUT b integer) AS $$
    return [list a [expr {$1 * 3}] b [expr {$2 * 3}]]
$$ LANGUAGE pltcl;

CALL tcl_triple(5, 10);

Tip

The result list can be made from an array representation of the desired tuple with the array get Tcl command. For example:

CREATE FUNCTION raise_pay(employee, delta int) RETURNS employee AS $$
    set 1(salary) [expr {$1(salary) + $2}]
    return [array get 1]
$$ LANGUAGE pltcl;

PL/Tcl functions can return sets. To do this, the Tcl code should call return_next once per row to be returned, passing either the appropriate value when returning a scalar type, or a list of column name/value pairs when returning a composite type. Here is an example returning a scalar type:

CREATE FUNCTION sequence(int, int) RETURNS SETOF int AS $$
    for {set i $1} {$i < $2} {incr i} {
        return_next $i
    }
$$ LANGUAGE pltcl;

and here is one returning a composite type:

CREATE FUNCTION table_of_squares(int, int) RETURNS TABLE (x int, x2 int) AS $$
    for {set i $1} {$i < $2} {incr i} {
        return_next [list x $i x2 [expr {$i * $i}]]
    }
$$ LANGUAGE pltcl;

Prev | Up | Next
44.1. Overview | Home | 44.3. Data Values in PL/Tcl


44.3. Data Values in PL/Tcl
Prev Up Chapter 44. PL/Tcl — Tcl Procedural Language Home Next

44.3. Data Values in PL/Tcl

The argument values supplied to a PL/Tcl function's code are simply the input arguments converted to text form (just as if they had been displayed by a SELECT statement). Conversely, the return and return_next commands will accept any string that is acceptable input format for the function's declared result type, or for the specified column of a composite result type.

Prev | Up | Next
44.2. PL/Tcl Functions and Arguments | Home | 44.4. Global Data in PL/Tcl


44.4. Global Data in PL/Tcl
Prev Up Chapter 44. PL/Tcl — Tcl Procedural Language Home Next

44.4. Global Data in PL/Tcl

Sometimes it is useful to have some global data that is held between two calls to a function or is shared between different functions. This is easily done in PL/Tcl, but there are some restrictions that must be understood.

For security reasons, PL/Tcl executes functions called by any one SQL role in a separate Tcl interpreter for that role. This prevents accidental or malicious interference by one user with the behavior of another user's PL/Tcl functions. Each such interpreter will have its own values for any “global” Tcl variables. Thus, two PL/Tcl functions will share the same global variables if and only if they are executed by the same SQL role. In an application wherein a single session executes code under multiple SQL roles (via SECURITY DEFINER functions, use of SET ROLE, etc) you may need to take explicit steps to ensure that PL/Tcl functions can share data. To do that, make sure that functions that should communicate are owned by the same user, and mark them SECURITY DEFINER. You must of course take care that such functions can't be used to do anything unintended.

All PL/TclU functions used in a session execute in the same Tcl interpreter, which of course is distinct from the interpreter(s) used for PL/Tcl functions. So global data is automatically shared between PL/TclU functions. This is not considered a security risk because all PL/TclU functions execute at the same trust level, namely that of a database superuser.

To help protect PL/Tcl functions from unintentionally interfering with each other, a global array is made available to each function via the upvar command. The global name of this variable is the function's internal name, and the local name is GD. It is recommended that GD be used for persistent private data of a function. Use regular Tcl global variables only for values that you specifically intend to be shared among multiple functions. (Note that the GD arrays are only global within a particular interpreter, so they do not bypass the security restrictions mentioned above.)

An example of using GD appears in the spi_execp example below.

Prev | Up | Next
44.3. Data Values in PL/Tcl | Home | 44.5. Database Access from PL/Tcl


44.5. Database Access from PL/Tcl
Prev Up Chapter 44. PL/Tcl — Tcl Procedural Language Home Next

44.5. Database Access from PL/Tcl

In this section, we follow the usual Tcl convention of using question marks, rather than brackets, to indicate an optional element in a syntax synopsis. The following commands are available to access the database from the body of a PL/Tcl function:

spi_exec ?-count n? ?-array name? command ?loop-body?

Executes an SQL command given as a string. An error in the command causes an error to be raised. Otherwise, the return value of spi_exec is the number of rows processed (selected, inserted, updated, or deleted) by the command, or zero if the command is a utility statement. In addition, if the command is a SELECT statement, the values of the selected columns are placed in Tcl variables as described below.

The optional -count value tells spi_exec the maximum number of rows to process in the command. The effect of this is comparable to setting up a query as a cursor and then saying FETCH n.

If the command is a SELECT statement, the values of the result columns are placed into Tcl variables named after the columns. If the -array option is given, the column values are instead stored into elements of the named associative array, with the column names used as array indexes. In addition, the current row number within the result (counting from zero) is stored into the array element named “.tupno”, unless that name is in use as a column name in the result.

If the command is a SELECT statement and no loop-body script is given, then only the first row of results are stored into Tcl variables or array elements; remaining rows, if any, are ignored. No storing occurs if the query returns no rows. (This case can be detected by checking the result of spi_exec.) For example:

spi_exec "SELECT count(*) AS cnt FROM pg_proc"

will set the Tcl variable $cnt to the number of rows in the pg_proc system catalog.

If the optional loop-body argument is given, it is a piece of Tcl script that is executed once for each row in the query result. (loop-body is ignored if the given command is not a SELECT.) The values of the current row's columns are stored into Tcl variables or array elements before each iteration. For example:

spi_exec -array C "SELECT * FROM pg_class" {
    elog DEBUG "have table $C(relname)"
}

will print a log message for every row of pg_class. This feature works similarly to other Tcl looping constructs; in particular continue and break work in the usual way inside the loop body.

If a column of a query result is null, the target variable for it is “unset” rather than being set.

spi_prepare query typelist

Prepares and saves a query plan for later execution. The saved plan will be retained for the life of the current session.

The query can use parameters, that is, placeholders for values to be supplied whenever the plan is actually executed. In the query string, refer to parameters by the symbols $1 ... $n. If the query uses parameters, the names of the parameter types must be given as a Tcl list. (Write an empty list for typelist if no parameters are used.)

The return value from spi_prepare is a query ID to be used in subsequent calls to spi_execp. See spi_execp for an example.

spi_execp ?-count n? ?-array name? ?-nulls string? queryid ?value-list? ?loop-body?

Executes a query previously prepared with spi_prepare. queryid is the ID returned by spi_prepare. If the query references parameters, a value-list must be supplied. This is a Tcl list of actual values for the parameters. The list must be the same length as the parameter type list previously given to spi_prepare. Omit value-list if the query has no parameters.

The optional value for -nulls is a string of spaces and 'n' characters telling spi_execp which of the parameters are null values. If given, it must have exactly the same length as the value-list. If it is not given, all the parameter values are nonnull.

Except for the way in which the query and its parameters are specified, spi_execp works just like spi_exec. The -count, -array, and loop-body options are the same, and so is the result value.

Here's an example of a PL/Tcl function using a prepared plan:

CREATE FUNCTION t1_count(integer, integer) RETURNS integer AS $$
    if {![ info exists GD(plan) ]} {
        # prepare the saved plan on the first call
        set GD(plan) [ spi_prepare \
                "SELECT count(*) AS cnt FROM t1 WHERE num >= \$1 AND num <= \$2" \
                [ list int4 int4 ] ]
    }
    spi_execp -count 1 $GD(plan) [ list $1 $2 ]
    return $cnt
$$ LANGUAGE pltcl;

We need backslashes inside the query string given to spi_prepare to ensure that the $n markers will be passed through to spi_prepare as-is, and not replaced by Tcl variable substitution.

subtransaction command

The Tcl script contained in command is executed within an SQL subtransaction. If the script returns an error, that entire subtransaction is rolled back before returning the error out to the surrounding Tcl code. See Section 44.9 for more details and an example.

quote string

Doubles all occurrences of single quote and backslash characters in the given string. This can be used to safely quote strings that are to be inserted into SQL commands given to spi_exec or spi_prepare. For example, think about an SQL command string like:

"SELECT '$val' AS ret"

where the Tcl variable val actually contains doesn't. This would result in the final command string:

SELECT 'doesn't' AS ret

which would cause a parse error during spi_exec or spi_prepare. To work properly, the submitted command should contain:

SELECT 'doesn''t' AS ret

which can be formed in PL/Tcl using:

"SELECT '[ quote $val ]' AS ret"

One advantage of spi_execp is that you don't have to quote parameter values like this, since the parameters are never parsed as part of an SQL command string.

elog level msg

Emits a log or error message. Possible levels are DEBUG, LOG, INFO, NOTICE, WARNING, ERROR, and FATAL. ERROR raises an error condition; if this is not trapped by the surrounding Tcl code, the error propagates out to the calling query, causing the current transaction or subtransaction to be aborted. This is effectively the same as the Tcl error command. FATAL aborts the transaction and causes the current session to shut down. (There is probably no good reason to use this error level in PL/Tcl functions, but it's provided for completeness.) The other levels only generate messages of different priority levels. Whether messages of a particular priority are reported to the client, written to the server log, or both is controlled by the log_min_messages and client_min_messages configuration variables. See Chapter 20 and Section 44.8 for more information.

Prev | Up | Next
44.4. Global Data in PL/Tcl | Home | 44.6. Trigger Functions in PL/Tcl


44.6. Trigger Functions in PL/Tcl
Prev Up Chapter 44. PL/Tcl — Tcl Procedural Language Home Next

44.6. Trigger Functions in PL/Tcl

Trigger functions can be written in PL/Tcl. PostgreSQL requires that a function that is to be called as a trigger must be declared as a function with no arguments and a return type of trigger.

The information from the trigger manager is passed to the function body in the following variables:

$TG_name

The name of the trigger from the CREATE TRIGGER statement.

$TG_relid

The object ID of the table that caused the trigger function to be invoked.

$TG_table_name

The name of the table that caused the trigger function to be invoked.

$TG_table_schema

The schema of the table that caused the trigger function to be invoked.

$TG_relatts

A Tcl list of the table column names, prefixed with an empty list element. So looking up a column name in the list with Tcl's lsearch command returns the element's number starting with 1 for the first column, the same way the columns are customarily numbered in PostgreSQL. (Empty list elements also appear in the positions of columns that have been dropped, so that the attribute numbering is correct for columns to their right.)

$TG_when

The string BEFORE, AFTER, or INSTEAD OF, depending on the type of trigger event.

$TG_level

The string ROW or STATEMENT depending on the type of trigger event.

$TG_op

The string INSERT, UPDATE, DELETE, or TRUNCATE depending on the type of trigger event.

$NEW

An associative array containing the values of the new table row for INSERT or UPDATE actions, or empty for DELETE. The array is indexed by column name. Columns that are null will not appear in the array. This is not set for statement-level triggers.

$OLD

An associative array containing the values of the old table row for UPDATE or DELETE actions, or empty for INSERT. The array is indexed by column name. Columns that are null will not appear in the array. This is not set for statement-level triggers.

$args

A Tcl list of the arguments to the function as given in the CREATE TRIGGER statement. These arguments are also accessible as $1 ... $n in the function body.

The return value from a trigger function can be one of the strings OK or SKIP, or a list of column name/value pairs. If the return value is OK, the operation (INSERT/UPDATE/DELETE) that fired the trigger will proceed normally. SKIP tells the trigger manager to silently suppress the operation for this row. If a list is returned, it tells PL/Tcl to return a modified row to the trigger manager; the contents of the modified row are specified by the column names and values in the list. Any columns not mentioned in the list are set to null. Returning a modified row is only meaningful for row-level BEFORE INSERT or UPDATE triggers, for which the modified row will be inserted instead of the one given in $NEW; or for row-level INSTEAD OF INSERT or UPDATE triggers where the returned row is used as the source data for INSERT RETURNING or UPDATE RETURNING clauses. In row-level BEFORE DELETE or INSTEAD OF DELETE triggers, returning a modified row has the same effect as returning OK, that is the operation proceeds. The trigger return value is ignored for all other types of triggers.

Tip

The result list can be made from an array representation of the modified tuple with the array get Tcl command.

Here's a little example trigger function that forces an integer value in a table to keep track of the number of updates that are performed on the row. For new rows inserted, the value is initialized to 0 and then incremented on every update operation.

CREATE FUNCTION trigfunc_modcount() RETURNS trigger AS $$
    switch $TG_op {
        INSERT {
            set NEW($1) 0
        }
        UPDATE {
            set NEW($1) $OLD($1)
            incr NEW($1)
        }
        default {
            return OK
        }
    }
    return [array get NEW]
$$ LANGUAGE pltcl;

CREATE TABLE mytab (num integer, description text, modcnt integer);

CREATE TRIGGER trig_mytab_modcount BEFORE INSERT OR UPDATE ON mytab
    FOR EACH ROW EXECUTE FUNCTION trigfunc_modcount('modcnt');

Notice that the trigger function itself does not know the column name; that's supplied from the trigger arguments. This lets the trigger function be reused with different tables.

Prev | Up | Next
44.5. Database Access from PL/Tcl | Home | 44.7. Event Trigger Functions in PL/Tcl


44.7. Event Trigger Functions in PL/Tcl
Prev Up Chapter 44. PL/Tcl — Tcl Procedural Language Home Next

44.7. Event Trigger Functions in PL/Tcl

Event trigger functions can be written in PL/Tcl. PostgreSQL requires that a function that is to be called as an event trigger must be declared as a function with no arguments and a return type of event_trigger.

The information from the trigger manager is passed to the function body in the following variables:

$TG_event

The name of the event the trigger is fired for.

$TG_tag

The command tag for which the trigger is fired.

The return value of the trigger function is ignored.

Here's a little example event trigger function that simply raises a NOTICE message each time a supported command is executed:

CREATE OR REPLACE FUNCTION tclsnitch() RETURNS event_trigger AS $$
  elog NOTICE "tclsnitch: $TG_event $TG_tag"
$$ LANGUAGE pltcl;

CREATE EVENT TRIGGER tcl_a_snitch ON ddl_command_start EXECUTE FUNCTION tclsnitch();

Prev | Up | Next
44.6. Trigger Functions in PL/Tcl | Home | 44.8. Error Handling in PL/Tcl


44.8. Error Handling in PL/Tcl
Prev Up Chapter 44. PL/Tcl — Tcl Procedural Language Home Next

44.8. Error Handling in PL/Tcl

Tcl code within or called from a PL/Tcl function can raise an error, either by executing some invalid operation or by generating an error using the Tcl error command or PL/Tcl's elog command. Such errors can be caught within Tcl using the Tcl catch command. If an error is not caught but is allowed to propagate out to the top level of execution of the PL/Tcl function, it is reported as an SQL error in the function's calling query.

Conversely, SQL errors that occur within PL/Tcl's spi_exec, spi_prepare, and spi_execp commands are reported as Tcl errors, so they are catchable by Tcl's catch command. (Each of these PL/Tcl commands runs its SQL operation in a subtransaction, which is rolled back on error, so that any partially-completed operation is automatically cleaned up.) Again, if an error propagates out to the top level without being caught, it turns back into an SQL error.

Tcl provides an errorCode variable that can represent additional information about an error in a form that is easy for Tcl programs to interpret. The contents are in Tcl list format, and the first word identifies the subsystem or library reporting the error; beyond that the contents are left to the individual subsystem or library. For database errors reported by PL/Tcl commands, the first word is POSTGRES, the second word is the PostgreSQL version number, and additional words are field name/value pairs providing detailed information about the error. Fields SQLSTATE, condition, and message are always supplied (the first two represent the error code and condition name as shown in Appendix A). Fields that may be present include detail, hint, context, schema, table, column, datatype, constraint, statement, cursor_position, filename, lineno, and funcname.

A convenient way to work with PL/Tcl's errorCode information is to load it into an array, so that the field names become array subscripts. Code for doing that might look like

if {[catch { spi_exec $sql_command }]} {
    if {[lindex $::errorCode 0] == "POSTGRES"} {
        array set errorArray $::errorCode
        if {$errorArray(condition) == "undefined_table"} {
            # deal with missing table
        } else {
            # deal with some other type of SQL error
        }
    }
}

(The double colons explicitly specify that errorCode is a global variable.)

Prev | Up | Next
44.7. Event Trigger Functions in PL/Tcl | Home | 44.9. Explicit Subtransactions in PL/Tcl


44.9. Explicit Subtransactions in PL/Tcl
Prev Up Chapter 44. PL/Tcl — Tcl Procedural Language Home Next

44.9. Explicit Subtransactions in PL/Tcl

Recovering from errors caused by database access as described in Section 44.8 can lead to an undesirable situation where some operations succeed before one of them fails, and after recovering from that error the data is left in an inconsistent state. PL/Tcl offers a solution to this problem in the form of explicit subtransactions.

Consider a function that implements a transfer between two accounts:

CREATE FUNCTION transfer_funds() RETURNS void AS $$
    if [catch {
        spi_exec "UPDATE accounts SET balance = balance - 100 WHERE account_name = 'joe'"
        spi_exec "UPDATE accounts SET balance = balance + 100 WHERE account_name = 'mary'"
    } errormsg] {
        set result [format "error transferring funds: %s" $errormsg]
    } else {
        set result "funds transferred successfully"
    }
    spi_exec "INSERT INTO operations (result) VALUES ('[quote $result]')"
$$ LANGUAGE pltcl;

If the second UPDATE statement results in an exception being raised, this function will log the failure, but the result of the first UPDATE will nevertheless be committed. In other words, the funds will be withdrawn from Joe's account, but will not be transferred to Mary's account. This happens because each spi_exec is a separate subtransaction, and only one of those subtransactions got rolled back.

To handle such cases, you can wrap multiple database operations in an explicit subtransaction, which will succeed or roll back as a whole. PL/Tcl provides a subtransaction command to manage this. We can rewrite our function as:

CREATE FUNCTION transfer_funds2() RETURNS void AS $$
    if [catch {
        subtransaction {
            spi_exec "UPDATE accounts SET balance = balance - 100 WHERE account_name = 'joe'"
            spi_exec "UPDATE accounts SET balance = balance + 100 WHERE account_name = 'mary'"
        }
    } errormsg] {
        set result [format "error transferring funds: %s" $errormsg]
    } else {
        set result "funds transferred successfully"
    }
    spi_exec "INSERT INTO operations (result) VALUES ('[quote $result]')"
$$ LANGUAGE pltcl;

Note that use of catch is still required for this purpose. Otherwise the error would propagate to the top level of the function, preventing the desired insertion into the operations table. The subtransaction command does not trap errors, it only assures that all database operations executed inside its scope will be rolled back together when an error is reported.

A rollback of an explicit subtransaction occurs on any error reported by the contained Tcl code, not only errors originating from database access. Thus a regular Tcl exception raised inside a subtransaction command will also cause the subtransaction to be rolled back. However, non-error exits out of the contained Tcl code (for instance, due to return) do not cause a rollback.

Prev | Up | Next
44.8. Error Handling in PL/Tcl | Home | 44.10. Transaction Management


44.10. Transaction Management
Prev Up Chapter 44. PL/Tcl — Tcl Procedural Language Home Next

44.10. Transaction Management

In a procedure called from the top level or an anonymous code block (DO command) called from the top level it is possible to control transactions. To commit the current transaction, call the commit command. To roll back the current transaction, call the rollback command. (Note that it is not possible to run the SQL commands COMMIT or ROLLBACK via spi_exec or similar. It has to be done using these functions.) After a transaction is ended, a new transaction is automatically started, so there is no separate command for that.

Here is an example:

CREATE PROCEDURE transaction_test1()
LANGUAGE pltcl
AS $$
for {set i 0} {$i < 10} {incr i} {
    spi_exec "INSERT INTO test1 (a) VALUES ($i)"
    if {$i % 2 == 0} {
        commit
    } else {
        rollback
    }
}
$$;

CALL transaction_test1();

Transactions cannot be ended when an explicit subtransaction is active.

Prev | Up | Next
44.9. Explicit Subtransactions in PL/Tcl | Home | 44.11. PL/Tcl Configuration


44.11. PL/Tcl Configuration
Prev Up Chapter 44. PL/Tcl — Tcl Procedural Language Home Next

44.11. PL/Tcl Configuration

This section lists configuration parameters that affect PL/Tcl.

pltcl.start_proc (string)

This parameter, if set to a nonempty string, specifies the name (possibly schema-qualified) of a parameterless PL/Tcl function that is to be executed whenever a new Tcl interpreter is created for PL/Tcl. Such a function can perform per-session initialization, such as loading additional Tcl code. A new Tcl interpreter is created when a PL/Tcl function is first executed in a database session, or when an additional interpreter has to be created because a PL/Tcl function is called by a new SQL role.

The referenced function must be written in the pltcl language, and must not be marked SECURITY DEFINER. (These restrictions ensure that it runs in the interpreter it's supposed to initialize.) The current user must have permission to call it, too.

If the function fails with an error it will abort the function call that caused the new interpreter to be created and propagate out to the calling query, causing the current transaction or subtransaction to be aborted. Any actions already done within Tcl won't be undone; however, that interpreter won't be used again. If the language is used again the initialization will be attempted again within a fresh Tcl interpreter.

Only superusers can change this setting. Although this setting can be changed within a session, such changes will not affect Tcl interpreters that have already been created.

pltclu.start_proc (string)

This parameter is exactly like pltcl.start_proc, except that it applies to PL/TclU. The referenced function must be written in the pltclu language.

Prev | Up | Next
44.10. Transaction Management | Home | 44.12. Tcl Procedure Names


44.12. Tcl Procedure Names
Prev Up Chapter 44. PL/Tcl — Tcl Procedural Language Home Next

44.12. Tcl Procedure Names

In PostgreSQL, the same function name can be used for different function definitions as long as the number of arguments or their types differ. Tcl, however, requires all procedure names to be distinct. PL/Tcl deals with this by making the internal Tcl procedure names contain the object ID of the function from the system table pg_proc as part of their name. Thus, PostgreSQL functions with the same name and different argument types will be different Tcl procedures, too. This is not normally a concern for a PL/Tcl programmer, but it might be visible when debugging.

Prev | Up | Next
44.11. PL/Tcl Configuration | Home | Chapter 45. PL/Perl — Perl Procedural Language


Chapter 45. PL/Perl — Perl Procedural Language
Prev Up Part V. Server Programming Home Next

Chapter 45. PL/Perl — Perl Procedural Language

Table of Contents

45.1. PL/Perl Functions and Arguments
45.2. Data Values in PL/Perl
45.3. Built-in Functions
45.3.1. Database Access from PL/Perl
45.3.2. Utility Functions in PL/Perl
45.4. Global Values in PL/Perl
45.5. Trusted and Untrusted PL/Perl
45.6. PL/Perl Triggers
45.7. PL/Perl Event Triggers
45.8. PL/Perl Under the Hood
45.8.1. Configuration
45.8.2. Limitations and Missing Features

PL/Perl is a loadable procedural language that enables you to write PostgreSQL functions and procedures in the Perl programming language.

The main advantage to using PL/Perl is that this allows use, within stored functions and procedures, of the manyfold “string munging” operators and functions available for Perl. Parsing complex strings might be easier using Perl than it is with the string functions and control structures provided in PL/pgSQL.

To install PL/Perl in a particular database, use CREATE EXTENSION plperl.

Tip

If a language is installed into template1, all subsequently created databases will have the language installed automatically.

Note

Users of source packages must specially enable the build of PL/Perl during the installation process. (Refer to Chapter 17 for more information.) Users of binary packages might find PL/Perl in a separate subpackage.

Prev | Up | Next
44.12. Tcl Procedure Names | Home | 45.1. PL/Perl Functions and Arguments


45.1. PL/Perl Functions and Arguments
Prev Up Chapter 45. PL/Perl — Perl Procedural Language Home Next

45.1. PL/Perl Functions and Arguments

To create a function in the PL/Perl language, use the standard CREATE FUNCTION syntax:

CREATE FUNCTION funcname (argument-types)
RETURNS return-type
-- function attributes can go here
AS $$
    # PL/Perl function body goes here
$$ LANGUAGE plperl;

The body of the function is ordinary Perl code. In fact, the PL/Perl glue code wraps it inside a Perl subroutine. A PL/Perl function is called in a scalar context, so it can't return a list. You can return non-scalar values (arrays, records, and sets) by returning a reference, as discussed below.

In a PL/Perl procedure, any return value from the Perl code is ignored.

PL/Perl also supports anonymous code blocks called with the DO statement:

DO $$
    # PL/Perl code
$$ LANGUAGE plperl;

An anonymous code block receives no arguments, and whatever value it might return is discarded. Otherwise it behaves just like a function.

Note

The use of named nested subroutines is dangerous in Perl, especially if they refer to lexical variables in the enclosing scope. Because a PL/Perl function is wrapped in a subroutine, any named subroutine you place inside one will be nested. In general, it is far safer to create anonymous subroutines which you call via a coderef. For more information, see the entries for Variable "%s" will not stay shared and Variable "%s" is not available in the perldiag man page, or search the Internet for “perl nested named subroutine”.

The syntax of the CREATE FUNCTION command requires the function body to be written as a string constant. It is usually most convenient to use dollar quoting (see Section 4.1.2.4) for the string constant. If you choose to use escape string syntax E'', you must double any single quote marks (') and backslashes (\) used in the body of the function (see Section 4.1.2.1).

Arguments and results are handled as in any other Perl subroutine: arguments are passed in @_, and a result value is returned with return or as the last expression evaluated in the function.

For example, a function returning the greater of two integer values could be defined as:

CREATE FUNCTION perl_max (integer, integer) RETURNS integer AS $$
    if ($_[0] > $_[1]) { return $_[0]; }
    return $_[1];
$$ LANGUAGE plperl;

Note

Arguments will be converted from the database's encoding to UTF-8 for use inside PL/Perl, and then converted from UTF-8 back to the database encoding upon return.

If an SQL null value is passed to a function, the argument value will appear as “undefined” in Perl. The above function definition will not behave very nicely with null inputs (in fact, it will act as though they are zeroes). We could add STRICT to the function definition to make PostgreSQL do something more reasonable: if a null value is passed, the function will not be called at all, but will just return a null result automatically. Alternatively, we could check for undefined inputs in the function body. For example, suppose that we wanted perl_max with one null and one nonnull argument to return the nonnull argument, rather than a null value:

CREATE FUNCTION perl_max (integer, integer) RETURNS integer AS $$
    my ($x, $y) = @_;
    if (not defined $x) {
        return undef if not defined $y;
        return $y;
    }
    return $x if not defined $y;
    return $x if $x > $y;
    return $y;
$$ LANGUAGE plperl;

As shown above, to return an SQL null value from a PL/Perl function, return an undefined value. This can be done whether the function is strict or not.

Anything in a function argument that is not a reference is a string, which is in the standard PostgreSQL external text representation for the relevant data type. In the case of ordinary numeric or text types, Perl will just do the right thing and the programmer will normally not have to worry about it. However, in other cases the argument will need to be converted into a form that is more usable in Perl. For example, the decode_bytea function can be used to convert an argument of type bytea into unescaped binary.

Similarly, values passed back to PostgreSQL must be in the external text representation format. For example, the encode_bytea function can be used to escape binary data for a return value of type bytea.

One case that is particularly important is boolean values. As just stated, the default behavior for bool values is that they are passed to Perl as text, thus either 't' or 'f'. This is problematic, since Perl will not treat 'f' as false! It is possible to improve matters by using a “transform” (see CREATE TRANSFORM). Suitable transforms are provided by the bool_plperl extension. To use it, install the extension:

CREATE EXTENSION bool_plperl;  -- or bool_plperlu for PL/PerlU

Then use the TRANSFORM function attribute for a PL/Perl function that takes or returns bool, for example:

CREATE FUNCTION perl_and(bool, bool) RETURNS bool
TRANSFORM FOR TYPE bool
AS $$
  my ($a, $b) = @_;
  return $a && $b;
$$ LANGUAGE plperl;

When this transform is applied, bool arguments will be seen by Perl as being 1 or empty, thus properly true or false. If the function result is type bool, it will be true or false according to whether Perl would evaluate the returned value as true. Similar transformations are also performed for boolean query arguments and results of SPI queries performed inside the function (Section 45.3.1).

Perl can return PostgreSQL arrays as references to Perl arrays. Here is an example:

CREATE OR REPLACE function returns_array()
RETURNS text[][] AS $$
    return [['a"b','c,d'],['e\\f','g']];
$$ LANGUAGE plperl;

select returns_array();

Perl passes PostgreSQL arrays as a blessed PostgreSQL::InServer::ARRAY object. This object may be treated as an array reference or a string, allowing for backward compatibility with Perl code written for PostgreSQL versions below 9.1 to run. For example:

CREATE OR REPLACE FUNCTION concat_array_elements(text[]) RETURNS TEXT AS $$
    my $arg = shift;
    my $result = "";
    return undef if (!defined $arg);

    # as an array reference
    for (@$arg) {
        $result .= $_;
    }

    # also works as a string
    $result .= $arg;

    return $result;
$$ LANGUAGE plperl;

SELECT concat_array_elements(ARRAY['PL','/','Perl']);

Note

Multidimensional arrays are represented as references to lower-dimensional arrays of references in a way common to every Perl programmer.

Composite-type arguments are passed to the function as references to hashes. The keys of the hash are the attribute names of the composite type. Here is an example:

CREATE TABLE employee (
    name text,
    basesalary integer,
    bonus integer
);

CREATE FUNCTION empcomp(employee) RETURNS integer AS $$
    my ($emp) = @_;
    return $emp->{basesalary} + $emp->{bonus};
$$ LANGUAGE plperl;

SELECT name, empcomp(employee.*) FROM employee;

A PL/Perl function can return a composite-type result using the same approach: return a reference to a hash that has the required attributes. For example:

CREATE TYPE testrowperl AS (f1 integer, f2 text, f3 text);

CREATE OR REPLACE FUNCTION perl_row() RETURNS testrowperl AS $$
    return {f2 => 'hello', f1 => 1, f3 => 'world'};
$$ LANGUAGE plperl;

SELECT * FROM perl_row();

Any columns in the declared result data type that are not present in the hash will be returned as null values.

Similarly, output arguments of procedures can be returned as a hash reference:

CREATE PROCEDURE perl_triple(INOUT a integer, INOUT b integer) AS $$
    my ($a, $b) = @_;
    return {a => $a * 3, b => $b * 3};
$$ LANGUAGE plperl;

CALL perl_triple(5, 10);

PL/Perl functions can also return sets of either scalar or composite types. Usually you'll want to return rows one at a time, both to speed up startup time and to keep from queuing up the entire result set in memory. You can do this with return_next as illustrated below. Note that after the last return_next, you must put either return or (better) return undef.

CREATE OR REPLACE FUNCTION perl_set_int(int)
RETURNS SETOF INTEGER AS $$
    foreach (0..$_[0]) {
        return_next($_);
    }
    return undef;
$$ LANGUAGE plperl;

SELECT * FROM perl_set_int(5);

CREATE OR REPLACE FUNCTION perl_set()
RETURNS SETOF testrowperl AS $$
    return_next({ f1 => 1, f2 => 'Hello', f3 => 'World' });
    return_next({ f1 => 2, f2 => 'Hello', f3 => 'PostgreSQL' });
    return_next({ f1 => 3, f2 => 'Hello', f3 => 'PL/Perl' });
    return undef;
$$ LANGUAGE plperl;

For small result sets, you can return a reference to an array that contains either scalars, references to arrays, or references to hashes for simple types, array types, and composite types, respectively. Here are some simple examples of returning the entire result set as an array reference:

CREATE OR REPLACE FUNCTION perl_set_int(int) RETURNS SETOF INTEGER AS $$
    return [0..$_[0]];
$$ LANGUAGE plperl;

SELECT * FROM perl_set_int(5);

CREATE OR REPLACE FUNCTION perl_set() RETURNS SETOF testrowperl AS $$
    return [
        { f1 => 1, f2 => 'Hello', f3 => 'World' },
        { f1 => 2, f2 => 'Hello', f3 => 'PostgreSQL' },
        { f1 => 3, f2 => 'Hello', f3 => 'PL/Perl' }
    ];
$$ LANGUAGE plperl;

SELECT * FROM perl_set();

If you wish to use the strict pragma with your code you have a few options. For temporary global use you can SET plperl.use_strict to true. This will affect subsequent compilations of PL/Perl functions, but not functions already compiled in the current session. For permanent global use you can set plperl.use_strict to true in the postgresql.conf file.

For permanent use in specific functions you can simply put:

use strict;

at the top of the function body.

The feature pragma is also available to use if your Perl is version 5.10.0 or higher.

Prev | Up | Next
Chapter 45. PL/Perl — Perl Procedural Language | Home | 45.2. Data Values in PL/Perl


45.2. Data Values in PL/Perl
Prev Up Chapter 45. PL/Perl — Perl Procedural Language Home Next

45.2. Data Values in PL/Perl

The argument values supplied to a PL/Perl function's code are simply the input arguments converted to text form (just as if they had been displayed by a SELECT statement). Conversely, the return and return_next commands will accept any string that is acceptable input format for the function's declared return type.

If this behavior is inconvenient for a particular case, it can be improved by using a transform, as already illustrated for bool values. Several examples of transform modules are included in the PostgreSQL distribution.

Prev | Up | Next
45.1. PL/Perl Functions and Arguments | Home | 45.3. Built-in Functions


45.3. Built-in Functions
Prev Up Chapter 45. PL/Perl — Perl Procedural Language Home Next

45.3. Built-in Functions

45.3.1. Database Access from PL/Perl
45.3.2. Utility Functions in PL/Perl

45.3.1. Database Access from PL/Perl

Access to the database itself from your Perl function can be done via the following functions:

spi_exec_query(query [, max-rows])

spi_exec_query executes an SQL command and returns the entire row set as a reference to an array of hash references. You should only use this command when you know that the result set will be relatively small. Here is an example of a query (SELECT command) with the optional maximum number of rows:

$rv = spi_exec_query('SELECT * FROM my_table', 5);

This returns up to 5 rows from the table my_table. If my_table has a column my_column, you can get that value from row $i of the result like this:

$foo = $rv->{rows}[$i]->{my_column};

The total number of rows returned from a SELECT query can be accessed like this:

$nrows = $rv->{processed}

Here is an example using a different command type:

$query = "INSERT INTO my_table VALUES (1, 'test')";
$rv = spi_exec_query($query);

You can then access the command status (e.g., SPI_OK_INSERT) like this:

$res = $rv->{status};

To get the number of rows affected, do:

$nrows = $rv->{processed};

Here is a complete example:

CREATE TABLE test (
    i int,
    v varchar
);

INSERT INTO test (i, v) VALUES (1, 'first line');
INSERT INTO test (i, v) VALUES (2, 'second line');
INSERT INTO test (i, v) VALUES (3, 'third line');
INSERT INTO test (i, v) VALUES (4, 'immortal');

CREATE OR REPLACE FUNCTION test_munge() RETURNS SETOF test AS $$
    my $rv = spi_exec_query('select i, v from test;');
    my $status = $rv->{status};
    my $nrows = $rv->{processed};
    foreach my $rn (0 .. $nrows - 1) {
        my $row = $rv->{rows}[$rn];
        $row->{i} += 200 if defined($row->{i});
        $row->{v} =~ tr/A-Za-z/a-zA-Z/ if (defined($row->{v}));
        return_next($row);
    }
    return undef;
$$ LANGUAGE plperl;

SELECT * FROM test_munge();

spi_query(command)
spi_fetchrow(cursor)
spi_cursor_close(cursor)

spi_query and spi_fetchrow work together as a pair for row sets which might be large, or for cases where you wish to return rows as they arrive. spi_fetchrow works only with spi_query. The following example illustrates how you use them together:

CREATE TYPE foo_type AS (the_num INTEGER, the_text TEXT);

CREATE OR REPLACE FUNCTION lotsa_md5 (INTEGER) RETURNS SETOF foo_type AS $$
    use Digest::MD5 qw(md5_hex);
    my $file = '/usr/share/dict/words';
    my $t = localtime;
    elog(NOTICE, "opening file $file at $t" );
    open my $fh, '<', $file # ooh, it's a file access!
        or elog(ERROR, "cannot open $file for reading: $!");
    my @words = <$fh>;
    close $fh;
    $t = localtime;
    elog(NOTICE, "closed file $file at $t");
    chomp(@words);
    my $row;
    my $sth = spi_query("SELECT * FROM generate_series(1,$_[0]) AS b(a)");
    while (defined ($row = spi_fetchrow($sth))) {
        return_next({
            the_num => $row->{a},
            the_text => md5_hex($words[rand @words])
        });
    }
    return;
$$ LANGUAGE plperlu;

SELECT * from lotsa_md5(500);

Normally, spi_fetchrow should be repeated until it returns undef, indicating that there are no more rows to read. The cursor returned by spi_query is automatically freed when spi_fetchrow returns undef. If you do not wish to read all the rows, instead call spi_cursor_close to free the cursor. Failure to do so will result in memory leaks.

spi_prepare(command, argument types)
spi_query_prepared(plan, arguments)
spi_exec_prepared(plan [, attributes], arguments)
spi_freeplan(plan)

spi_prepare, spi_query_prepared, spi_exec_prepared, and spi_freeplan implement the same functionality but for prepared queries. spi_prepare accepts a query string with numbered argument placeholders ($1, $2, etc) and a string list of argument types:

$plan = spi_prepare('SELECT * FROM test WHERE id > $1 AND name = $2',
                                                     'INTEGER', 'TEXT');

Once a query plan is prepared by a call to spi_prepare, the plan can be used instead of the string query, either in spi_exec_prepared, where the result is the same as returned by spi_exec_query, or in spi_query_prepared which returns a cursor exactly as spi_query does, which can be later passed to spi_fetchrow. The optional second parameter to spi_exec_prepared is a hash reference of attributes; the only attribute currently supported is limit, which sets the maximum number of rows returned by a query.

The advantage of prepared queries is that is it possible to use one prepared plan for more than one query execution. After the plan is not needed anymore, it can be freed with spi_freeplan:

CREATE OR REPLACE FUNCTION init() RETURNS VOID AS $$
        $_SHARED{my_plan} = spi_prepare('SELECT (now() + $1)::date AS now',
                                        'INTERVAL');
$$ LANGUAGE plperl;

CREATE OR REPLACE FUNCTION add_time( INTERVAL ) RETURNS TEXT AS $$
        return spi_exec_prepared(
                $_SHARED{my_plan},
                $_[0]
        )->{rows}->[0]->{now};
$$ LANGUAGE plperl;

CREATE OR REPLACE FUNCTION done() RETURNS VOID AS $$
        spi_freeplan( $_SHARED{my_plan});
        undef $_SHARED{my_plan};
$$ LANGUAGE plperl;

SELECT init();
SELECT add_time('1 day'), add_time('2 days'), add_time('3 days');
SELECT done();

  add_time  |  add_time  |  add_time
------------+------------+------------
 2005-12-10 | 2005-12-11 | 2005-12-12

Note that the parameter subscript in spi_prepare is defined via $1, $2, $3, etc, so avoid declaring query strings in double quotes that might easily lead to hard-to-catch bugs.

Another example illustrates usage of an optional parameter in spi_exec_prepared:

CREATE TABLE hosts AS SELECT id, ('192.168.1.'||id)::inet AS address
                      FROM generate_series(1,3) AS id;

CREATE OR REPLACE FUNCTION init_hosts_query() RETURNS VOID AS $$
        $_SHARED{plan} = spi_prepare('SELECT * FROM hosts
                                      WHERE address << $1', 'inet');
$$ LANGUAGE plperl;

CREATE OR REPLACE FUNCTION query_hosts(inet) RETURNS SETOF hosts AS $$
        return spi_exec_prepared(
                $_SHARED{plan},
                {limit => 2},
                $_[0]
        )->{rows};
$$ LANGUAGE plperl;

CREATE OR REPLACE FUNCTION release_hosts_query() RETURNS VOID AS $$
        spi_freeplan($_SHARED{plan});
        undef $_SHARED{plan};
$$ LANGUAGE plperl;

SELECT init_hosts_query();
SELECT query_hosts('192.168.1.0/30');
SELECT release_hosts_query();

    query_hosts
-----------------
 (1,192.168.1.1)
 (2,192.168.1.2)
(2 rows)

spi_commit()
spi_rollback()

Commit or roll back the current transaction. This can only be called in a procedure or anonymous code block (DO command) called from the top level. (Note that it is not possible to run the SQL commands COMMIT or ROLLBACK via spi_exec_query or similar. It has to be done using these functions.) After a transaction is ended, a new transaction is automatically started, so there is no separate function for that.

Here is an example:

CREATE PROCEDURE transaction_test1()
LANGUAGE plperl
AS $$
foreach my $i (0..9) {
    spi_exec_query("INSERT INTO test1 (a) VALUES ($i)");
    if ($i % 2 == 0) {
        spi_commit();
    } else {
        spi_rollback();
    }
}
$$;

CALL transaction_test1();

45.3.2. Utility Functions in PL/Perl

elog(level, msg)

Emit a log or error message. Possible levels are DEBUG, LOG, INFO, NOTICE, WARNING, and ERROR. ERROR raises an error condition; if this is not trapped by the surrounding Perl code, the error propagates out to the calling query, causing the current transaction or subtransaction to be aborted. This is effectively the same as the Perl die command. The other levels only generate messages of different priority levels. Whether messages of a particular priority are reported to the client, written to the server log, or both is controlled by the log_min_messages and client_min_messages configuration variables. See Chapter 20 for more information.

quote_literal(string)

Return the given string suitably quoted to be used as a string literal in an SQL statement string. Embedded single-quotes and backslashes are properly doubled. Note that quote_literal returns undef on undef input; if the argument might be undef, quote_nullable is often more suitable.

quote_nullable(string)

Return the given string suitably quoted to be used as a string literal in an SQL statement string; or, if the argument is undef, return the unquoted string "NULL". Embedded single-quotes and backslashes are properly doubled.

quote_ident(string)

Return the given string suitably quoted to be used as an identifier in an SQL statement string. Quotes are added only if necessary (i.e., if the string contains non-identifier characters or would be case-folded). Embedded quotes are properly doubled.

decode_bytea(string)

Return the unescaped binary data represented by the contents of the given string, which should be bytea encoded.

encode_bytea(string)

Return the bytea encoded form of the binary data contents of the given string.

encode_array_literal(array)
encode_array_literal(array, delimiter)

Returns the contents of the referenced array as a string in array literal format (see Section 8.15.2). Returns the argument value unaltered if it's not a reference to an array. The delimiter used between elements of the array literal defaults to ", " if a delimiter is not specified or is undef.

encode_typed_literal(value, typename)

Converts a Perl variable to the value of the data type passed as a second argument and returns a string representation of this value. Correctly handles nested arrays and values of composite types.

encode_array_constructor(array)

Returns the contents of the referenced array as a string in array constructor format (see Section 4.2.12). Individual values are quoted using quote_nullable. Returns the argument value, quoted using quote_nullable, if it's not a reference to an array.

looks_like_number(string)

Returns a true value if the content of the given string looks like a number, according to Perl, returns false otherwise. Returns undef if the argument is undef. Leading and trailing space is ignored. Inf and Infinity are regarded as numbers.

is_array_ref(argument)

Returns a true value if the given argument may be treated as an array reference, that is, if ref of the argument is ARRAY or PostgreSQL::InServer::ARRAY. Returns false otherwise.

Prev | Up | Next
45.2. Data Values in PL/Perl | Home | 45.4. Global Values in PL/Perl


45.4. Global Values in PL/Perl
Prev Up Chapter 45. PL/Perl — Perl Procedural Language Home Next

45.4. Global Values in PL/Perl

You can use the global hash %_SHARED to store data, including code references, between function calls for the lifetime of the current session.

Here is a simple example for shared data:

CREATE OR REPLACE FUNCTION set_var(name text, val text) RETURNS text AS $$
    if ($_SHARED{$_[0]} = $_[1]) {
        return 'ok';
    } else {
        return "cannot set shared variable $_[0] to $_[1]";
    }
$$ LANGUAGE plperl;

CREATE OR REPLACE FUNCTION get_var(name text) RETURNS text AS $$
    return $_SHARED{$_[0]};
$$ LANGUAGE plperl;

SELECT set_var('sample', 'Hello, PL/Perl!  How''s tricks?');
SELECT get_var('sample');

Here is a slightly more complicated example using a code reference:

CREATE OR REPLACE FUNCTION myfuncs() RETURNS void AS $$
    $_SHARED{myquote} = sub {
        my $arg = shift;
        $arg =~ s/(['\\])/\\$1/g;
        return "'$arg'";
    };
$$ LANGUAGE plperl;

SELECT myfuncs(); /* initializes the function */

/* Set up a function that uses the quote function */

CREATE OR REPLACE FUNCTION use_quote(TEXT) RETURNS text AS $$
    my $text_to_quote = shift;
    my $qfunc = $_SHARED{myquote};
    return &$qfunc($text_to_quote);
$$ LANGUAGE plperl;

(You could have replaced the above with the one-liner return $_SHARED{myquote}->($_[0]); at the expense of readability.)

For security reasons, PL/Perl executes functions called by any one SQL role in a separate Perl interpreter for that role. This prevents accidental or malicious interference by one user with the behavior of another user's PL/Perl functions. Each such interpreter has its own value of the %_SHARED variable and other global state. Thus, two PL/Perl functions will share the same value of %_SHARED if and only if they are executed by the same SQL role. In an application wherein a single session executes code under multiple SQL roles (via SECURITY DEFINER functions, use of SET ROLE, etc) you may need to take explicit steps to ensure that PL/Perl functions can share data via %_SHARED. To do that, make sure that functions that should communicate are owned by the same user, and mark them SECURITY DEFINER. You must of course take care that such functions can't be used to do anything unintended.

Prev | Up | Next
45.3. Built-in Functions | Home | 45.5. Trusted and Untrusted PL/Perl


45.5. Trusted and Untrusted PL/Perl
Prev Up Chapter 45. PL/Perl — Perl Procedural Language Home Next

45.5. Trusted and Untrusted PL/Perl

Normally, PL/Perl is installed as a “trusted” programming language named plperl. In this setup, certain Perl operations are disabled to preserve security. In general, the operations that are restricted are those that interact with the environment. This includes file handle operations, require, and use (for external modules). There is no way to access internals of the database server process or to gain OS-level access with the permissions of the server process, as a C function can do. Thus, any unprivileged database user can be permitted to use this language.

Here is an example of a function that will not work because file system operations are not allowed for security reasons:

CREATE FUNCTION badfunc() RETURNS integer AS $$
    my $tmpfile = "/tmp/badfile";
    open my $fh, '>', $tmpfile
        or elog(ERROR, qq{could not open the file "$tmpfile": $!});
    print $fh "Testing writing to a file\n";
    close $fh or elog(ERROR, qq{could not close the file "$tmpfile": $!});
    return 1;
$$ LANGUAGE plperl;

The creation of this function will fail as its use of a forbidden operation will be caught by the validator.

Sometimes it is desirable to write Perl functions that are not restricted. For example, one might want a Perl function that sends mail. To handle these cases, PL/Perl can also be installed as an “untrusted” language (usually called PL/PerlU). In this case the full Perl language is available. When installing the language, the language name plperlu will select the untrusted PL/Perl variant.

The writer of a PL/PerlU function must take care that the function cannot be used to do anything unwanted, since it will be able to do anything that could be done by a user logged in as the database administrator. Note that the database system allows only database superusers to create functions in untrusted languages.

If the above function was created by a superuser using the language plperlu, execution would succeed.

In the same way, anonymous code blocks written in Perl can use restricted operations if the language is specified as plperlu rather than plperl, but the caller must be a superuser.

Note

While PL/Perl functions run in a separate Perl interpreter for each SQL role, all PL/PerlU functions executed in a given session run in a single Perl interpreter (which is not any of the ones used for PL/Perl functions). This allows PL/PerlU functions to share data freely, but no communication can occur between PL/Perl and PL/PerlU functions.

Note

Perl cannot support multiple interpreters within one process unless it was built with the appropriate flags, namely either usemultiplicity or useithreads. (usemultiplicity is preferred unless you actually need to use threads. For more details, see the perlembed man page.) If PL/Perl is used with a copy of Perl that was not built this way, then it is only possible to have one Perl interpreter per session, and so any one session can only execute either PL/PerlU functions, or PL/Perl functions that are all called by the same SQL role.

Prev | Up | Next
45.4. Global Values in PL/Perl | Home | 45.6. PL/Perl Triggers


45.6. PL/Perl Triggers
Prev Up Chapter 45. PL/Perl — Perl Procedural Language Home Next

45.6. PL/Perl Triggers

PL/Perl can be used to write trigger functions. In a trigger function, the hash reference $_TD contains information about the current trigger event. $_TD is a global variable, which gets a separate local value for each invocation of the trigger. The fields of the $_TD hash reference are:

$_TD->{new}{foo}

NEW value of column foo

$_TD->{old}{foo}

OLD value of column foo

$_TD->{name}

Name of the trigger being called

$_TD->{event}

Trigger event: INSERT, UPDATE, DELETE, TRUNCATE, or UNKNOWN

$_TD->{when}

When the trigger was called: BEFORE, AFTER, INSTEAD OF, or UNKNOWN

$_TD->{level}

The trigger level: ROW, STATEMENT, or UNKNOWN

$_TD->{relid}

OID of the table on which the trigger fired

$_TD->{table_name}

Name of the table on which the trigger fired

$_TD->{relname}

Name of the table on which the trigger fired. This has been deprecated, and could be removed in a future release. Please use $_TD->{table_name} instead.

$_TD->{table_schema}

Name of the schema in which the table on which the trigger fired, is

$_TD->{argc}

Number of arguments of the trigger function

@{$_TD->{args}}

Arguments of the trigger function. Does not exist if $_TD->{argc} is 0.

Row-level triggers can return one of the following:

return;

Execute the operation

"SKIP"

Don't execute the operation

"MODIFY"

Indicates that the NEW row was modified by the trigger function

Here is an example of a trigger function, illustrating some of the above:

CREATE TABLE test (
    i int,
    v varchar
);

CREATE OR REPLACE FUNCTION valid_id() RETURNS trigger AS $$
    if (($_TD->{new}{i} >= 100) || ($_TD->{new}{i} <= 0)) {
        return "SKIP";    # skip INSERT/UPDATE command
    } elsif ($_TD->{new}{v} ne "immortal") {
        $_TD->{new}{v} .= "(modified by trigger)";
        return "MODIFY";  # modify row and execute INSERT/UPDATE command
    } else {
        return;           # execute INSERT/UPDATE command
    }
$$ LANGUAGE plperl;

CREATE TRIGGER test_valid_id_trig
    BEFORE INSERT OR UPDATE ON test
    FOR EACH ROW EXECUTE FUNCTION valid_id();

Prev | Up | Next
45.5. Trusted and Untrusted PL/Perl | Home | 45.7. PL/Perl Event Triggers


45.7. PL/Perl Event Triggers
Prev Up Chapter 45. PL/Perl — Perl Procedural Language Home Next

45.7. PL/Perl Event Triggers

PL/Perl can be used to write event trigger functions. In an event trigger function, the hash reference $_TD contains information about the current trigger event. $_TD is a global variable, which gets a separate local value for each invocation of the trigger. The fields of the $_TD hash reference are:

$_TD->{event}

The name of the event the trigger is fired for.

$_TD->{tag}

The command tag for which the trigger is fired.

The return value of the trigger function is ignored.

Here is an example of an event trigger function, illustrating some of the above:

CREATE OR REPLACE FUNCTION perlsnitch() RETURNS event_trigger AS $$
  elog(NOTICE, "perlsnitch: " . $_TD->{event} . " " . $_TD->{tag} . " ");
$$ LANGUAGE plperl;

CREATE EVENT TRIGGER perl_a_snitch
    ON ddl_command_start
    EXECUTE FUNCTION perlsnitch();

Prev | Up | Next
45.6. PL/Perl Triggers | Home | 45.8. PL/Perl Under the Hood


45.8. PL/Perl Under the Hood
Prev Up Chapter 45. PL/Perl — Perl Procedural Language Home Next

45.8. PL/Perl Under the Hood

45.8.1. Configuration
45.8.2. Limitations and Missing Features

45.8.1. Configuration

This section lists configuration parameters that affect PL/Perl.

plperl.on_init (string)

Specifies Perl code to be executed when a Perl interpreter is first initialized, before it is specialized for use by plperl or plperlu. The SPI functions are not available when this code is executed. If the code fails with an error it will abort the initialization of the interpreter and propagate out to the calling query, causing the current transaction or subtransaction to be aborted.

The Perl code is limited to a single string. Longer code can be placed into a module and loaded by the on_init string. Examples:

plperl.on_init = 'require "plperlinit.pl"'
plperl.on_init = 'use lib "/my/app"; use MyApp::PgInit;'

Any modules loaded by plperl.on_init, either directly or indirectly, will be available for use by plperl. This may create a security risk. To see what modules have been loaded you can use:

DO 'elog(WARNING, join ", ", sort keys %INC)' LANGUAGE plperl;

Initialization will happen in the postmaster if the plperl library is included in shared_preload_libraries, in which case extra consideration should be given to the risk of destabilizing the postmaster. The principal reason for making use of this feature is that Perl modules loaded by plperl.on_init need be loaded only at postmaster start, and will be instantly available without loading overhead in individual database sessions. However, keep in mind that the overhead is avoided only for the first Perl interpreter used by a database session — either PL/PerlU, or PL/Perl for the first SQL role that calls a PL/Perl function. Any additional Perl interpreters created in a database session will have to execute plperl.on_init afresh. Also, on Windows there will be no savings whatsoever from preloading, since the Perl interpreter created in the postmaster process does not propagate to child processes.

This parameter can only be set in the postgresql.conf file or on the server command line.

plperl.on_plperl_init (string)
plperl.on_plperlu_init (string)

These parameters specify Perl code to be executed when a Perl interpreter is specialized for plperl or plperlu respectively. This will happen when a PL/Perl or PL/PerlU function is first executed in a database session, or when an additional interpreter has to be created because the other language is called or a PL/Perl function is called by a new SQL role. This follows any initialization done by plperl.on_init. The SPI functions are not available when this code is executed. The Perl code in plperl.on_plperl_init is executed after “locking down” the interpreter, and thus it can only perform trusted operations.

If the code fails with an error it will abort the initialization and propagate out to the calling query, causing the current transaction or subtransaction to be aborted. Any actions already done within Perl won't be undone; however, that interpreter won't be used again. If the language is used again the initialization will be attempted again within a fresh Perl interpreter.

Only superusers can change these settings. Although these settings can be changed within a session, such changes will not affect Perl interpreters that have already been used to execute functions.

plperl.use_strict (boolean)

When set true subsequent compilations of PL/Perl functions will have the strict pragma enabled. This parameter does not affect functions already compiled in the current session.

45.8.2. Limitations and Missing Features

The following features are currently missing from PL/Perl, but they would make welcome contributions.

  • PL/Perl functions cannot call each other directly.

  • SPI is not yet fully implemented.

  • If you are fetching very large data sets using spi_exec_query, you should be aware that these will all go into memory. You can avoid this by using spi_query/spi_fetchrow as illustrated earlier.

    A similar problem occurs if a set-returning function passes a large set of rows back to PostgreSQL via return. You can avoid this problem too by instead using return_next for each row returned, as shown previously.

  • When a session ends normally, not due to a fatal error, any END blocks that have been defined are executed. Currently no other actions are performed. Specifically, file handles are not automatically flushed and objects are not automatically destroyed.

Prev | Up | Next
45.7. PL/Perl Event Triggers | Home | Chapter 46. PL/Python — Python Procedural Language


Chapter 46. PL/Python — Python Procedural Language
Prev Up Part V. Server Programming Home Next

Chapter 46. PL/Python — Python Procedural Language

Table of Contents

46.1. Python 2 vs. Python 3
46.2. PL/Python Functions
46.3. Data Values
46.3.1. Data Type Mapping
46.3.2. Null, None
46.3.3. Arrays, Lists
46.3.4. Composite Types
46.3.5. Set-Returning Functions
46.4. Sharing Data
46.5. Anonymous Code Blocks
46.6. Trigger Functions
46.7. Database Access
46.7.1. Database Access Functions
46.7.2. Trapping Errors
46.8. Explicit Subtransactions
46.8.1. Subtransaction Context Managers
46.8.2. Older Python Versions
46.9. Transaction Management
46.10. Utility Functions
46.11. Environment Variables

The PL/Python procedural language allows PostgreSQL functions and procedures to be written in the Python language.

To install PL/Python in a particular database, use CREATE EXTENSION plpythonu (but see also Section 46.1).

Tip

If a language is installed into template1, all subsequently created databases will have the language installed automatically.

PL/Python is only available as an “untrusted” language, meaning it does not offer any way of restricting what users can do in it and is therefore named plpythonu. A trusted variant plpython might become available in the future if a secure execution mechanism is developed in Python. The writer of a function in untrusted PL/Python must take care that the function cannot be used to do anything unwanted, since it will be able to do anything that could be done by a user logged in as the database administrator. Only superusers can create functions in untrusted languages such as plpythonu.

Note

Users of source packages must specially enable the build of PL/Python during the installation process. (Refer to the installation instructions for more information.) Users of binary packages might find PL/Python in a separate subpackage.

Prev | Up | Next
45.8. PL/Perl Under the Hood | Home | 46.1. Python 2 vs. Python 3


46.1. Python 2 vs. Python 3
Prev Up Chapter 46. PL/Python — Python Procedural Language Home Next

46.1. Python 2 vs. Python 3

PL/Python supports both the Python 2 and Python 3 language variants. (The PostgreSQL installation instructions might contain more precise information about the exact supported minor versions of Python.) Because the Python 2 and Python 3 language variants are incompatible in some important aspects, the following naming and transitioning scheme is used by PL/Python to avoid mixing them:

  • The PostgreSQL language named plpython2u implements PL/Python based on the Python 2 language variant.

  • The PostgreSQL language named plpython3u implements PL/Python based on the Python 3 language variant.

  • The language named plpythonu implements PL/Python based on the default Python language variant, which is currently Python 2. (This default is independent of what any local Python installations might consider to be their “default”, for example, what /usr/bin/python might be.) The default will probably be changed to Python 3 in a distant future release of PostgreSQL, depending on the progress of the migration to Python 3 in the Python community.

This scheme is analogous to the recommendations in PEP 394 regarding the naming and transitioning of the python command.

It depends on the build configuration or the installed packages whether PL/Python for Python 2 or Python 3 or both are available.

Tip

The built variant depends on which Python version was found during the installation or which version was explicitly set using the PYTHON environment variable; see Section 17.4. To make both variants of PL/Python available in one installation, the source tree has to be configured and built twice.

This results in the following usage and migration strategy:

  • Existing users and users who are currently not interested in Python 3 use the language name plpythonu and don't have to change anything for the foreseeable future. It is recommended to gradually “future-proof” the code via migration to Python 2.6/2.7 to simplify the eventual migration to Python 3.

    In practice, many PL/Python functions will migrate to Python 3 with few or no changes.

  • Users who know that they have heavily Python 2 dependent code and don't plan to ever change it can make use of the plpython2u language name. This will continue to work into the very distant future, until Python 2 support might be completely dropped by PostgreSQL.

  • Users who want to dive into Python 3 can use the plpython3u language name, which will keep working forever by today's standards. In the distant future, when Python 3 might become the default, they might like to remove the “3” for aesthetic reasons.

  • Daredevils, who want to build a Python-3-only operating system environment, can change the contents of plpythonu's extension control and script files to make plpythonu be equivalent to plpython3u, keeping in mind that this would make their installation incompatible with most of the rest of the world.

See also the document What's New In Python 3.0 for more information about porting to Python 3.

It is not allowed to use PL/Python based on Python 2 and PL/Python based on Python 3 in the same session, because the symbols in the dynamic modules would clash, which could result in crashes of the PostgreSQL server process. There is a check that prevents mixing Python major versions in a session, which will abort the session if a mismatch is detected. It is possible, however, to use both PL/Python variants in the same database, from separate sessions.

Prev | Up | Next
Chapter 46. PL/Python — Python Procedural Language | Home | 46.2. PL/Python Functions


46.2. PL/Python Functions
Prev Up Chapter 46. PL/Python — Python Procedural Language Home Next

46.2. PL/Python Functions

Functions in PL/Python are declared via the standard CREATE FUNCTION syntax:

CREATE FUNCTION funcname (argument-list)
  RETURNS return-type
AS $$
  # PL/Python function body
$$ LANGUAGE plpythonu;

The body of a function is simply a Python script. When the function is called, its arguments are passed as elements of the list args; named arguments are also passed as ordinary variables to the Python script. Use of named arguments is usually more readable. The result is returned from the Python code in the usual way, with return or yield (in case of a result-set statement). If you do not provide a return value, Python returns the default None. PL/Python translates Python's None into the SQL null value. In a procedure, the result from the Python code must be None (typically achieved by ending the procedure without a return statement or by using a return statement without argument); otherwise, an error will be raised.

For example, a function to return the greater of two integers can be defined as:

CREATE FUNCTION pymax (a integer, b integer)
  RETURNS integer
AS $$
  if a > b:
    return a
  return b
$$ LANGUAGE plpythonu;

The Python code that is given as the body of the function definition is transformed into a Python function. For example, the above results in:

def __plpython_procedure_pymax_23456():
  if a > b:
    return a
  return b

assuming that 23456 is the OID assigned to the function by PostgreSQL.

The arguments are set as global variables. Because of the scoping rules of Python, this has the subtle consequence that an argument variable cannot be reassigned inside the function to the value of an expression that involves the variable name itself, unless the variable is redeclared as global in the block. For example, the following won't work:

CREATE FUNCTION pystrip(x text)
  RETURNS text
AS $$
  x = x.strip()  # error
  return x
$$ LANGUAGE plpythonu;

because assigning to x makes x a local variable for the entire block, and so the x on the right-hand side of the assignment refers to a not-yet-assigned local variable x, not the PL/Python function parameter. Using the global statement, this can be made to work:

CREATE FUNCTION pystrip(x text)
  RETURNS text
AS $$
  global x
  x = x.strip()  # ok now
  return x
$$ LANGUAGE plpythonu;

But it is advisable not to rely on this implementation detail of PL/Python. It is better to treat the function parameters as read-only.

Prev | Up | Next
46.1. Python 2 vs. Python 3 | Home | 46.3. Data Values


46.3. Data Values
Prev Up Chapter 46. PL/Python — Python Procedural Language Home Next

46.3. Data Values

46.3.1. Data Type Mapping
46.3.2. Null, None
46.3.3. Arrays, Lists
46.3.4. Composite Types
46.3.5. Set-Returning Functions

Generally speaking, the aim of PL/Python is to provide a “natural” mapping between the PostgreSQL and the Python worlds. This informs the data mapping rules described below.

46.3.1. Data Type Mapping

When a PL/Python function is called, its arguments are converted from their PostgreSQL data type to a corresponding Python type:

  • PostgreSQL boolean is converted to Python bool.

  • PostgreSQL smallint and int are converted to Python int. PostgreSQL bigint and oid are converted to long in Python 2 and to int in Python 3.

  • PostgreSQL real and double are converted to Python float.

  • PostgreSQL numeric is converted to Python Decimal. This type is imported from the cdecimal package if that is available. Otherwise, decimal.Decimal from the standard library will be used. cdecimal is significantly faster than decimal. In Python 3.3 and up, however, cdecimal has been integrated into the standard library under the name decimal, so there is no longer any difference.

  • PostgreSQL bytea is converted to Python str in Python 2 and to bytes in Python 3. In Python 2, the string should be treated as a byte sequence without any character encoding.

  • All other data types, including the PostgreSQL character string types, are converted to a Python str. In Python 2, this string will be in the PostgreSQL server encoding; in Python 3, it will be a Unicode string like all strings.

  • For nonscalar data types, see below.

When a PL/Python function returns, its return value is converted to the function's declared PostgreSQL return data type as follows:

  • When the PostgreSQL return type is boolean, the return value will be evaluated for truth according to the Python rules. That is, 0 and empty string are false, but notably 'f' is true.

  • When the PostgreSQL return type is bytea, the return value will be converted to a string (Python 2) or bytes (Python 3) using the respective Python built-ins, with the result being converted to bytea.

  • For all other PostgreSQL return types, the return value is converted to a string using the Python built-in str, and the result is passed to the input function of the PostgreSQL data type. (If the Python value is a float, it is converted using the repr built-in instead of str, to avoid loss of precision.)

    Strings in Python 2 are required to be in the PostgreSQL server encoding when they are passed to PostgreSQL. Strings that are not valid in the current server encoding will raise an error, but not all encoding mismatches can be detected, so garbage data can still result when this is not done correctly. Unicode strings are converted to the correct encoding automatically, so it can be safer and more convenient to use those. In Python 3, all strings are Unicode strings.

  • For nonscalar data types, see below.

Note that logical mismatches between the declared PostgreSQL return type and the Python data type of the actual return object are not flagged; the value will be converted in any case.

46.3.2. Null, None

If an SQL null value is passed to a function, the argument value will appear as None in Python. For example, the function definition of pymax shown in Section 46.2 will return the wrong answer for null inputs. We could add STRICT to the function definition to make PostgreSQL do something more reasonable: if a null value is passed, the function will not be called at all, but will just return a null result automatically. Alternatively, we could check for null inputs in the function body:

CREATE FUNCTION pymax (a integer, b integer)
  RETURNS integer
AS $$
  if (a is None) or (b is None):
    return None
  if a > b:
    return a
  return b
$$ LANGUAGE plpythonu;

As shown above, to return an SQL null value from a PL/Python function, return the value None. This can be done whether the function is strict or not.

46.3.3. Arrays, Lists

SQL array values are passed into PL/Python as a Python list. To return an SQL array value out of a PL/Python function, return a Python list:

CREATE FUNCTION return_arr()
  RETURNS int[]
AS $$
return [1, 2, 3, 4, 5]
$$ LANGUAGE plpythonu;

SELECT return_arr();
 return_arr
-------------
 {1,2,3,4,5}
(1 row)

Multidimensional arrays are passed into PL/Python as nested Python lists. A 2-dimensional array is a list of lists, for example. When returning a multi-dimensional SQL array out of a PL/Python function, the inner lists at each level must all be of the same size. For example:

CREATE FUNCTION test_type_conversion_array_int4(x int4[]) RETURNS int4[] AS $$
plpy.info(x, type(x))
return x
$$ LANGUAGE plpythonu;

SELECT * FROM test_type_conversion_array_int4(ARRAY[[1,2,3],[4,5,6]]);
INFO:  ([[1, 2, 3], [4, 5, 6]], <type 'list'>)
 test_type_conversion_array_int4
---------------------------------
 {{1,2,3},{4,5,6}}
(1 row)

Other Python sequences, like tuples, are also accepted for backwards-compatibility with PostgreSQL versions 9.6 and below, when multi-dimensional arrays were not supported. However, they are always treated as one-dimensional arrays, because they are ambiguous with composite types. For the same reason, when a composite type is used in a multi-dimensional array, it must be represented by a tuple, rather than a list.

Note that in Python, strings are sequences, which can have undesirable effects that might be familiar to Python programmers:

CREATE FUNCTION return_str_arr()
  RETURNS varchar[]
AS $$
return "hello"
$$ LANGUAGE plpythonu;

SELECT return_str_arr();
 return_str_arr
----------------
 {h,e,l,l,o}
(1 row)

46.3.4. Composite Types

Composite-type arguments are passed to the function as Python mappings. The element names of the mapping are the attribute names of the composite type. If an attribute in the passed row has the null value, it has the value None in the mapping. Here is an example:

CREATE TABLE employee (
  name text,
  salary integer,
  age integer
);

CREATE FUNCTION overpaid (e employee)
  RETURNS boolean
AS $$
  if e["salary"] > 200000:
    return True
  if (e["age"] < 30) and (e["salary"] > 100000):
    return True
  return False
$$ LANGUAGE plpythonu;

There are multiple ways to return row or composite types from a Python function. The following examples assume we have:

CREATE TYPE named_value AS (
  name   text,
  value  integer
);

A composite result can be returned as a:

Sequence type (a tuple or list, but not a set because it is not indexable)

Returned sequence objects must have the same number of items as the composite result type has fields. The item with index 0 is assigned to the first field of the composite type, 1 to the second and so on. For example:

CREATE FUNCTION make_pair (name text, value integer)
  RETURNS named_value
AS $$
  return ( name, value )
  # or alternatively, as list: return [ name, value ]
$$ LANGUAGE plpythonu;

To return an SQL null for any column, insert None at the corresponding position.

When an array of composite types is returned, it cannot be returned as a list, because it is ambiguous whether the Python list represents a composite type, or another array dimension.

Mapping (dictionary)

The value for each result type column is retrieved from the mapping with the column name as key. Example:

CREATE FUNCTION make_pair (name text, value integer)
  RETURNS named_value
AS $$
  return { "name": name, "value": value }
$$ LANGUAGE plpythonu;

Any extra dictionary key/value pairs are ignored. Missing keys are treated as errors. To return an SQL null value for any column, insert None with the corresponding column name as the key.

Object (any object providing method __getattr__)

This works the same as a mapping. Example:

CREATE FUNCTION make_pair (name text, value integer)
  RETURNS named_value
AS $$
  class named_value:
    def __init__ (self, n, v):
      self.name = n
      self.value = v
  return named_value(name, value)

  # or simply
  class nv: pass
  nv.name = name
  nv.value = value
  return nv
$$ LANGUAGE plpythonu;

Functions with OUT parameters are also supported. For example:

CREATE FUNCTION multiout_simple(OUT i integer, OUT j integer) AS $$
return (1, 2)
$$ LANGUAGE plpythonu;

SELECT * FROM multiout_simple();

Output parameters of procedures are passed back the same way. For example:

CREATE PROCEDURE python_triple(INOUT a integer, INOUT b integer) AS $$
return (a * 3, b * 3)
$$ LANGUAGE plpythonu;

CALL python_triple(5, 10);

46.3.5. Set-Returning Functions

A PL/Python function can also return sets of scalar or composite types. There are several ways to achieve this because the returned object is internally turned into an iterator. The following examples assume we have composite type:

CREATE TYPE greeting AS (
  how text,
  who text
);

A set result can be returned from a:

Sequence type (tuple, list, set)

CREATE FUNCTION greet (how text)
  RETURNS SETOF greeting
AS $$
  # return tuple containing lists as composite types
  # all other combinations work also
  return ( [ how, "World" ], [ how, "PostgreSQL" ], [ how, "PL/Python" ] )
$$ LANGUAGE plpythonu;

Iterator (any object providing __iter__ and next methods)

CREATE FUNCTION greet (how text)
  RETURNS SETOF greeting
AS $$
  class producer:
    def __init__ (self, how, who):
      self.how = how
      self.who = who
      self.ndx = -1

    def __iter__ (self):
      return self

    def next (self):
      self.ndx += 1
      if self.ndx == len(self.who):
        raise StopIteration
      return ( self.how, self.who[self.ndx] )

  return producer(how, [ "World", "PostgreSQL", "PL/Python" ])
$$ LANGUAGE plpythonu;

Generator (yield)

CREATE FUNCTION greet (how text)
  RETURNS SETOF greeting
AS $$
  for who in [ "World", "PostgreSQL", "PL/Python" ]:
    yield ( how, who )
$$ LANGUAGE plpythonu;

Set-returning functions with OUT parameters (using RETURNS SETOF record) are also supported. For example:

CREATE FUNCTION multiout_simple_setof(n integer, OUT integer, OUT integer) RETURNS SETOF record AS $$
return [(1, 2)] * n
$$ LANGUAGE plpythonu;

SELECT * FROM multiout_simple_setof(3);

Prev | Up | Next
46.2. PL/Python Functions | Home | 46.4. Sharing Data


46.4. Sharing Data
Prev Up Chapter 46. PL/Python — Python Procedural Language Home Next

46.4. Sharing Data

The global dictionary SD is available to store private data between repeated calls to the same function. The global dictionary GD is public data, that is available to all Python functions within a session; use with care.

Each function gets its own execution environment in the Python interpreter, so that global data and function arguments from myfunc are not available to myfunc2. The exception is the data in the GD dictionary, as mentioned above.

Prev | Up | Next
46.3. Data Values | Home | 46.5. Anonymous Code Blocks


46.5. Anonymous Code Blocks
Prev Up Chapter 46. PL/Python — Python Procedural Language Home Next

46.5. Anonymous Code Blocks

PL/Python also supports anonymous code blocks called with the DO statement:

DO $$
    # PL/Python code
$$ LANGUAGE plpythonu;

An anonymous code block receives no arguments, and whatever value it might return is discarded. Otherwise it behaves just like a function.

Prev | Up | Next
46.4. Sharing Data | Home | 46.6. Trigger Functions


46.6. Trigger Functions
Prev Up Chapter 46. PL/Python — Python Procedural Language Home Next

46.6. Trigger Functions

When a function is used as a trigger, the dictionary TD contains trigger-related values:

TD["event"]

contains the event as a string: INSERT, UPDATE, DELETE, or TRUNCATE.

TD["when"]

contains one of BEFORE, AFTER, or INSTEAD OF.

TD["level"]

contains ROW or STATEMENT.

TD["new"]
TD["old"]

For a row-level trigger, one or both of these fields contain the respective trigger rows, depending on the trigger event.

TD["name"]

contains the trigger name.

TD["table_name"]

contains the name of the table on which the trigger occurred.

TD["table_schema"]

contains the schema of the table on which the trigger occurred.

TD["relid"]

contains the OID of the table on which the trigger occurred.

TD["args"]

If the CREATE TRIGGER command included arguments, they are available in TD["args"][0] to TD["args"][n-1].

If TD["when"] is BEFORE or INSTEAD OF and TD["level"] is ROW, you can return None or "OK" from the Python function to indicate the row is unmodified, "SKIP" to abort the event, or if TD["event"] is INSERT or UPDATE you can return "MODIFY" to indicate you've modified the new row. Otherwise the return value is ignored.

Prev | Up | Next
46.5. Anonymous Code Blocks | Home | 46.7. Database Access


46.7. Database Access
Prev Up Chapter 46. PL/Python — Python Procedural Language Home Next

46.7. Database Access

46.7.1. Database Access Functions
46.7.2. Trapping Errors

The PL/Python language module automatically imports a Python module called plpy. The functions and constants in this module are available to you in the Python code as plpy.foo.

46.7.1. Database Access Functions

The plpy module provides several functions to execute database commands:

plpy.execute(query [, max-rows])

Calling plpy.execute with a query string and an optional row limit argument causes that query to be run and the result to be returned in a result object.

The result object emulates a list or dictionary object. The result object can be accessed by row number and column name. For example:

rv = plpy.execute("SELECT * FROM my_table", 5)

returns up to 5 rows from my_table. If my_table has a column my_column, it would be accessed as:

foo = rv[i]["my_column"]

The number of rows returned can be obtained using the built-in len function.

The result object has these additional methods:

nrows()

Returns the number of rows processed by the command. Note that this is not necessarily the same as the number of rows returned. For example, an UPDATE command will set this value but won't return any rows (unless RETURNING is used).

status()

The SPI_execute() return value.

colnames()
coltypes()
coltypmods()

Return a list of column names, list of column type OIDs, and list of type-specific type modifiers for the columns, respectively.

These methods raise an exception when called on a result object from a command that did not produce a result set, e.g., UPDATE without RETURNING, or DROP TABLE. But it is OK to use these methods on a result set containing zero rows.

__str__()

The standard __str__ method is defined so that it is possible for example to debug query execution results using plpy.debug(rv).

The result object can be modified.

Note that calling plpy.execute will cause the entire result set to be read into memory. Only use that function when you are sure that the result set will be relatively small. If you don't want to risk excessive memory usage when fetching large results, use plpy.cursor rather than plpy.execute.

plpy.prepare(query [, argtypes])
plpy.execute(plan [, arguments [, max-rows]])

plpy.prepare prepares the execution plan for a query. It is called with a query string and a list of parameter types, if you have parameter references in the query. For example:

plan = plpy.prepare("SELECT last_name FROM my_users WHERE first_name = $1", ["text"])

text is the type of the variable you will be passing for $1. The second argument is optional if you don't want to pass any parameters to the query.

After preparing a statement, you use a variant of the function plpy.execute to run it:

rv = plpy.execute(plan, ["name"], 5)

Pass the plan as the first argument (instead of the query string), and a list of values to substitute into the query as the second argument. The second argument is optional if the query does not expect any parameters. The third argument is the optional row limit as before.

Alternatively, you can call the execute method on the plan object:

rv = plan.execute(["name"], 5)

Query parameters and result row fields are converted between PostgreSQL and Python data types as described in Section 46.3.

When you prepare a plan using the PL/Python module it is automatically saved. Read the SPI documentation (Chapter 47) for a description of what this means. In order to make effective use of this across function calls one needs to use one of the persistent storage dictionaries SD or GD (see Section 46.4). For example:

CREATE FUNCTION usesavedplan() RETURNS trigger AS $$
    if "plan" in SD:
        plan = SD["plan"]
    else:
        plan = plpy.prepare("SELECT 1")
        SD["plan"] = plan
    # rest of function
$$ LANGUAGE plpythonu;

plpy.cursor(query)
plpy.cursor(plan [, arguments])

The plpy.cursor function accepts the same arguments as plpy.execute (except for the row limit) and returns a cursor object, which allows you to process large result sets in smaller chunks. As with plpy.execute, either a query string or a plan object along with a list of arguments can be used, or the cursor function can be called as a method of the plan object.

The cursor object provides a fetch method that accepts an integer parameter and returns a result object. Each time you call fetch, the returned object will contain the next batch of rows, never larger than the parameter value. Once all rows are exhausted, fetch starts returning an empty result object. Cursor objects also provide an iterator interface, yielding one row at a time until all rows are exhausted. Data fetched that way is not returned as result objects, but rather as dictionaries, each dictionary corresponding to a single result row.

An example of two ways of processing data from a large table is:

CREATE FUNCTION count_odd_iterator() RETURNS integer AS $$
odd = 0
for row in plpy.cursor("select num from largetable"):
    if row['num'] % 2:
         odd += 1
return odd
$$ LANGUAGE plpythonu;

CREATE FUNCTION count_odd_fetch(batch_size integer) RETURNS integer AS $$
odd = 0
cursor = plpy.cursor("select num from largetable")
while True:
    rows = cursor.fetch(batch_size)
    if not rows:
        break
    for row in rows:
        if row['num'] % 2:
            odd += 1
return odd
$$ LANGUAGE plpythonu;

CREATE FUNCTION count_odd_prepared() RETURNS integer AS $$
odd = 0
plan = plpy.prepare("select num from largetable where num % $1 <> 0", ["integer"])
rows = list(plpy.cursor(plan, [2]))  # or: = list(plan.cursor([2]))

return len(rows)
$$ LANGUAGE plpythonu;

Cursors are automatically disposed of. But if you want to explicitly release all resources held by a cursor, use the close method. Once closed, a cursor cannot be fetched from anymore.

Tip

Do not confuse objects created by plpy.cursor with DB-API cursors as defined by the Python Database API specification. They don't have anything in common except for the name.

46.7.2. Trapping Errors

Functions accessing the database might encounter errors, which will cause them to abort and raise an exception. Both plpy.execute and plpy.prepare can raise an instance of a subclass of plpy.SPIError, which by default will terminate the function. This error can be handled just like any other Python exception, by using the try/except construct. For example:

CREATE FUNCTION try_adding_joe() RETURNS text AS $$
    try:
        plpy.execute("INSERT INTO users(username) VALUES ('joe')")
    except plpy.SPIError:
        return "something went wrong"
    else:
        return "Joe added"
$$ LANGUAGE plpythonu;

The actual class of the exception being raised corresponds to the specific condition that caused the error. Refer to Table A.1 for a list of possible conditions. The module plpy.spiexceptions defines an exception class for each PostgreSQL condition, deriving their names from the condition name. For instance, division_by_zero becomes DivisionByZero, unique_violation becomes UniqueViolation, fdw_error becomes FdwError, and so on. Each of these exception classes inherits from SPIError. This separation makes it easier to handle specific errors, for instance:

CREATE FUNCTION insert_fraction(numerator int, denominator int) RETURNS text AS $$
from plpy import spiexceptions
try:
    plan = plpy.prepare("INSERT INTO fractions (frac) VALUES ($1 / $2)", ["int", "int"])
    plpy.execute(plan, [numerator, denominator])
except spiexceptions.DivisionByZero:
    return "denominator cannot equal zero"
except spiexceptions.UniqueViolation:
    return "already have that fraction"
except plpy.SPIError as e:
    return "other error, SQLSTATE %s" % e.sqlstate
else:
    return "fraction inserted"
$$ LANGUAGE plpythonu;

Note that because all exceptions from the plpy.spiexceptions module inherit from SPIError, an except clause handling it will catch any database access error.

As an alternative way of handling different error conditions, you can catch the SPIError exception and determine the specific error condition inside the except block by looking at the sqlstate attribute of the exception object. This attribute is a string value containing the “SQLSTATE” error code. This approach provides approximately the same functionality

Prev | Up | Next
46.6. Trigger Functions | Home | 46.8. Explicit Subtransactions


46.8. Explicit Subtransactions
Prev Up Chapter 46. PL/Python — Python Procedural Language Home Next

46.8. Explicit Subtransactions

46.8.1. Subtransaction Context Managers
46.8.2. Older Python Versions

Recovering from errors caused by database access as described in Section 46.7.2 can lead to an undesirable situation where some operations succeed before one of them fails, and after recovering from that error the data is left in an inconsistent state. PL/Python offers a solution to this problem in the form of explicit subtransactions.

46.8.1. Subtransaction Context Managers

Consider a function that implements a transfer between two accounts:

CREATE FUNCTION transfer_funds() RETURNS void AS $$
try:
    plpy.execute("UPDATE accounts SET balance = balance - 100 WHERE account_name = 'joe'")
    plpy.execute("UPDATE accounts SET balance = balance + 100 WHERE account_name = 'mary'")
except plpy.SPIError as e:
    result = "error transferring funds: %s" % e.args
else:
    result = "funds transferred correctly"
plan = plpy.prepare("INSERT INTO operations (result) VALUES ($1)", ["text"])
plpy.execute(plan, [result])
$$ LANGUAGE plpythonu;

If the second UPDATE statement results in an exception being raised, this function will report the error, but the result of the first UPDATE will nevertheless be committed. In other words, the funds will be withdrawn from Joe's account, but will not be transferred to Mary's account.

To avoid such issues, you can wrap your plpy.execute calls in an explicit subtransaction. The plpy module provides a helper object to manage explicit subtransactions that gets created with the plpy.subtransaction() function. Objects created by this function implement the context manager interface. Using explicit subtransactions we can rewrite our function as:

CREATE FUNCTION transfer_funds2() RETURNS void AS $$
try:
    with plpy.subtransaction():
        plpy.execute("UPDATE accounts SET balance = balance - 100 WHERE account_name = 'joe'")
        plpy.execute("UPDATE accounts SET balance = balance + 100 WHERE account_name = 'mary'")
except plpy.SPIError as e:
    result = "error transferring funds: %s" % e.args
else:
    result = "funds transferred correctly"
plan = plpy.prepare("INSERT INTO operations (result) VALUES ($1)", ["text"])
plpy.execute(plan, [result])
$$ LANGUAGE plpythonu;

Note that the use of try/catch is still required. Otherwise the exception would propagate to the top of the Python stack and would cause the whole function to abort with a PostgreSQL error, so that the operations table would not have any row inserted into it. The subtransaction context manager does not trap errors, it only assures that all database operations executed inside its scope will be atomically committed or rolled back. A rollback of the subtransaction block occurs on any kind of exception exit, not only ones caused by errors originating from database access. A regular Python exception raised inside an explicit subtransaction block would also cause the subtransaction to be rolled back.

46.8.2. Older Python Versions

Context managers syntax using the with keyword is available by default in Python 2.6. For compatibility with older Python versions, you can call the subtransaction manager's __enter__ and __exit__ functions using the enter and exit convenience aliases. The example function that transfers funds could be written as:

CREATE FUNCTION transfer_funds_old() RETURNS void AS $$
try:
    subxact = plpy.subtransaction()
    subxact.enter()
    try:
        plpy.execute("UPDATE accounts SET balance = balance - 100 WHERE account_name = 'joe'")
        plpy.execute("UPDATE accounts SET balance = balance + 100 WHERE account_name = 'mary'")
    except:
        import sys
        subxact.exit(*sys.exc_info())
        raise
    else:
        subxact.exit(None, None, None)
except plpy.SPIError as e:
    result = "error transferring funds: %s" % e.args
else:
    result = "funds transferred correctly"

plan = plpy.prepare("INSERT INTO operations (result) VALUES ($1)", ["text"])
plpy.execute(plan, [result])
$$ LANGUAGE plpythonu;

Prev | Up | Next
46.7. Database Access | Home | 46.9. Transaction Management


46.9. Transaction Management
Prev Up Chapter 46. PL/Python — Python Procedural Language Home Next

46.9. Transaction Management

In a procedure called from the top level or an anonymous code block (DO command) called from the top level it is possible to control transactions. To commit the current transaction, call plpy.commit(). To roll back the current transaction, call plpy.rollback(). (Note that it is not possible to run the SQL commands COMMIT or ROLLBACK via plpy.execute or similar. It has to be done using these functions.) After a transaction is ended, a new transaction is automatically started, so there is no separate function for that.

Here is an example:

CREATE PROCEDURE transaction_test1()
LANGUAGE plpythonu
AS $$
for i in range(0, 10):
    plpy.execute("INSERT INTO test1 (a) VALUES (%d)" % i)
    if i % 2 == 0:
        plpy.commit()
    else:
        plpy.rollback()
$$;

CALL transaction_test1();

Transactions cannot be ended when an explicit subtransaction is active.

Prev | Up | Next
46.8. Explicit Subtransactions | Home | 46.10. Utility Functions


46.10. Utility Functions
Prev Up Chapter 46. PL/Python — Python Procedural Language Home Next

46.10. Utility Functions

The plpy module also provides the functions

plpy.debug(msg, **kwargs)
plpy.log(msg, **kwargs)
plpy.info(msg, **kwargs)
plpy.notice(msg, **kwargs)
plpy.warning(msg, **kwargs)
plpy.error(msg, **kwargs)
plpy.fatal(msg, **kwargs)

plpy.error and plpy.fatal actually raise a Python exception which, if uncaught, propagates out to the calling query, causing the current transaction or subtransaction to be aborted. raise plpy.Error(msg) and raise plpy.Fatal(msg) are equivalent to calling plpy.error(msg) and plpy.fatal(msg), respectively but the raise form does not allow passing keyword arguments. The other functions only generate messages of different priority levels. Whether messages of a particular priority are reported to the client, written to the server log, or both is controlled by the log_min_messages and client_min_messages configuration variables. See Chapter 20 for more information.

The msg argument is given as a positional argument. For backward compatibility, more than one positional argument can be given. In that case, the string representation of the tuple of positional arguments becomes the message reported to the client.

The following keyword-only arguments are accepted:

detail
hint
sqlstate
schema_name
table_name
column_name
datatype_name
constraint_name

The string representation of the objects passed as keyword-only arguments is used to enrich the messages reported to the client. For example:

CREATE FUNCTION raise_custom_exception() RETURNS void AS $$
plpy.error("custom exception message",
           detail="some info about exception",
           hint="hint for users")
$$ LANGUAGE plpythonu;

=# SELECT raise_custom_exception();
ERROR:  plpy.Error: custom exception message
DETAIL:  some info about exception
HINT:  hint for users
CONTEXT:  Traceback (most recent call last):
  PL/Python function "raise_custom_exception", line 4, in <module>
    hint="hint for users")
PL/Python function "raise_custom_exception"

Another set of utility functions are plpy.quote_literal(string), plpy.quote_nullable(string), and plpy.quote_ident(string). They are equivalent to the built-in quoting functions described in Section 9.4. They are useful when constructing ad-hoc queries. A PL/Python equivalent of dynamic SQL from Example 43.1 would be:

plpy.execute("UPDATE tbl SET %s = %s WHERE key = %s" % (
    plpy.quote_ident(colname),
    plpy.quote_nullable(newvalue),
    plpy.quote_literal(keyvalue)))

Prev | Up | Next
46.9. Transaction Management | Home | 46.11. Environment Variables


46.11. Environment Variables
Prev Up Chapter 46. PL/Python — Python Procedural Language Home Next

46.11. Environment Variables

Some of the environment variables that are accepted by the Python interpreter can also be used to affect PL/Python behavior. They would need to be set in the environment of the main PostgreSQL server process, for example in a start script. The available environment variables depend on the version of Python; see the Python documentation for details. At the time of this writing, the following environment variables have an affect on PL/Python, assuming an adequate Python version:

  • PYTHONHOME

  • PYTHONPATH

  • PYTHONY2K

  • PYTHONOPTIMIZE

  • PYTHONDEBUG

  • PYTHONVERBOSE

  • PYTHONCASEOK

  • PYTHONDONTWRITEBYTECODE

  • PYTHONIOENCODING

  • PYTHONUSERBASE

  • PYTHONHASHSEED

(It appears to be a Python implementation detail beyond the control of PL/Python that some of the environment variables listed on the python man page are only effective in a command-line interpreter and not an embedded Python interpreter.)

Prev | Up | Next
46.10. Utility Functions | Home | Chapter 47. Server Programming Interface


Chapter 48. Background Worker Processes
Prev Up Part V. Server Programming Home Next

Chapter 48. Background Worker Processes

PostgreSQL can be extended to run user-supplied code in separate processes. Such processes are started, stopped and monitored by postgres, which permits them to have a lifetime closely linked to the server's status. These processes have the option to attach to PostgreSQL's shared memory area and to connect to databases internally; they can also run multiple transactions serially, just like a regular client-connected server process. Also, by linking to libpq they can connect to the server and behave like a regular client application.

Warning

There are considerable robustness and security risks in using background worker processes because, being written in the C language, they have unrestricted access to data. Administrators wishing to enable modules that include background worker processes should exercise extreme caution. Only carefully audited modules should be permitted to run background worker processes.

Background workers can be initialized at the time that PostgreSQL is started by including the module name in shared_preload_libraries. A module wishing to run a background worker can register it by calling RegisterBackgroundWorker(BackgroundWorker *worker) from its _PG_init() function. Background workers can also be started after the system is up and running by calling RegisterDynamicBackgroundWorker(BackgroundWorker *worker, BackgroundWorkerHandle **handle). Unlike RegisterBackgroundWorker, which can only be called from within the postmaster process, RegisterDynamicBackgroundWorker must be called from a regular backend or another background worker.

The structure BackgroundWorker is defined thus:

typedef void (*bgworker_main_type)(Datum main_arg);
typedef struct BackgroundWorker
{
    char        bgw_name[BGW_MAXLEN];
    char        bgw_type[BGW_MAXLEN];
    int         bgw_flags;
    BgWorkerStartTime bgw_start_time;
    int         bgw_restart_time;       /* in seconds, or BGW_NEVER_RESTART */
    char        bgw_library_name[BGW_MAXLEN];
    char        bgw_function_name[BGW_MAXLEN];
    Datum       bgw_main_arg;
    char        bgw_extra[BGW_EXTRALEN];
    int         bgw_notify_pid;
} BackgroundWorker;

bgw_name and bgw_type are strings to be used in log messages, process listings and similar contexts. bgw_type should be the same for all background workers of the same type, so that it is possible to group such workers in a process listing, for example. bgw_name on the other hand can contain additional information about the specific process. (Typically, the string for bgw_name will contain the type somehow, but that is not strictly required.)

bgw_flags is a bitwise-or'd bit mask indicating the capabilities that the module wants. Possible values are:

BGWORKER_SHMEM_ACCESS

Requests shared memory access. Workers without shared memory access cannot access any of PostgreSQL's shared data structures, such as heavyweight or lightweight locks, shared buffers, or any custom data structures which the worker itself may wish to create and use.

BGWORKER_BACKEND_DATABASE_CONNECTION

Requests the ability to establish a database connection through which it can later run transactions and queries. A background worker using BGWORKER_BACKEND_DATABASE_CONNECTION to connect to a database must also attach shared memory using BGWORKER_SHMEM_ACCESS, or worker start-up will fail.

bgw_start_time is the server state during which postgres should start the process; it can be one of BgWorkerStart_PostmasterStart (start as soon as postgres itself has finished its own initialization; processes requesting this are not eligible for database connections), BgWorkerStart_ConsistentState (start as soon as a consistent state has been reached in a hot standby, allowing processes to connect to databases and run read-only queries), and BgWorkerStart_RecoveryFinished (start as soon as the system has entered normal read-write state). Note the last two values are equivalent in a server that's not a hot standby. Note that this setting only indicates when the processes are to be started; they do not stop when a different state is reached.

bgw_restart_time is the interval, in seconds, that postgres should wait before restarting the process in the event that it crashes. It can be any positive value, or BGW_NEVER_RESTART, indicating not to restart the process in case of a crash.

bgw_library_name is the name of a library in which the initial entry point for the background worker should be sought. The named library will be dynamically loaded by the worker process and bgw_function_name will be used to identify the function to be called. If loading a function from the core code, this must be set to "postgres".

bgw_function_name is the name of a function in a dynamically loaded library which should be used as the initial entry point for a new background worker.

bgw_main_arg is the Datum argument to the background worker main function. This main function should take a single argument of type Datum and return void. bgw_main_arg will be passed as the argument. In addition, the global variable MyBgworkerEntry points to a copy of the BackgroundWorker structure passed at registration time; the worker may find it helpful to examine this structure.

On Windows (and anywhere else where EXEC_BACKEND is defined) or in dynamic background workers it is not safe to pass a Datum by reference, only by value. If an argument is required, it is safest to pass an int32 or other small value and use that as an index into an array allocated in shared memory. If a value like a cstring or text is passed then the pointer won't be valid from the new background worker process.

bgw_extra can contain extra data to be passed to the background worker. Unlike bgw_main_arg, this data is not passed as an argument to the worker's main function, but it can be accessed via MyBgworkerEntry, as discussed above.

bgw_notify_pid is the PID of a PostgreSQL backend process to which the postmaster should send SIGUSR1 when the process is started or exits. It should be 0 for workers registered at postmaster startup time, or when the backend registering the worker does not wish to wait for the worker to start up. Otherwise, it should be initialized to MyProcPid.

Once running, the process can connect to a database by calling BackgroundWorkerInitializeConnection(char *dbname, char *username, uint32 flags) or BackgroundWorkerInitializeConnectionByOid(Oid dboid, Oid useroid, uint32 flags). This allows the process to run transactions and queries using the SPI interface. If dbname is NULL or dboid is InvalidOid, the session is not connected to any particular database, but shared catalogs can be accessed. If username is NULL or useroid is InvalidOid, the process will run as the superuser created during initdb. If BGWORKER_BYPASS_ALLOWCONN is specified as flags it is possible to bypass the restriction to connect to databases not allowing user connections. A background worker can only call one of these two functions, and only once. It is not possible to switch databases.

Signals are initially blocked when control reaches the background worker's main function, and must be unblocked by it; this is to allow the process to customize its signal handlers, if necessary. Signals can be unblocked in the new process by calling BackgroundWorkerUnblockSignals and blocked by calling BackgroundWorkerBlockSignals.

If bgw_restart_time for a background worker is configured as BGW_NEVER_RESTART, or if it exits with an exit code of 0 or is terminated by TerminateBackgroundWorker, it will be automatically unregistered by the postmaster on exit. Otherwise, it will be restarted after the time period configured via bgw_restart_time, or immediately if the postmaster reinitializes the cluster due to a backend failure. Backends which need to suspend execution only temporarily should use an interruptible sleep rather than exiting; this can be achieved by calling WaitLatch(). Make sure the WL_POSTMASTER_DEATH flag is set when calling that function, and verify the return code for a prompt exit in the emergency case that postgres itself has terminated.

When a background worker is registered using the RegisterDynamicBackgroundWorker function, it is possible for the backend performing the registration to obtain information regarding the status of the worker. Backends wishing to do this should pass the address of a BackgroundWorkerHandle * as the second argument to RegisterDynamicBackgroundWorker. If the worker is successfully registered, this pointer will be initialized with an opaque handle that can subsequently be passed to GetBackgroundWorkerPid(BackgroundWorkerHandle *, pid_t *) or TerminateBackgroundWorker(BackgroundWorkerHandle *). GetBackgroundWorkerPid can be used to poll the status of the worker: a return value of BGWH_NOT_YET_STARTED indicates that the worker has not yet been started by the postmaster; BGWH_STOPPED indicates that it has been started but is no longer running; and BGWH_STARTED indicates that it is currently running. In this last case, the PID will also be returned via the second argument. TerminateBackgroundWorker causes the postmaster to send SIGTERM to the worker if it is running, and to unregister it as soon as it is not.

In some cases, a process which registers a background worker may wish to wait for the worker to start up. This can be accomplished by initializing bgw_notify_pid to MyProcPid and then passing the BackgroundWorkerHandle * obtained at registration time to WaitForBackgroundWorkerStartup(BackgroundWorkerHandle *handle, pid_t *) function. This function will block until the postmaster has attempted to start the background worker, or until the postmaster dies. If the background worker is running, the return value will be BGWH_STARTED, and the PID will be written to the provided address. Otherwise, the return value will be BGWH_STOPPED or BGWH_POSTMASTER_DIED.

A process can also wait for a background worker to shut down, by using the WaitForBackgroundWorkerShutdown(BackgroundWorkerHandle *handle) function and passing the BackgroundWorkerHandle * obtained at registration. This function will block until the background worker exits, or postmaster dies. When the background worker exits, the return value is BGWH_STOPPED, if postmaster dies it will return BGWH_POSTMASTER_DIED.

Background workers can send asynchronous notification messages, either by using the NOTIFY command via SPI, or directly via Async_Notify(). Such notifications will be sent at transaction commit. Background workers should not register to receive asynchronous notifications with the LISTEN command, as there is no infrastructure for a worker to consume such notifications.

The src/test/modules/worker_spi module contains a working example, which demonstrates some useful techniques.

The maximum number of registered background workers is limited by max_worker_processes.

Prev | Up | Next
SPI_start_transaction | Home | Chapter 49. Logical Decoding


Chapter 49. Logical Decoding
Prev Up Part V. Server Programming Home Next

Chapter 49. Logical Decoding

Table of Contents

49.1. Logical Decoding Examples
49.2. Logical Decoding Concepts
49.2.1. Logical Decoding
49.2.2. Replication Slots
49.2.3. Output Plugins
49.2.4. Exported Snapshots
49.3. Streaming Replication Protocol Interface
49.4. Logical Decoding SQL Interface
49.5. System Catalogs Related to Logical Decoding
49.6. Logical Decoding Output Plugins
49.6.1. Initialization Function
49.6.2. Capabilities
49.6.3. Output Modes
49.6.4. Output Plugin Callbacks
49.6.5. Functions for Producing Output
49.7. Logical Decoding Output Writers
49.8. Synchronous Replication Support for Logical Decoding
49.8.1. Overview
49.8.2. Caveats
49.9. Streaming of Large Transactions for Logical Decoding
49.10. Two-phase Commit Support for Logical Decoding

PostgreSQL provides infrastructure to stream the modifications performed via SQL to external consumers. This functionality can be used for a variety of purposes, including replication solutions and auditing.

Changes are sent out in streams identified by logical replication slots.

The format in which those changes are streamed is determined by the output plugin used. An example plugin is provided in the PostgreSQL distribution. Additional plugins can be written to extend the choice of available formats without modifying any core code. Every output plugin has access to each individual new row produced by INSERT and the new row version created by UPDATE. Availability of old row versions for UPDATE and DELETE depends on the configured replica identity (see REPLICA IDENTITY).

Changes can be consumed either using the streaming replication protocol (see Section 53.4 and Section 49.3), or by calling functions via SQL (see Section 49.4). It is also possible to write additional methods of consuming the output of a replication slot without modifying core code (see Section 49.7).

Prev | Up | Next
Chapter 48. Background Worker Processes | Home | 49.1. Logical Decoding Examples


49.1. Logical Decoding Examples
Prev Up Chapter 49. Logical Decoding Home Next

49.1. Logical Decoding Examples

The following example demonstrates controlling logical decoding using the SQL interface.

Before you can use logical decoding, you must set wal_level to logical and max_replication_slots to at least 1. Then, you should connect to the target database (in the example below, postgres) as a superuser.

postgres=# -- Create a slot named 'regression_slot' using the output plugin 'test_decoding'
postgres=# SELECT * FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding', false, true);
    slot_name    |    lsn
-----------------+-----------
 regression_slot | 0/16B1970
(1 row)

postgres=# SELECT slot_name, plugin, slot_type, database, active, restart_lsn, confirmed_flush_lsn FROM pg_replication_slots;
    slot_name    |    plugin     | slot_type | database | active | restart_lsn | confirmed_flush_lsn
-----------------+---------------+-----------+----------+--------+-------------+-----------------
 regression_slot | test_decoding | logical   | postgres | f      | 0/16A4408   | 0/16A4440
(1 row)

postgres=# -- There are no changes to see yet
postgres=# SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
 lsn | xid | data
-----+-----+------
(0 rows)

postgres=# CREATE TABLE data(id serial primary key, data text);
CREATE TABLE

postgres=# -- DDL isn't replicated, so all you'll see is the transaction
postgres=# SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
    lsn    |  xid  |     data
-----------+-------+--------------
 0/BA2DA58 | 10297 | BEGIN 10297
 0/BA5A5A0 | 10297 | COMMIT 10297
(2 rows)

postgres=# -- Once changes are read, they're consumed and not emitted
postgres=# -- in a subsequent call:
postgres=# SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
 lsn | xid | data
-----+-----+------
(0 rows)

postgres=# BEGIN;
postgres=*# INSERT INTO data(data) VALUES('1');
postgres=*# INSERT INTO data(data) VALUES('2');
postgres=*# COMMIT;

postgres=# SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
    lsn    |  xid  |                          data
-----------+-------+---------------------------------------------------------
 0/BA5A688 | 10298 | BEGIN 10298
 0/BA5A6F0 | 10298 | table public.data: INSERT: id[integer]:1 data[text]:'1'
 0/BA5A7F8 | 10298 | table public.data: INSERT: id[integer]:2 data[text]:'2'
 0/BA5A8A8 | 10298 | COMMIT 10298
(4 rows)

postgres=# INSERT INTO data(data) VALUES('3');

postgres=# -- You can also peek ahead in the change stream without consuming changes
postgres=# SELECT * FROM pg_logical_slot_peek_changes('regression_slot', NULL, NULL);
    lsn    |  xid  |                          data
-----------+-------+---------------------------------------------------------
 0/BA5A8E0 | 10299 | BEGIN 10299
 0/BA5A8E0 | 10299 | table public.data: INSERT: id[integer]:3 data[text]:'3'
 0/BA5A990 | 10299 | COMMIT 10299
(3 rows)

postgres=# -- The next call to pg_logical_slot_peek_changes() returns the same changes again
postgres=# SELECT * FROM pg_logical_slot_peek_changes('regression_slot', NULL, NULL);
    lsn    |  xid  |                          data
-----------+-------+---------------------------------------------------------
 0/BA5A8E0 | 10299 | BEGIN 10299
 0/BA5A8E0 | 10299 | table public.data: INSERT: id[integer]:3 data[text]:'3'
 0/BA5A990 | 10299 | COMMIT 10299
(3 rows)

postgres=# -- options can be passed to output plugin, to influence the formatting
postgres=# SELECT * FROM pg_logical_slot_peek_changes('regression_slot', NULL, NULL, 'include-timestamp', 'on');
    lsn    |  xid  |                          data
-----------+-------+---------------------------------------------------------
 0/BA5A8E0 | 10299 | BEGIN 10299
 0/BA5A8E0 | 10299 | table public.data: INSERT: id[integer]:3 data[text]:'3'
 0/BA5A990 | 10299 | COMMIT 10299 (at 2017-05-10 12:07:21.272494-04)
(3 rows)

postgres=# -- Remember to destroy a slot you no longer need to stop it consuming
postgres=# -- server resources:
postgres=# SELECT pg_drop_replication_slot('regression_slot');
 pg_drop_replication_slot
-----------------------

(1 row)

The following example shows how logical decoding is controlled over the streaming replication protocol, using the program pg_recvlogical included in the PostgreSQL distribution. This requires that client authentication is set up to allow replication connections (see Section 27.2.5.1) and that max_wal_senders is set sufficiently high to allow an additional connection.

$ pg_recvlogical -d postgres --slot=test --create-slot
$ pg_recvlogical -d postgres --slot=test --start -f -
Control+Z
$ psql -d postgres -c "INSERT INTO data(data) VALUES('4');"
$ fg
BEGIN 693
table public.data: INSERT: id[integer]:4 data[text]:'4'
COMMIT 693
Control+C
$ pg_recvlogical -d postgres --slot=test --drop-slot

The following example shows SQL interface that can be used to decode prepared transactions. Before you use two-phase commit commands, you must set max_prepared_transactions to at least 1. You must also have set the two-phase parameter as 'true' while creating the slot using pg_create_logical_replication_slot Note that we will stream the entire transaction after the commit if it is not already decoded.

postgres=# BEGIN;
postgres=*# INSERT INTO data(data) VALUES('5');
postgres=*# PREPARE TRANSACTION 'test_prepared1';

postgres=# SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
    lsn    | xid |                          data
-----------+-----+---------------------------------------------------------
 0/1689DC0 | 529 | BEGIN 529
 0/1689DC0 | 529 | table public.data: INSERT: id[integer]:3 data[text]:'5'
 0/1689FC0 | 529 | PREPARE TRANSACTION 'test_prepared1', txid 529
(3 rows)

postgres=# COMMIT PREPARED 'test_prepared1';
postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NULL);
    lsn    | xid |                    data
-----------+-----+--------------------------------------------
 0/168A060 | 529 | COMMIT PREPARED 'test_prepared1', txid 529
(4 row)

postgres=#-- you can also rollback a prepared transaction
postgres=# BEGIN;
postgres=*# INSERT INTO data(data) VALUES('6');
postgres=*# PREPARE TRANSACTION 'test_prepared2';
postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NULL);
    lsn    | xid |                          data
-----------+-----+---------------------------------------------------------
 0/168A180 | 530 | BEGIN 530
 0/168A1E8 | 530 | table public.data: INSERT: id[integer]:4 data[text]:'6'
 0/168A430 | 530 | PREPARE TRANSACTION 'test_prepared2', txid 530
(3 rows)

postgres=# ROLLBACK PREPARED 'test_prepared2';
postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NULL);
    lsn    | xid |                     data
-----------+-----+----------------------------------------------
 0/168A4B8 | 530 | ROLLBACK PREPARED 'test_prepared2', txid 530
(1 row)

Prev | Up | Next
Chapter 49. Logical Decoding | Home | 49.2. Logical Decoding Concepts


49.2. Logical Decoding Concepts
Prev Up Chapter 49. Logical Decoding Home Next

49.2. Logical Decoding Concepts

49.2.1. Logical Decoding
49.2.2. Replication Slots
49.2.3. Output Plugins
49.2.4. Exported Snapshots

49.2.1. Logical Decoding

Logical decoding is the process of extracting all persistent changes to a database's tables into a coherent, easy to understand format which can be interpreted without detailed knowledge of the database's internal state.

In PostgreSQL, logical decoding is implemented by decoding the contents of the write-ahead log, which describe changes on a storage level, into an application-specific form such as a stream of tuples or SQL statements.

49.2.2. Replication Slots

In the context of logical replication, a slot represents a stream of changes that can be replayed to a client in the order they were made on the origin server. Each slot streams a sequence of changes from a single database.

Note

PostgreSQL also has streaming replication slots (see Section 27.2.5), but they are used somewhat differently there.

A replication slot has an identifier that is unique across all databases in a PostgreSQL cluster. Slots persist independently of the connection using them and are crash-safe.

A logical slot will emit each change just once in normal operation. The current position of each slot is persisted only at checkpoint, so in the case of a crash the slot may return to an earlier LSN, which will then cause recent changes to be sent again when the server restarts. Logical decoding clients are responsible for avoiding ill effects from handling the same message more than once. Clients may wish to record the last LSN they saw when decoding and skip over any repeated data or (when using the replication protocol) request that decoding start from that LSN rather than letting the server determine the start point. The Replication Progress Tracking feature is designed for this purpose, refer to replication origins.

Multiple independent slots may exist for a single database. Each slot has its own state, allowing different consumers to receive changes from different points in the database change stream. For most applications, a separate slot will be required for each consumer.

A logical replication slot knows nothing about the state of the receiver(s). It's even possible to have multiple different receivers using the same slot at different times; they'll just get the changes following on from when the last receiver stopped consuming them. Only one receiver may consume changes from a slot at any given time.

Caution

Replication slots persist across crashes and know nothing about the state of their consumer(s). They will prevent removal of required resources even when there is no connection using them. This consumes storage because neither required WAL nor required rows from the system catalogs can be removed by VACUUM as long as they are required by a replication slot. In extreme cases this could cause the database to shut down to prevent transaction ID wraparound (see Section 25.1.5). So if a slot is no longer required it should be dropped.

49.2.3. Output Plugins

Output plugins transform the data from the write-ahead log's internal representation into the format the consumer of a replication slot desires.

49.2.4. Exported Snapshots

When a new replication slot is created using the streaming replication interface (see CREATE_REPLICATION_SLOT), a snapshot is exported (see Section 9.27.5), which will show exactly the state of the database after which all changes will be included in the change stream. This can be used to create a new replica by using SET TRANSACTION SNAPSHOT to read the state of the database at the moment the slot was created. This transaction can then be used to dump the database's state at that point in time, which afterwards can be updated using the slot's contents without losing any changes.

Creation of a snapshot is not always possible. In particular, it will fail when connected to a hot standby. Applications that do not require snapshot export may suppress it with the NOEXPORT_SNAPSHOT option.

Prev | Up | Next
49.1. Logical Decoding Examples | Home | 49.3. Streaming Replication Protocol Interface


49.3. Streaming Replication Protocol Interface
Prev Up Chapter 49. Logical Decoding Home Next

49.3. Streaming Replication Protocol Interface

The commands

  • CREATE_REPLICATION_SLOT slot_name LOGICAL output_plugin

  • DROP_REPLICATION_SLOT slot_name [ WAIT ]

  • START_REPLICATION SLOT slot_name LOGICAL ...

are used to create, drop, and stream changes from a replication slot, respectively. These commands are only available over a replication connection; they cannot be used via SQL. See Section 53.4 for details on these commands.

The command pg_recvlogical can be used to control logical decoding over a streaming replication connection. (It uses these commands internally.)

Prev | Up | Next
49.2. Logical Decoding Concepts | Home | 49.4. Logical Decoding SQL Interface


49.4. Logical Decoding SQL Interface
Prev Up Chapter 49. Logical Decoding Home Next

49.4. Logical Decoding SQL Interface

See Section 9.27.6 for detailed documentation on the SQL-level API for interacting with logical decoding.

Synchronous replication (see Section 27.2.8) is only supported on replication slots used over the streaming replication interface. The function interface and additional, non-core interfaces do not support synchronous replication.

Prev | Up | Next
49.3. Streaming Replication Protocol Interface | Home | 49.5. System Catalogs Related to Logical Decoding


49.5. System Catalogs Related to Logical Decoding
Prev Up Chapter 49. Logical Decoding Home Next

49.5. System Catalogs Related to Logical Decoding

The pg_replication_slots view and the pg_stat_replication view provide information about the current state of replication slots and streaming replication connections respectively. These views apply to both physical and logical replication. The pg_stat_replication_slots view provides statistics information about the logical replication slots.

Prev | Up | Next
49.4. Logical Decoding SQL Interface | Home | 49.6. Logical Decoding Output Plugins


49.6. Logical Decoding Output Plugins
Prev Up Chapter 49. Logical Decoding Home Next

49.6. Logical Decoding Output Plugins

49.6.1. Initialization Function
49.6.2. Capabilities
49.6.3. Output Modes
49.6.4. Output Plugin Callbacks
49.6.5. Functions for Producing Output

An example output plugin can be found in the contrib/test_decoding subdirectory of the PostgreSQL source tree.

49.6.1. Initialization Function

An output plugin is loaded by dynamically loading a shared library with the output plugin's name as the library base name. The normal library search path is used to locate the library. To provide the required output plugin callbacks and to indicate that the library is actually an output plugin it needs to provide a function named _PG_output_plugin_init. This function is passed a struct that needs to be filled with the callback function pointers for individual actions.

typedef struct OutputPluginCallbacks
{
    LogicalDecodeStartupCB startup_cb;
    LogicalDecodeBeginCB begin_cb;
    LogicalDecodeChangeCB change_cb;
    LogicalDecodeTruncateCB truncate_cb;
    LogicalDecodeCommitCB commit_cb;
    LogicalDecodeMessageCB message_cb;
    LogicalDecodeFilterByOriginCB filter_by_origin_cb;
    LogicalDecodeShutdownCB shutdown_cb;
    LogicalDecodeFilterPrepareCB filter_prepare_cb;
    LogicalDecodeBeginPrepareCB begin_prepare_cb;
    LogicalDecodePrepareCB prepare_cb;
    LogicalDecodeCommitPreparedCB commit_prepared_cb;
    LogicalDecodeRollbackPreparedCB rollback_prepared_cb;
    LogicalDecodeStreamStartCB stream_start_cb;
    LogicalDecodeStreamStopCB stream_stop_cb;
    LogicalDecodeStreamAbortCB stream_abort_cb;
    LogicalDecodeStreamPrepareCB stream_prepare_cb;
    LogicalDecodeStreamCommitCB stream_commit_cb;
    LogicalDecodeStreamChangeCB stream_change_cb;
    LogicalDecodeStreamMessageCB stream_message_cb;
    LogicalDecodeStreamTruncateCB stream_truncate_cb;
} OutputPluginCallbacks;

typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);

The begin_cb, change_cb and commit_cb callbacks are required, while startup_cb, filter_by_origin_cb, truncate_cb, and shutdown_cb are optional. If truncate_cb is not set but a TRUNCATE is to be decoded, the action will be ignored.

An output plugin may also define functions to support streaming of large, in-progress transactions. The stream_start_cb, stream_stop_cb, stream_abort_cb, stream_commit_cb, stream_change_cb, and stream_prepare_cb are required, while stream_message_cb and stream_truncate_cb are optional.

An output plugin may also define functions to support two-phase commits, which allows actions to be decoded on the PREPARE TRANSACTION. The begin_prepare_cb, prepare_cb, stream_prepare_cb, commit_prepared_cb and rollback_prepared_cb callbacks are required, while filter_prepare_cb is optional.

49.6.2. Capabilities

To decode, format and output changes, output plugins can use most of the backend's normal infrastructure, including calling output functions. Read only access to relations is permitted as long as only relations are accessed that either have been created by initdb in the pg_catalog schema, or have been marked as user provided catalog tables using

ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);

Note that access to user catalog tables or regular system catalog tables in the output plugins has to be done via the systable_* scan APIs only. Access via the heap_* scan APIs will error out. Additionally, any actions leading to transaction ID assignment are prohibited. That, among others, includes writing to tables, performing DDL changes, and calling pg_current_xact_id().

49.6.3. Output Modes

Output plugin callbacks can pass data to the consumer in nearly arbitrary formats. For some use cases, like viewing the changes via SQL, returning data in a data type that can contain arbitrary data (e.g., bytea) is cumbersome. If the output plugin only outputs textual data in the server's encoding, it can declare that by setting OutputPluginOptions.output_type to OUTPUT_PLUGIN_TEXTUAL_OUTPUT instead of OUTPUT_PLUGIN_BINARY_OUTPUT in the startup callback. In that case, all the data has to be in the server's encoding so that a text datum can contain it. This is checked in assertion-enabled builds.

49.6.4. Output Plugin Callbacks

An output plugin gets notified about changes that are happening via various callbacks it needs to provide.

Concurrent transactions are decoded in commit order, and only changes belonging to a specific transaction are decoded between the begin and commit callbacks. Transactions that were rolled back explicitly or implicitly never get decoded. Successful savepoints are folded into the transaction containing them in the order they were executed within that transaction. A transaction that is prepared for a two-phase commit using PREPARE TRANSACTION will also be decoded if the output plugin callbacks needed for decoding them are provided. It is possible that the current prepared transaction which is being decoded is aborted concurrently via a ROLLBACK PREPARED command. In that case, the logical decoding of this transaction will be aborted too. All the changes of such a transaction are skipped once the abort is detected and the prepare_cb callback is invoked. Thus even in case of a concurrent abort, enough information is provided to the output plugin for it to properly deal with ROLLBACK PREPARED once that is decoded.

Note

Only transactions that have already safely been flushed to disk will be decoded. That can lead to a COMMIT not immediately being decoded in a directly following pg_logical_slot_get_changes() when synchronous_commit is set to off.

49.6.4.1. Startup Callback

The optional startup_cb callback is called whenever a replication slot is created or asked to stream changes, independent of the number of changes that are ready to be put out.

typedef void (*LogicalDecodeStartupCB) (struct LogicalDecodingContext *ctx,
                                        OutputPluginOptions *options,
                                        bool is_init);

The is_init parameter will be true when the replication slot is being created and false otherwise. options points to a struct of options that output plugins can set:

typedef struct OutputPluginOptions
{
    OutputPluginOutputType output_type;
    bool        receive_rewrites;
} OutputPluginOptions;

output_type has to either be set to OUTPUT_PLUGIN_TEXTUAL_OUTPUT or OUTPUT_PLUGIN_BINARY_OUTPUT. See also Section 49.6.3. If receive_rewrites is true, the output plugin will also be called for changes made by heap rewrites during certain DDL operations. These are of interest to plugins that handle DDL replication, but they require special handling.

The startup callback should validate the options present in ctx->output_plugin_options. If the output plugin needs to have a state, it can use ctx->output_plugin_private to store it.

49.6.4.2. Shutdown Callback

The optional shutdown_cb callback is called whenever a formerly active replication slot is not used anymore and can be used to deallocate resources private to the output plugin. The slot isn't necessarily being dropped, streaming is just being stopped.

typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);

49.6.4.3. Transaction Begin Callback

The required begin_cb callback is called whenever a start of a committed transaction has been decoded. Aborted transactions and their contents never get decoded.

typedef void (*LogicalDecodeBeginCB) (struct LogicalDecodingContext *ctx,
                                      ReorderBufferTXN *txn);

The txn parameter contains meta information about the transaction, like the time stamp at which it has been committed and its XID.

49.6.4.4. Transaction End Callback

The required commit_cb callback is called whenever a transaction commit has been decoded. The change_cb callbacks for all modified rows will have been called before this, if there have been any modified rows.

typedef void (*LogicalDecodeCommitCB) (struct LogicalDecodingContext *ctx,
                                       ReorderBufferTXN *txn,
                                       XLogRecPtr commit_lsn);

49.6.4.5. Change Callback

The required change_cb callback is called for every individual row modification inside a transaction, may it be an INSERT, UPDATE, or DELETE. Even if the original command modified several rows at once the callback will be called individually for each row. The change_cb callback may access system or user catalog tables to aid in the process of outputting the row modification details. In case of decoding a prepared (but yet uncommitted) transaction or decoding of an uncommitted transaction, this change callback might also error out due to simultaneous rollback of this very same transaction. In that case, the logical decoding of this aborted transaction is stopped gracefully.

typedef void (*LogicalDecodeChangeCB) (struct LogicalDecodingContext *ctx,
                                       ReorderBufferTXN *txn,
                                       Relation relation,
                                       ReorderBufferChange *change);

The ctx and txn parameters have the same contents as for the begin_cb and commit_cb callbacks, but additionally the relation descriptor relation points to the relation the row belongs to and a struct change describing the row modification are passed in.

Note

Only changes in user defined tables that are not unlogged (see UNLOGGED) and not temporary (see TEMPORARY or TEMP) can be extracted using logical decoding.

49.6.4.6. Truncate Callback

The truncate_cb callback is called for a TRUNCATE command.

typedef void (*LogicalDecodeTruncateCB) (struct LogicalDecodingContext *ctx,
                                         ReorderBufferTXN *txn,
                                         int nrelations,
                                         Relation relations[],
                                         ReorderBufferChange *change);

The parameters are analogous to the change_cb callback. However, because TRUNCATE actions on tables connected by foreign keys need to be executed together, this callback receives an array of relations instead of just a single one. See the description of the TRUNCATE statement for details.

49.6.4.7. Origin Filter Callback

The optional filter_by_origin_cb callback is called to determine whether data that has been replayed from origin_id is of interest to the output plugin.

typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ctx,
                                               RepOriginId origin_id);

The ctx parameter has the same contents as for the other callbacks. No information but the origin is available. To signal that changes originating on the passed in node are irrelevant, return true, causing them to be filtered away; false otherwise. The other callbacks will not be called for transactions and changes that have been filtered away.

This is useful when implementing cascading or multidirectional replication solutions. Filtering by the origin allows to prevent replicating the same changes back and forth in such setups. While transactions and changes also carry information about the origin, filtering via this callback is noticeably more efficient.

49.6.4.8. Generic Message Callback

The optional message_cb callback is called whenever a logical decoding message has been decoded.

typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
                                        ReorderBufferTXN *txn,
                                        XLogRecPtr message_lsn,
                                        bool transactional,
                                        const char *prefix,
                                        Size message_size,
                                        const char *message);

The txn parameter contains meta information about the transaction, like the time stamp at which it has been committed and its XID. Note however that it can be NULL when the message is non-transactional and the XID was not assigned yet in the transaction which logged the message. The lsn has WAL location of the message. The transactional says if the message was sent as transactional or not. Similar to the change callback, in case of decoding a prepared (but yet uncommitted) transaction or decoding of an uncommitted transaction, this message callback might also error out due to simultaneous rollback of this very same transaction. In that case, the logical decoding of this aborted transaction is stopped gracefully. The prefix is arbitrary null-terminated prefix which can be used for identifying interesting messages for the current plugin. And finally the message parameter holds the actual message of message_size size.

Extra care should be taken to ensure that the prefix the output plugin considers interesting is unique. Using name of the extension or the output plugin itself is often a good choice.

49.6.4.9. Prepare Filter Callback

The optional filter_prepare_cb callback is called to determine whether data that is part of the current two-phase commit transaction should be considered for decoding at this prepare stage or later as a regular one-phase transaction at COMMIT PREPARED time. To signal that decoding should be skipped, return true; false otherwise. When the callback is not defined, false is assumed (i.e. no filtering, all transactions using two-phase commit are decoded in two phases as well).

typedef bool (*LogicalDecodeFilterPrepareCB) (struct LogicalDecodingContext *ctx,
                                              TransactionId xid,
                                              const char *gid);

The ctx parameter has the same contents as for the other callbacks. The parameters xid and gid provide two different ways to identify the transaction. The later COMMIT PREPARED or ROLLBACK PREPARED carries both identifiers, providing an output plugin the choice of what to use.

The callback may be invoked multiple times per transaction to decode and must provide the same static answer for a given pair of xid and gid every time it is called.

49.6.4.10. Transaction Begin Prepare Callback

The required begin_prepare_cb callback is called whenever the start of a prepared transaction has been decoded. The gid field, which is part of the txn parameter, can be used in this callback to check if the plugin has already received this PREPARE in which case it can either error out or skip the remaining changes of the transaction.

typedef void (*LogicalDecodeBeginPrepareCB) (struct LogicalDecodingContext *ctx,
                                             ReorderBufferTXN *txn);

49.6.4.11. Transaction Prepare Callback

The required prepare_cb callback is called whenever a transaction which is prepared for two-phase commit has been decoded. The change_cb callback for all modified rows will have been called before this, if there have been any modified rows. The gid field, which is part of the txn parameter, can be used in this callback.

typedef void (*LogicalDecodePrepareCB) (struct LogicalDecodingContext *ctx,
                                        ReorderBufferTXN *txn,
                                        XLogRecPtr prepare_lsn);

49.6.4.12. Transaction Commit Prepared Callback

The required commit_prepared_cb callback is called whenever a transaction COMMIT PREPARED has been decoded. The gid field, which is part of the txn parameter, can be used in this callback.

typedef void (*LogicalDecodeCommitPreparedCB) (struct LogicalDecodingContext *ctx,
                                               ReorderBufferTXN *txn,
                                               XLogRecPtr commit_lsn);

49.6.4.13. Transaction Rollback Prepared Callback

The required rollback_prepared_cb callback is called whenever a transaction ROLLBACK PREPARED has been decoded. The gid field, which is part of the txn parameter, can be used in this callback. The parameters prepare_end_lsn and prepare_time can be used to check if the plugin has received this PREPARE TRANSACTION in which case it can apply the rollback, otherwise, it can skip the rollback operation. The gid alone is not sufficient because the downstream node can have a prepared transaction with same identifier.

typedef void (*LogicalDecodeRollbackPreparedCB) (struct LogicalDecodingContext *ctx,
                                                 ReorderBufferTXN *txn,
                                                 XLogRecPtr prepare_end_lsn,
                                                 TimestampTz prepare_time);

49.6.4.14. Stream Start Callback

The stream_start_cb callback is called when opening a block of streamed changes from an in-progress transaction.

typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
                                            ReorderBufferTXN *txn);

49.6.4.15. Stream Stop Callback

The stream_stop_cb callback is called when closing a block of streamed changes from an in-progress transaction.

typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
                                           ReorderBufferTXN *txn);

49.6.4.16. Stream Abort Callback

The stream_abort_cb callback is called to abort a previously streamed transaction.

typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
                                            ReorderBufferTXN *txn,
                                            XLogRecPtr abort_lsn);

49.6.4.17. Stream Prepare Callback

The stream_prepare_cb callback is called to prepare a previously streamed transaction as part of a two-phase commit.

typedef void (*LogicalDecodeStreamPrepareCB) (struct LogicalDecodingContext *ctx,
                                              ReorderBufferTXN *txn,
                                              XLogRecPtr prepare_lsn);

49.6.4.18. Stream Commit Callback

The stream_commit_cb callback is called to commit a previously streamed transaction.

typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
                                             ReorderBufferTXN *txn,
                                             XLogRecPtr commit_lsn);

49.6.4.19. Stream Change Callback

The stream_change_cb callback is called when sending a change in a block of streamed changes (demarcated by stream_start_cb and stream_stop_cb calls). The actual changes are not displayed as the transaction can abort at a later point in time and we don't decode changes for aborted transactions.

typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
                                             ReorderBufferTXN *txn,
                                             Relation relation,
                                             ReorderBufferChange *change);

49.6.4.20. Stream Message Callback

The stream_message_cb callback is called when sending a generic message in a block of streamed changes (demarcated by stream_start_cb and stream_stop_cb calls). The message contents for transactional messages are not displayed as the transaction can abort at a later point in time and we don't decode changes for aborted transactions.

typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
                                              ReorderBufferTXN *txn,
                                              XLogRecPtr message_lsn,
                                              bool transactional,
                                              const char *prefix,
                                              Size message_size,
                                              const char *message);

49.6.4.21. Stream Truncate Callback

The stream_truncate_cb callback is called for a TRUNCATE command in a block of streamed changes (demarcated by stream_start_cb and stream_stop_cb calls).

typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
                                               ReorderBufferTXN *txn,
                                               int nrelations,
                                               Relation relations[],
                                               ReorderBufferChange *change);

The parameters are analogous to the stream_change_cb callback. However, because TRUNCATE actions on tables connected by foreign keys need to be executed together, this callback receives an array of relations instead of just a single one. See the description of the TRUNCATE statement for details.

49.6.5. Functions for Producing Output

To actually produce output, output plugins can write data to the StringInfo output buffer in ctx->out when inside the begin_cb, commit_cb, or change_cb callbacks. Before writing to the output buffer, OutputPluginPrepareWrite(ctx, last_write) has to be called, and after finishing writing to the buffer, OutputPluginWrite(ctx, last_write) has to be called to perform the write. The last_write indicates whether a particular write was the callback's last write.

The following example shows how to output data to the consumer of an output plugin:

OutputPluginPrepareWrite(ctx, true);
appendStringInfo(ctx->out, "BEGIN %u", txn->xid);
OutputPluginWrite(ctx, true);

Prev | Up | Next
49.5. System Catalogs Related to Logical Decoding | Home | 49.7. Logical Decoding Output Writers


49.7. Logical Decoding Output Writers
Prev Up Chapter 49. Logical Decoding Home Next

49.7. Logical Decoding Output Writers

It is possible to add more output methods for logical decoding. For details, see src/backend/replication/logical/logicalfuncs.c. Essentially, three functions need to be provided: one to read WAL, one to prepare writing output, and one to write the output (see Section 49.6.5).

Prev | Up | Next
49.6. Logical Decoding Output Plugins | Home | 49.8. Synchronous Replication Support for Logical Decoding


49.8. Synchronous Replication Support for Logical Decoding
Prev Up Chapter 49. Logical Decoding Home Next

49.8. Synchronous Replication Support for Logical Decoding

49.8.1. Overview
49.8.2. Caveats

49.8.1. Overview

Logical decoding can be used to build synchronous replication solutions with the same user interface as synchronous replication for streaming replication. To do this, the streaming replication interface (see Section 49.3) must be used to stream out data. Clients have to send Standby status update (F) (see Section 53.4) messages, just like streaming replication clients do.

Note

A synchronous replica receiving changes via logical decoding will work in the scope of a single database. Since, in contrast to that, synchronous_standby_names currently is server wide, this means this technique will not work properly if more than one database is actively used.

49.8.2. Caveats

In synchronous replication setup, a deadlock can happen, if the transaction has locked [user] catalog tables exclusively. See Section 49.6.2 for information on user catalog tables. This is because logical decoding of transactions can lock catalog tables to access them. To avoid this users must refrain from taking an exclusive lock on [user] catalog tables. This can happen in the following ways:

  • Issuing an explicit LOCK on pg_class in a transaction.

  • Perform CLUSTER on pg_class in a transaction.

  • PREPARE TRANSACTION after LOCK command on pg_class and allow logical decoding of two-phase transactions.

  • PREPARE TRANSACTION after CLUSTER command on pg_trigger and allow logical decoding of two-phase transactions. This will lead to deadlock only when published table have a trigger.

  • Executing TRUNCATE on [user] catalog table in a transaction.

Note that these commands that can cause deadlock apply to not only explicitly indicated system catalog tables above but also to any other [user] catalog table.

Prev | Up | Next
49.7. Logical Decoding Output Writers | Home | 49.9. Streaming of Large Transactions for Logical Decoding


49.9. Streaming of Large Transactions for Logical Decoding
Prev Up Chapter 49. Logical Decoding Home Next

49.9. Streaming of Large Transactions for Logical Decoding

The basic output plugin callbacks (e.g., begin_cb, change_cb, commit_cb and message_cb) are only invoked when the transaction actually commits. The changes are still decoded from the transaction log, but are only passed to the output plugin at commit (and discarded if the transaction aborts).

This means that while the decoding happens incrementally, and may spill to disk to keep memory usage under control, all the decoded changes have to be transmitted when the transaction finally commits (or more precisely, when the commit is decoded from the transaction log). Depending on the size of the transaction and network bandwidth, the transfer time may significantly increase the apply lag.

To reduce the apply lag caused by large transactions, an output plugin may provide additional callback to support incremental streaming of in-progress transactions. There are multiple required streaming callbacks (stream_start_cb, stream_stop_cb, stream_abort_cb, stream_commit_cb and stream_change_cb) and two optional callbacks (stream_message_cb and stream_truncate_cb).

When streaming an in-progress transaction, the changes (and messages) are streamed in blocks demarcated by stream_start_cb and stream_stop_cb callbacks. Once all the decoded changes are transmitted, the transaction can be committed using the the stream_commit_cb callback (or possibly aborted using the stream_abort_cb callback). If two-phase commits are supported, the transaction can be prepared using the stream_prepare_cb callback, COMMIT PREPARED using the commit_prepared_cb callback or aborted using the rollback_prepared_cb.

One example sequence of streaming callback calls for one transaction may look like this:

stream_start_cb(...);   <-- start of first block of changes
  stream_change_cb(...);
  stream_change_cb(...);
  stream_message_cb(...);
  stream_change_cb(...);
  ...
  stream_change_cb(...);
stream_stop_cb(...);    <-- end of first block of changes

stream_start_cb(...);   <-- start of second block of changes
  stream_change_cb(...);
  stream_change_cb(...);
  stream_change_cb(...);
  ...
  stream_message_cb(...);
  stream_change_cb(...);
stream_stop_cb(...);    <-- end of second block of changes

stream_commit_cb(...);  <-- commit of the streamed transaction

The actual sequence of callback calls may be more complicated, of course. There may be blocks for multiple streamed transactions, some of the transactions may get aborted, etc.

Similar to spill-to-disk behavior, streaming is triggered when the total amount of changes decoded from the WAL (for all in-progress transactions) exceeds the limit defined by logical_decoding_work_mem setting. At that point, the largest top-level transaction (measured by the amount of memory currently used for decoded changes) is selected and streamed. However, in some cases we still have to spill to disk even if streaming is enabled because we exceed the memory threshold but still have not decoded the complete tuple e.g., only decoded toast table insert but not the main table insert.

Even when streaming large transactions, the changes are still applied in commit order, preserving the same guarantees as the non-streaming mode.

Prev | Up | Next
49.8. Synchronous Replication Support for Logical Decoding | Home | 49.10. Two-phase Commit Support for Logical Decoding


49.10. Two-phase Commit Support for Logical Decoding
Prev Up Chapter 49. Logical Decoding Home Next

49.10. Two-phase Commit Support for Logical Decoding

With the basic output plugin callbacks (eg., begin_cb, change_cb, commit_cb and message_cb) two-phase commit commands like PREPARE TRANSACTION, COMMIT PREPARED and ROLLBACK PREPARED are not decoded. While the PREPARE TRANSACTION is ignored, COMMIT PREPARED is decoded as a COMMIT and ROLLBACK PREPARED is decoded as a ROLLBACK.

To support the streaming of two-phase commands, an output plugin needs to provide additional callbacks. There are multiple two-phase commit callbacks that are required, (begin_prepare_cb, prepare_cb, commit_prepared_cb, rollback_prepared_cb and stream_prepare_cb) and an optional callback (filter_prepare_cb).

If the output plugin callbacks for decoding two-phase commit commands are provided, then on PREPARE TRANSACTION, the changes of that transaction are decoded, passed to the output plugin, and the prepare_cb callback is invoked. This differs from the basic decoding setup where changes are only passed to the output plugin when a transaction is committed. The start of a prepared transaction is indicated by the begin_prepare_cb callback.

When a prepared transaction is rolled back using the ROLLBACK PREPARED, then the rollback_prepared_cb callback is invoked and when the prepared transaction is committed using COMMIT PREPARED, then the commit_prepared_cb callback is invoked.

Optionally the output plugin can define filtering rules via filter_prepare_cb to decode only specific transaction in two phases. This can be achieved by pattern matching on the gid or via lookups using the xid.

The users that want to decode prepared transactions need to be careful about below mentioned points:

  • If the prepared transaction has locked [user] catalog tables exclusively then decoding prepare can block till the main transaction is committed.

  • The logical replication solution that builds distributed two phase commit using this feature can deadlock if the prepared transaction has locked [user] catalog tables exclusively. To avoid this users must refrain from having locks on catalog tables (e.g. explicit LOCK command) in such transactions. See Section 49.8.2 for the details.

Prev | Up | Next
49.9. Streaming of Large Transactions for Logical Decoding | Home | Chapter 50. Replication Progress Tracking


Part VI. Reference
Prev Up PostgreSQL 14.4 Documentation Home Next

Part VI. Reference

The entries in this Reference are meant to provide in reasonable length an authoritative, complete, and formal summary about their respective subjects. More information about the use of PostgreSQL, in narrative, tutorial, or example form, can be found in other parts of this book. See the cross-references listed on each reference page.

The reference entries are also available as traditional “man” pages.

Table of Contents

I. SQL Commands
ABORT — abort the current transaction
ALTER AGGREGATE — change the definition of an aggregate function
ALTER COLLATION — change the definition of a collation
ALTER CONVERSION — change the definition of a conversion
ALTER DATABASE — change a database
ALTER DEFAULT PRIVILEGES — define default access privileges
ALTER DOMAIN — change the definition of a domain
ALTER EVENT TRIGGER — change the definition of an event trigger
ALTER EXTENSION — change the definition of an extension
ALTER FOREIGN DATA WRAPPER — change the definition of a foreign-data wrapper
ALTER FOREIGN TABLE — change the definition of a foreign table
ALTER FUNCTION — change the definition of a function
ALTER GROUP — change role name or membership
ALTER INDEX — change the definition of an index
ALTER LANGUAGE — change the definition of a procedural language
ALTER LARGE OBJECT — change the definition of a large object
ALTER MATERIALIZED VIEW — change the definition of a materialized view
ALTER OPERATOR — change the definition of an operator
ALTER OPERATOR CLASS — change the definition of an operator class
ALTER OPERATOR FAMILY — change the definition of an operator family
ALTER POLICY — change the definition of a row-level security policy
ALTER PROCEDURE — change the definition of a procedure
ALTER PUBLICATION — change the definition of a publication
ALTER ROLE — change a database role
ALTER ROUTINE — change the definition of a routine
ALTER RULE — change the definition of a rule
ALTER SCHEMA — change the definition of a schema
ALTER SEQUENCE — change the definition of a sequence generator
ALTER SERVER — change the definition of a foreign server
ALTER STATISTICS — change the definition of an extended statistics object
ALTER SUBSCRIPTION — change the definition of a subscription
ALTER SYSTEM — change a server configuration parameter
ALTER TABLE — change the definition of a table
ALTER TABLESPACE — change the definition of a tablespace
ALTER TEXT SEARCH CONFIGURATION — change the definition of a text search configuration
ALTER TEXT SEARCH DICTIONARY — change the definition of a text search dictionary
ALTER TEXT SEARCH PARSER — change the definition of a text search parser
ALTER TEXT SEARCH TEMPLATE — change the definition of a text search template
ALTER TRIGGER — change the definition of a trigger
ALTER TYPE — change the definition of a type
ALTER USER — change a database role
ALTER USER MAPPING — change the definition of a user mapping
ALTER VIEW — change the definition of a view
ANALYZE — collect statistics about a database
BEGIN — start a transaction block
CALL — invoke a procedure
CHECKPOINT — force a write-ahead log checkpoint
CLOSE — close a cursor
CLUSTER — cluster a table according to an index
COMMENT — define or change the comment of an object
COMMIT — commit the current transaction
COMMIT PREPARED — commit a transaction that was earlier prepared for two-phase commit
COPY — copy data between a file and a table
CREATE ACCESS METHOD — define a new access method
CREATE AGGREGATE — define a new aggregate function
CREATE CAST — define a new cast
CREATE COLLATION — define a new collation
CREATE CONVERSION — define a new encoding conversion
CREATE DATABASE — create a new database
CREATE DOMAIN — define a new domain
CREATE EVENT TRIGGER — define a new event trigger
CREATE EXTENSION — install an extension
CREATE FOREIGN DATA WRAPPER — define a new foreign-data wrapper
CREATE FOREIGN TABLE — define a new foreign table
CREATE FUNCTION — define a new function
CREATE GROUP — define a new database role
CREATE INDEX — define a new index
CREATE LANGUAGE — define a new procedural language
CREATE MATERIALIZED VIEW — define a new materialized view
CREATE OPERATOR — define a new operator
CREATE OPERATOR CLASS — define a new operator class
CREATE OPERATOR FAMILY — define a new operator family
CREATE POLICY — define a new row-level security policy for a table
CREATE PROCEDURE — define a new procedure
CREATE PUBLICATION — define a new publication
CREATE ROLE — define a new database role
CREATE RULE — define a new rewrite rule
CREATE SCHEMA — define a new schema
CREATE SEQUENCE — define a new sequence generator
CREATE SERVER — define a new foreign server
CREATE STATISTICS — define extended statistics
CREATE SUBSCRIPTION — define a new subscription
CREATE TABLE — define a new table
CREATE TABLE AS — define a new table from the results of a query
CREATE TABLESPACE — define a new tablespace
CREATE TEXT SEARCH CONFIGURATION — define a new text search configuration
CREATE TEXT SEARCH DICTIONARY — define a new text search dictionary
CREATE TEXT SEARCH PARSER — define a new text search parser
CREATE TEXT SEARCH TEMPLATE — define a new text search template
CREATE TRANSFORM — define a new transform
CREATE TRIGGER — define a new trigger
CREATE TYPE — define a new data type
CREATE USER — define a new database role
CREATE USER MAPPING — define a new mapping of a user to a foreign server
CREATE VIEW — define a new view
DEALLOCATE — deallocate a prepared statement
DECLARE — define a cursor
DELETE — delete rows of a table
DISCARD — discard session state
DO — execute an anonymous code block
DROP ACCESS METHOD — remove an access method
DROP AGGREGATE — remove an aggregate function
DROP CAST — remove a cast
DROP COLLATION — remove a collation
DROP CONVERSION — remove a conversion
DROP DATABASE — remove a database
DROP DOMAIN — remove a domain
DROP EVENT TRIGGER — remove an event trigger
DROP EXTENSION — remove an extension
DROP FOREIGN DATA WRAPPER — remove a foreign-data wrapper
DROP FOREIGN TABLE — remove a foreign table
DROP FUNCTION — remove a function
DROP GROUP — remove a database role
DROP INDEX — remove an index
DROP LANGUAGE — remove a procedural language
DROP MATERIALIZED VIEW — remove a materialized view
DROP OPERATOR — remove an operator
DROP OPERATOR CLASS — remove an operator class
DROP OPERATOR FAMILY — remove an operator family
DROP OWNED — remove database objects owned by a database role
DROP POLICY — remove a row-level security policy from a table
DROP PROCEDURE — remove a procedure
DROP PUBLICATION — remove a publication
DROP ROLE — remove a database role
DROP ROUTINE — remove a routine
DROP RULE — remove a rewrite rule
DROP SCHEMA — remove a schema
DROP SEQUENCE — remove a sequence
DROP SERVER — remove a foreign server descriptor
DROP STATISTICS — remove extended statistics
DROP SUBSCRIPTION — remove a subscription
DROP TABLE — remove a table
DROP TABLESPACE — remove a tablespace
DROP TEXT SEARCH CONFIGURATION — remove a text search configuration
DROP TEXT SEARCH DICTIONARY — remove a text search dictionary
DROP TEXT SEARCH PARSER — remove a text search parser
DROP TEXT SEARCH TEMPLATE — remove a text search template
DROP TRANSFORM — remove a transform
DROP TRIGGER — remove a trigger
DROP TYPE — remove a data type
DROP USER — remove a database role
DROP USER MAPPING — remove a user mapping for a foreign server
DROP VIEW — remove a view
END — commit the current transaction
EXECUTE — execute a prepared statement
EXPLAIN — show the execution plan of a statement
FETCH — retrieve rows from a query using a cursor
GRANT — define access privileges
IMPORT FOREIGN SCHEMA — import table definitions from a foreign server
INSERT — create new rows in a table
LISTEN — listen for a notification
LOAD — load a shared library file
LOCK — lock a table
MOVE — position a cursor
NOTIFY — generate a notification
PREPARE — prepare a statement for execution
PREPARE TRANSACTION — prepare the current transaction for two-phase commit
REASSIGN OWNED — change the ownership of database objects owned by a database role
REFRESH MATERIALIZED VIEW — replace the contents of a materialized view
REINDEX — rebuild indexes
RELEASE SAVEPOINT — destroy a previously defined savepoint
RESET — restore the value of a run-time parameter to the default value
REVOKE — remove access privileges
ROLLBACK — abort the current transaction
ROLLBACK PREPARED — cancel a transaction that was earlier prepared for two-phase commit
ROLLBACK TO SAVEPOINT — roll back to a savepoint
SAVEPOINT — define a new savepoint within the current transaction
SECURITY LABEL — define or change a security label applied to an object
SELECT — retrieve rows from a table or view
SELECT INTO — define a new table from the results of a query
SET — change a run-time parameter
SET CONSTRAINTS — set constraint check timing for the current transaction
SET ROLE — set the current user identifier of the current session
SET SESSION AUTHORIZATION — set the session user identifier and the current user identifier of the current session
SET TRANSACTION — set the characteristics of the current transaction
SHOW — show the value of a run-time parameter
START TRANSACTION — start a transaction block
TRUNCATE — empty a table or set of tables
UNLISTEN — stop listening for a notification
UPDATE — update rows of a table
VACUUM — garbage-collect and optionally analyze a database
VALUES — compute a set of rows
II. PostgreSQL Client Applications
clusterdb — cluster a PostgreSQL database
createdb — create a new PostgreSQL database
createuser — define a new PostgreSQL user account
dropdb — remove a PostgreSQL database
dropuser — remove a PostgreSQL user account
ecpg — embedded SQL C preprocessor
pg_amcheck — checks for corruption in one or more PostgreSQL databases
pg_basebackup — take a base backup of a PostgreSQL cluster
pgbench — run a benchmark test on PostgreSQL
pg_config — retrieve information about the installed version of PostgreSQL
pg_dump — extract a PostgreSQL database into a script file or other archive file
pg_dumpall — extract a PostgreSQL database cluster into a script file
pg_isready — check the connection status of a PostgreSQL server
pg_receivewal — stream write-ahead logs from a PostgreSQL server
pg_recvlogical — control PostgreSQL logical decoding streams
pg_restore — restore a PostgreSQL database from an archive file created by pg_dump
pg_verifybackup — verify the integrity of a base backup of a PostgreSQL cluster
psql — PostgreSQL interactive terminal
reindexdb — reindex a PostgreSQL database
vacuumdb — garbage-collect and analyze a PostgreSQL database
III. PostgreSQL Server Applications
initdb — create a new PostgreSQL database cluster
pg_archivecleanup — clean up PostgreSQL WAL archive files
pg_checksums — enable, disable or check data checksums in a PostgreSQL database cluster
pg_controldata — display control information of a PostgreSQL database cluster
pg_ctl — initialize, start, stop, or control a PostgreSQL server
pg_resetwal — reset the write-ahead log and other control information of a PostgreSQL database cluster
pg_rewind — synchronize a PostgreSQL data directory with another data directory that was forked from it
pg_test_fsync — determine fastest wal_sync_method for PostgreSQL
pg_test_timing — measure timing overhead
pg_upgrade — upgrade a PostgreSQL server instance
pg_waldump — display a human-readable rendering of the write-ahead log of a PostgreSQL database cluster
postgres — PostgreSQL database server
postmaster — PostgreSQL database server
Prev | Up | Next
49.10. Two-phase Commit Support for Logical Decoding | Home | SQL Commands


PostgreSQL Client Applications
Prev Up Part VI. Reference Home Next

PostgreSQL Client Applications

This part contains reference information for PostgreSQL client applications and utilities. Not all of these commands are of general utility; some might require special privileges. The common feature of these applications is that they can be run on any host, independent of where the database server resides.

When specified on the command line, user and database names have their case preserved — the presence of spaces or special characters might require quoting. Table names and other identifiers do not have their case preserved, except where documented, and might require quoting.

Table of Contents

clusterdb — cluster a PostgreSQL database
createdb — create a new PostgreSQL database
createuser — define a new PostgreSQL user account
dropdb — remove a PostgreSQL database
dropuser — remove a PostgreSQL user account
ecpg — embedded SQL C preprocessor
pg_amcheck — checks for corruption in one or more PostgreSQL databases
pg_basebackup — take a base backup of a PostgreSQL cluster
pgbench — run a benchmark test on PostgreSQL
pg_config — retrieve information about the installed version of PostgreSQL
pg_dump — extract a PostgreSQL database into a script file or other archive file
pg_dumpall — extract a PostgreSQL database cluster into a script file
pg_isready — check the connection status of a PostgreSQL server
pg_receivewal — stream write-ahead logs from a PostgreSQL server
pg_recvlogical — control PostgreSQL logical decoding streams
pg_restore — restore a PostgreSQL database from an archive file created by pg_dump
pg_verifybackup — verify the integrity of a base backup of a PostgreSQL cluster
psql — PostgreSQL interactive terminal
reindexdb — reindex a PostgreSQL database
vacuumdb — garbage-collect and analyze a PostgreSQL database
Prev | Up | Next
VALUES | Home | clusterdb


clusterdb
Prev Up PostgreSQL Client Applications Home Next

clusterdb

clusterdb — cluster a PostgreSQL database

Synopsis

clusterdb [connection-option...] [ --verbose | -v ] [ --table | -t table ] ... [dbname]

clusterdb [connection-option...] [ --verbose | -v ] --all | -a

Description

clusterdb is a utility for reclustering tables in a PostgreSQL database. It finds tables that have previously been clustered, and clusters them again on the same index that was last used. Tables that have never been clustered are not affected.

clusterdb is a wrapper around the SQL command CLUSTER. There is no effective difference between clustering databases via this utility and via other methods for accessing the server.

Options

clusterdb accepts the following command-line arguments:

-a
--all

Cluster all databases.

[-d] dbname
[--dbname=]dbname

Specifies the name of the database to be clustered, when -a/--all is not used. If this is not specified, the database name is read from the environment variable PGDATABASE. If that is not set, the user name specified for the connection is used. The dbname can be a connection string. If so, connection string parameters will override any conflicting command line options.

-e
--echo

Echo the commands that clusterdb generates and sends to the server.

-q
--quiet

Do not display progress messages.

-t table
--table=table

Cluster table only. Multiple tables can be clustered by writing multiple -t switches.

-v
--verbose

Print detailed information during processing.

-V
--version

Print the clusterdb version and exit.

-?
--help

Show help about clusterdb command line arguments, and exit.

clusterdb also accepts the following command-line arguments for connection parameters:

-h host
--host=host

Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for the Unix domain socket.

-p port
--port=port

Specifies the TCP port or local Unix domain socket file extension on which the server is listening for connections.

-U username
--username=username

User name to connect as.

-w
--no-password

Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.

-W
--password

Force clusterdb to prompt for a password before connecting to a database.

This option is never essential, since clusterdb will automatically prompt for a password if the server demands password authentication. However, clusterdb will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt.

--maintenance-db=dbname

Specifies the name of the database to connect to to discover which databases should be clustered, when -a/--all is used. If not specified, the postgres database will be used, or if that does not exist, template1 will be used. This can be a connection string. If so, connection string parameters will override any conflicting command line options. Also, connection string parameters other than the database name itself will be re-used when connecting to other databases.

Environment

PGDATABASE
PGHOST
PGPORT
PGUSER

Default connection parameters

PG_COLOR

Specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

This utility, like most other PostgreSQL utilities, also uses the environment variables supported by libpq (see Section 34.15).

Diagnostics

In case of difficulty, see CLUSTER and psql for discussions of potential problems and error messages. The database server must be running at the targeted host. Also, any default connection settings and environment variables used by the libpq front-end library will apply.

Examples

To cluster the database test:

$ clusterdb test

To cluster a single table foo in a database named xyzzy:

$ clusterdb --table=foo xyzzy

See Also

CLUSTER
Prev | Up | Next
PostgreSQL Client Applications | Home | createdb


createdb
Prev Up PostgreSQL Client Applications Home Next

createdb

createdb — create a new PostgreSQL database

Synopsis

createdb [connection-option...] [option...] [dbname [description]]

Description

createdb creates a new PostgreSQL database.

Normally, the database user who executes this command becomes the owner of the new database. However, a different owner can be specified via the -O option, if the executing user has appropriate privileges.

createdb is a wrapper around the SQL command CREATE DATABASE. There is no effective difference between creating databases via this utility and via other methods for accessing the server.

Options

createdb accepts the following command-line arguments:

dbname

Specifies the name of the database to be created. The name must be unique among all PostgreSQL databases in this cluster. The default is to create a database with the same name as the current system user.

description

Specifies a comment to be associated with the newly created database.

-D tablespace
--tablespace=tablespace

Specifies the default tablespace for the database. (This name is processed as a double-quoted identifier.)

-e
--echo

Echo the commands that createdb generates and sends to the server.

-E encoding
--encoding=encoding

Specifies the character encoding scheme to be used in this database. The character sets supported by the PostgreSQL server are described in Section 24.3.1.

-l locale
--locale=locale

Specifies the locale to be used in this database. This is equivalent to specifying both --lc-collate and --lc-ctype.

--lc-collate=locale

Specifies the LC_COLLATE setting to be used in this database.

--lc-ctype=locale

Specifies the LC_CTYPE setting to be used in this database.

-O owner
--owner=owner

Specifies the database user who will own the new database. (This name is processed as a double-quoted identifier.)

-T template
--template=template

Specifies the template database from which to build this database. (This name is processed as a double-quoted identifier.)

-V
--version

Print the createdb version and exit.

-?
--help

Show help about createdb command line arguments, and exit.

The options -D, -l, -E, -O, and -T correspond to options of the underlying SQL command CREATE DATABASE; see there for more information about them.

createdb also accepts the following command-line arguments for connection parameters:

-h host
--host=host

Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for the Unix domain socket.

-p port
--port=port

Specifies the TCP port or the local Unix domain socket file extension on which the server is listening for connections.

-U username
--username=username

User name to connect as.

-w
--no-password

Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.

-W
--password

Force createdb to prompt for a password before connecting to a database.

This option is never essential, since createdb will automatically prompt for a password if the server demands password authentication. However, createdb will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt.

--maintenance-db=dbname

Specifies the name of the database to connect to when creating the new database. If not specified, the postgres database will be used; if that does not exist (or if it is the name of the new database being created), template1 will be used. This can be a connection string. If so, connection string parameters will override any conflicting command line options.

Environment

PGDATABASE

If set, the name of the database to create, unless overridden on the command line.

PGHOST
PGPORT
PGUSER

Default connection parameters. PGUSER also determines the name of the database to create, if it is not specified on the command line or by PGDATABASE.

PG_COLOR

Specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

This utility, like most other PostgreSQL utilities, also uses the environment variables supported by libpq (see Section 34.15).

Diagnostics

In case of difficulty, see CREATE DATABASE and psql for discussions of potential problems and error messages. The database server must be running at the targeted host. Also, any default connection settings and environment variables used by the libpq front-end library will apply.

Examples

To create the database demo using the default database server:

$ createdb demo

To create the database demo using the server on host eden, port 5000, using the template0 template database, here is the command-line command and the underlying SQL command:

$ createdb -p 5000 -h eden -T template0 -e demo
CREATE DATABASE demo TEMPLATE template0;

See Also

dropdb, CREATE DATABASE
Prev | Up | Next
clusterdb | Home | createuser


createuser
Prev Up PostgreSQL Client Applications Home Next

createuser

createuser — define a new PostgreSQL user account

Synopsis

createuser [connection-option...] [option...] [username]

Description

createuser creates a new PostgreSQL user (or more precisely, a role). Only superusers and users with CREATEROLE privilege can create new users, so createuser must be invoked by someone who can connect as a superuser or a user with CREATEROLE privilege.

If you wish to create a new superuser, you must connect as a superuser, not merely with CREATEROLE privilege. Being a superuser implies the ability to bypass all access permission checks within the database, so superuser access should not be granted lightly.

createuser is a wrapper around the SQL command CREATE ROLE. There is no effective difference between creating users via this utility and via other methods for accessing the server.

Options

createuser accepts the following command-line arguments:

username

Specifies the name of the PostgreSQL user to be created. This name must be different from all existing roles in this PostgreSQL installation.

-c number
--connection-limit=number

Set a maximum number of connections for the new user. The default is to set no limit.

-d
--createdb

The new user will be allowed to create databases.

-D
--no-createdb

The new user will not be allowed to create databases. This is the default.

-e
--echo

Echo the commands that createuser generates and sends to the server.

-E
--encrypted

This option is obsolete but still accepted for backward compatibility.

-g role
--role=role

Indicates role to which this role will be added immediately as a new member. Multiple roles to which this role will be added as a member can be specified by writing multiple -g switches.

-i
--inherit

The new role will automatically inherit privileges of roles it is a member of. This is the default.

-I
--no-inherit

The new role will not automatically inherit privileges of roles it is a member of.

--interactive

Prompt for the user name if none is specified on the command line, and also prompt for whichever of the options -d/-D, -r/-R, -s/-S is not specified on the command line. (This was the default behavior up to PostgreSQL 9.1.)

-l
--login

The new user will be allowed to log in (that is, the user name can be used as the initial session user identifier). This is the default.

-L
--no-login

The new user will not be allowed to log in. (A role without login privilege is still useful as a means of managing database permissions.)

-P
--pwprompt

If given, createuser will issue a prompt for the password of the new user. This is not necessary if you do not plan on using password authentication.

-r
--createrole

The new user will be allowed to create new roles (that is, this user will have CREATEROLE privilege).

-R
--no-createrole

The new user will not be allowed to create new roles. This is the default.

-s
--superuser

The new user will be a superuser.

-S
--no-superuser

The new user will not be a superuser. This is the default.

-V
--version

Print the createuser version and exit.

--replication

The new user will have the REPLICATION privilege, which is described more fully in the documentation for CREATE ROLE.

--no-replication

The new user will not have the REPLICATION privilege, which is described more fully in the documentation for CREATE ROLE.

-?
--help

Show help about createuser command line arguments, and exit.

createuser also accepts the following command-line arguments for connection parameters:

-h host
--host=host

Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for the Unix domain socket.

-p port
--port=port

Specifies the TCP port or local Unix domain socket file extension on which the server is listening for connections.

-U username
--username=username

User name to connect as (not the user name to create).

-w
--no-password

Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.

-W
--password

Force createuser to prompt for a password (for connecting to the server, not for the password of the new user).

This option is never essential, since createuser will automatically prompt for a password if the server demands password authentication. However, createuser will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt.

Environment

PGHOST
PGPORT
PGUSER

Default connection parameters

PG_COLOR

Specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

This utility, like most other PostgreSQL utilities, also uses the environment variables supported by libpq (see Section 34.15).

Diagnostics

In case of difficulty, see CREATE ROLE and psql for discussions of potential problems and error messages. The database server must be running at the targeted host. Also, any default connection settings and environment variables used by the libpq front-end library will apply.

Examples

To create a user joe on the default database server:

$ createuser joe

To create a user joe on the default database server with prompting for some additional attributes:

$ createuser --interactive joe
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n

To create the same user joe using the server on host eden, port 5000, with attributes explicitly specified, taking a look at the underlying command:

$ createuser -h eden -p 5000 -S -D -R -e joe
CREATE ROLE joe NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT LOGIN;

To create the user joe as a superuser, and assign a password immediately:

$ createuser -P -s -e joe
Enter password for new role: xyzzy
Enter it again: xyzzy
CREATE ROLE joe PASSWORD 'md5b5f5ba1a423792b526f799ae4eb3d59e' SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN;

In the above example, the new password isn't actually echoed when typed, but we show what was typed for clarity. As you see, the password is encrypted before it is sent to the client.

See Also

dropuser, CREATE ROLE
Prev | Up | Next
createdb | Home | dropdb


dropdb
Prev Up PostgreSQL Client Applications Home Next

dropdb

dropdb — remove a PostgreSQL database

Synopsis

dropdb [connection-option...] [option...] dbname

Description

dropdb destroys an existing PostgreSQL database. The user who executes this command must be a database superuser or the owner of the database.

dropdb is a wrapper around the SQL command DROP DATABASE. There is no effective difference between dropping databases via this utility and via other methods for accessing the server.

Options

dropdb accepts the following command-line arguments:

dbname

Specifies the name of the database to be removed.

-e
--echo

Echo the commands that dropdb generates and sends to the server.

-f
--force

Attempt to terminate all existing connections to the target database before dropping it. See DROP DATABASE for more information on this option.

-i
--interactive

Issues a verification prompt before doing anything destructive.

-V
--version

Print the dropdb version and exit.

--if-exists

Do not throw an error if the database does not exist. A notice is issued in this case.

-?
--help

Show help about dropdb command line arguments, and exit.

dropdb also accepts the following command-line arguments for connection parameters:

-h host
--host=host

Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for the Unix domain socket.

-p port
--port=port

Specifies the TCP port or local Unix domain socket file extension on which the server is listening for connections.

-U username
--username=username

User name to connect as.

-w
--no-password

Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.

-W
--password

Force dropdb to prompt for a password before connecting to a database.

This option is never essential, since dropdb will automatically prompt for a password if the server demands password authentication. However, dropdb will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt.

--maintenance-db=dbname

Specifies the name of the database to connect to in order to drop the target database. If not specified, the postgres database will be used; if that does not exist (or is the database being dropped), template1 will be used. This can be a connection string. If so, connection string parameters will override any conflicting command line options.

Environment

PGHOST
PGPORT
PGUSER

Default connection parameters

PG_COLOR

Specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

This utility, like most other PostgreSQL utilities, also uses the environment variables supported by libpq (see Section 34.15).

Diagnostics

In case of difficulty, see DROP DATABASE and psql for discussions of potential problems and error messages. The database server must be running at the targeted host. Also, any default connection settings and environment variables used by the libpq front-end library will apply.

Examples

To destroy the database demo on the default database server:

$ dropdb demo

To destroy the database demo using the server on host eden, port 5000, with verification and a peek at the underlying command:

$ dropdb -p 5000 -h eden -i -e demo
Database "demo" will be permanently deleted.
Are you sure? (y/n) y
DROP DATABASE demo;

See Also

createdb, DROP DATABASE
Prev | Up | Next
createuser | Home | dropuser


dropuser
Prev Up PostgreSQL Client Applications Home Next

dropuser

dropuser — remove a PostgreSQL user account

Synopsis

dropuser [connection-option...] [option...] [username]

Description

dropuser removes an existing PostgreSQL user. Only superusers and users with the CREATEROLE privilege can remove PostgreSQL users. (To remove a superuser, you must yourself be a superuser.)

dropuser is a wrapper around the SQL command DROP ROLE. There is no effective difference between dropping users via this utility and via other methods for accessing the server.

Options

dropuser accepts the following command-line arguments:

username

Specifies the name of the PostgreSQL user to be removed. You will be prompted for a name if none is specified on the command line and the -i/--interactive option is used.

-e
--echo

Echo the commands that dropuser generates and sends to the server.

-i
--interactive

Prompt for confirmation before actually removing the user, and prompt for the user name if none is specified on the command line.

-V
--version

Print the dropuser version and exit.

--if-exists

Do not throw an error if the user does not exist. A notice is issued in this case.

-?
--help

Show help about dropuser command line arguments, and exit.

dropuser also accepts the following command-line arguments for connection parameters:

-h host
--host=host

Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for the Unix domain socket.

-p port
--port=port

Specifies the TCP port or local Unix domain socket file extension on which the server is listening for connections.

-U username
--username=username

User name to connect as (not the user name to drop).

-w
--no-password

Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.

-W
--password

Force dropuser to prompt for a password before connecting to a database.

This option is never essential, since dropuser will automatically prompt for a password if the server demands password authentication. However, dropuser will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt.

Environment

PGHOST
PGPORT
PGUSER

Default connection parameters

PG_COLOR

Specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

This utility, like most other PostgreSQL utilities, also uses the environment variables supported by libpq (see Section 34.15).

Diagnostics

In case of difficulty, see DROP ROLE and psql for discussions of potential problems and error messages. The database server must be running at the targeted host. Also, any default connection settings and environment variables used by the libpq front-end library will apply.

Examples

To remove user joe from the default database server:

$ dropuser joe

To remove user joe using the server on host eden, port 5000, with verification and a peek at the underlying command:

$ dropuser -p 5000 -h eden -i -e joe
Role "joe" will be permanently removed.
Are you sure? (y/n) y
DROP ROLE joe;

See Also

createuser, DROP ROLE
Prev | Up | Next
dropdb | Home | ecpg


ecpg
Prev Up PostgreSQL Client Applications Home Next

ecpg

ecpg — embedded SQL C preprocessor

Synopsis

ecpg [option...] file...

Description

ecpg is the embedded SQL preprocessor for C programs. It converts C programs with embedded SQL statements to normal C code by replacing the SQL invocations with special function calls. The output files can then be processed with any C compiler tool chain.

ecpg will convert each input file given on the command line to the corresponding C output file. If an input file name does not have any extension, .pgc is assumed. The file's extension will be replaced by .c to construct the output file name. But the output file name can be overridden using the -o option.

If an input file name is just -, ecpg reads the program from standard input (and writes to standard output, unless that is overridden with -o).

This reference page does not describe the embedded SQL language. See Chapter 36 for more information on that topic.

Options

ecpg accepts the following command-line arguments:

-c

Automatically generate certain C code from SQL code. Currently, this works for EXEC SQL TYPE.

-C mode

Set a compatibility mode. mode can be INFORMIX, INFORMIX_SE, or ORACLE.

-D symbol

Define a C preprocessor symbol.

-h

Process header files. When this option is specified, the output file extension becomes .h not .c, and the default input file extension is .pgh not .pgc. Also, the -c option is forced on.

-i

Parse system include files as well.

-I directory

Specify an additional include path, used to find files included via EXEC SQL INCLUDE. Defaults are . (current directory), /usr/local/include, the PostgreSQL include directory which is defined at compile time (default: /usr/local/pgsql/include), and /usr/include, in that order.

-o filename

Specifies that ecpg should write all its output to the given filename. Write -o - to send all output to standard output.

-r option

Selects run-time behavior. Option can be one of the following:

no_indicator

Do not use indicators but instead use special values to represent null values. Historically there have been databases using this approach.

prepare

Prepare all statements before using them. Libecpg will keep a cache of prepared statements and reuse a statement if it gets executed again. If the cache runs full, libecpg will free the least used statement.

questionmarks

Allow question mark as placeholder for compatibility reasons. This used to be the default long ago.

-t

Turn on autocommit of transactions. In this mode, each SQL command is automatically committed unless it is inside an explicit transaction block. In the default mode, commands are committed only when EXEC SQL COMMIT is issued.

-v

Print additional information including the version and the "include" path.

--version

Print the ecpg version and exit.

-?
--help

Show help about ecpg command line arguments, and exit.

Notes

When compiling the preprocessed C code files, the compiler needs to be able to find the ECPG header files in the PostgreSQL include directory. Therefore, you might have to use the -I option when invoking the compiler (e.g., -I/usr/local/pgsql/include).

Programs using C code with embedded SQL have to be linked against the libecpg library, for example using the linker options -L/usr/local/pgsql/lib -lecpg.

The value of either of these directories that is appropriate for the installation can be found out using pg_config.

Examples

If you have an embedded SQL C source file named prog1.pgc, you can create an executable program using the following sequence of commands:

ecpg prog1.pgc
cc -I/usr/local/pgsql/include -c prog1.c
cc -o prog1 prog1.o -L/usr/local/pgsql/lib -lecpg

Prev | Up | Next
dropuser | Home | pg_amcheck


pg_amcheck
Prev Up PostgreSQL Client Applications Home Next

pg_amcheck

pg_amcheck — checks for corruption in one or more PostgreSQL databases

Synopsis

pg_amcheck [option...] [dbname]

Description

pg_amcheck supports running amcheck's corruption checking functions against one or more databases, with options to select which schemas, tables and indexes to check, which kinds of checking to perform, and whether to perform the checks in parallel, and if so, the number of parallel connections to establish and use.

Only table relations and btree indexes are currently supported. Other relation types are silently skipped.

If dbname is specified, it should be the name of a single database to check, and no other database selection options should be present. Otherwise, if any database selection options are present, all matching databases will be checked. If no such options are present, the default database will be checked. Database selection options include --all, --database and --exclude-database. They also include --relation, --exclude-relation, --table, --exclude-table, --index, and --exclude-index, but only when such options are used with a three-part pattern (e.g. mydb*.myschema*.myrel*). Finally, they include --schema and --exclude-schema when such options are used with a two-part pattern (e.g. mydb*.myschema*).

dbname can also be a connection string.

Options

The following command-line options control what is checked:

-a
--all

Check all databases, except for any excluded via --exclude-database.

-d pattern
--database=pattern

Check databases matching the specified pattern, except for any excluded by --exclude-database. This option can be specified more than once.

-D pattern
--exclude-database=pattern

Exclude databases matching the given pattern. This option can be specified more than once.

-i pattern
--index=pattern

Check indexes matching the specified pattern, unless they are otherwise excluded. This option can be specified more than once.

This is similar to the --relation option, except that it applies only to indexes, not tables.

-I pattern
--exclude-index=pattern

Exclude indexes matching the specified pattern. This option can be specified more than once.

This is similar to the --exclude-relation option, except that it applies only to indexes, not tables.

-r pattern
--relation=pattern

Check relations matching the specified pattern, unless they are otherwise excluded. This option can be specified more than once.

Patterns may be unqualified, e.g. myrel*, or they may be schema-qualified, e.g. myschema*.myrel* or database-qualified and schema-qualified, e.g. mydb*.myscheam*.myrel*. A database-qualified pattern will add matching databases to the list of databases to be checked.

-R pattern
--exclude-relation=pattern

Exclude relations matching the specified pattern. This option can be specified more than once.

As with --relation, the pattern may be unqualified, schema-qualified, or database- and schema-qualified.

-s pattern
--schema=pattern

Check tables and indexes in schemas matching the specified pattern, unless they are otherwise excluded. This option can be specified more than once.

To select only tables in schemas matching a particular pattern, consider using something like --table=SCHEMAPAT.* --no-dependent-indexes. To select only indexes, consider using something like --index=SCHEMAPAT.*.

A schema pattern may be database-qualified. For example, you may write --schema=mydb*.myschema* to select schemas matching myschema* in databases matching mydb*.

-S pattern
--exclude-schema=pattern

Exclude tables and indexes in schemas matching the specified pattern. This option can be specified more than once.

As with --schema, the pattern may be database-qualified.

-t pattern
--table=pattern

Check tables matching the specified pattern, unless they are otherwise excluded. This option can be specified more than once.

This is similar to the --relation option, except that it applies only to tables, not indexes.

-T pattern
--exclude-table=pattern

Exclude tables matching the specified pattern. This option can be specified more than once.

This is similar to the --exclude-relation option, except that it applies only to tables, not indexes.

--no-dependent-indexes

By default, if a table is checked, any btree indexes of that table will also be checked, even if they are not explicitly selected by an option such as --index or --relation. This option suppresses that behavior.

--no-dependent-toast

By default, if a table is checked, its toast table, if any, will also be checked, even if it is not explicitly selected by an option such as --table or --relation. This option suppresses that behavior.

--no-strict-names

By default, if an argument to --database, --table, --index, or --relation matches no objects, it is a fatal error. This option downgrades that error to a warning.

The following command-line options control checking of tables:

--exclude-toast-pointers

By default, whenever a toast pointer is encountered in a table, a lookup is performed to ensure that it references apparently-valid entries in the toast table. These checks can be quite slow, and this option can be used to skip them.

--on-error-stop

After reporting all corruptions on the first page of a table where corruption is found, stop processing that table relation and move on to the next table or index.

Note that index checking always stops after the first corrupt page. This option only has meaning relative to table relations.

--skip=option

If all-frozen is given, table corruption checks will skip over pages in all tables that are marked as all frozen.

If all-visible is given, table corruption checks will skip over pages in all tables that are marked as all visible.

By default, no pages are skipped. This can be specified as none, but since this is the default, it need not be mentioned.

--startblock=block

Start checking at the specified block number. An error will occur if the table relation being checked has fewer than this number of blocks. This option does not apply to indexes, and is probably only useful when checking a single table relation. See --endblock for further caveats.

--endblock=block

End checking at the specified block number. An error will occur if the table relation being checked has fewer than this number of blocks. This option does not apply to indexes, and is probably only useful when checking a single table relation. If both a regular table and a toast table are checked, this option will apply to both, but higher-numbered toast blocks may still be accessed while validating toast pointers, unless that is suppressed using --exclude-toast-pointers.

The following command-line options control checking of B-tree indexes:

--heapallindexed

For each index checked, verify the presence of all heap tuples as index tuples in the index using amcheck's heapallindexed option.

--parent-check

For each btree index checked, use amcheck's bt_index_parent_check function, which performs additional checks of parent/child relationships during index checking.

The default is to use amcheck's bt_index_check function, but note that use of the --rootdescend option implicitly selects bt_index_parent_check.

--rootdescend

For each index checked, re-find tuples on the leaf level by performing a new search from the root page for each tuple using amcheck's rootdescend option.

Use of this option implicitly also selects the --parent-check option.

This form of verification was originally written to help in the development of btree index features. It may be of limited use or even of no use in helping detect the kinds of corruption that occur in practice. It may also cause corruption checking to take considerably longer and consume considerably more resources on the server.

Warning

The extra checks performed against B-tree indexes when the --parent-check option or the --rootdescend option is specified require relatively strong relation-level locks. These checks are the only checks that will block concurrent data modification from INSERT, UPDATE, and DELETE commands.

The following command-line options control the connection to the server:

-h hostname
--host=hostname

Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for the Unix domain socket.

-p port
--port=port

Specifies the TCP port or local Unix domain socket file extension on which the server is listening for connections.

-U
--username=username

User name to connect as.

-w
--no-password

Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.

-W
--password

Force pg_amcheck to prompt for a password before connecting to a database.

This option is never essential, since pg_amcheck will automatically prompt for a password if the server demands password authentication. However, pg_amcheck will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt.

--maintenance-db=dbname

Specifies a database or connection string to be used to discover the list of databases to be checked. If neither --all nor any option including a database pattern is used, no such connection is required and this option does nothing. Otherwise, any connection string parameters other than the database name which are included in the value for this option will also be used when connecting to the databases being checked. If this option is omitted, the default is postgres or, if that fails, template1.

Other options are also available:

-e
--echo

Echo to stdout all SQL sent to the server.

-j num
--jobs=num

Use num concurrent connections to the server, or one per object to be checked, whichever is less.

The default is to use a single connection.

-P
--progress

Show progress information. Progress information includes the number of relations for which checking has been completed, and the total size of those relations. It also includes the total number of relations that will eventually be checked, and the estimated size of those relations.

-v
--verbose

Print more messages. In particular, this will print a message for each relation being checked, and will increase the level of detail shown for server errors.

-V
--version

Print the pg_amcheck version and exit.

--install-missing
--install-missing=schema

Install any missing extensions that are required to check the database(s). If not yet installed, each extension's objects will be installed into the given schema, or if not specified into schema pg_catalog.

At present, the only required extension is amcheck.

-?
--help

Show help about pg_amcheck command line arguments, and exit.

Notes

pg_amcheck is designed to work with PostgreSQL 14.0 and later.

See Also

amcheck
Prev | Up | Next
ecpg | Home | pg_basebackup


pg_basebackup
Prev Up PostgreSQL Client Applications Home Next

pg_basebackup

pg_basebackup — take a base backup of a PostgreSQL cluster

Synopsis

pg_basebackup [option...]

Description

pg_basebackup is used to take a base backup of a running PostgreSQL database cluster. The backup is taken without affecting other clients of the database, and can be used both for point-in-time recovery (see Section 26.3) and as the starting point for a log-shipping or streaming-replication standby server (see Section 27.2).

pg_basebackup makes an exact copy of the database cluster's files, while making sure the server is put into and out of backup mode automatically. Backups are always taken of the entire database cluster; it is not possible to back up individual databases or database objects. For selective backups, another tool such as pg_dump must be used.

The backup is made over a regular PostgreSQL connection that uses the replication protocol. The connection must be made with a user ID that has REPLICATION permissions (see Section 22.2) or is a superuser, and pg_hba.conf must permit the replication connection. The server must also be configured with max_wal_senders set high enough to provide at least one walsender for the backup plus one for WAL streaming (if used).

There can be multiple pg_basebackups running at the same time, but it is usually better from a performance point of view to take only one backup, and copy the result.

pg_basebackup can make a base backup from not only a primary server but also a standby. To take a backup from a standby, set up the standby so that it can accept replication connections (that is, set max_wal_senders and hot_standby, and configure its pg_hba.conf appropriately). You will also need to enable full_page_writes on the primary.

Note that there are some limitations in taking a backup from a standby:

  • The backup history file is not created in the database cluster backed up.

  • pg_basebackup cannot force the standby to switch to a new WAL file at the end of backup. When you are using -X none, if write activity on the primary is low, pg_basebackup may need to wait a long time for the last WAL file required for the backup to be switched and archived. In this case, it may be useful to run pg_switch_wal on the primary in order to trigger an immediate WAL file switch.

  • If the standby is promoted to be primary during backup, the backup fails.

  • All WAL records required for the backup must contain sufficient full-page writes, which requires you to enable full_page_writes on the primary and not to use a tool like pg_compresslog as archive_command to remove full-page writes from WAL files.

Whenever pg_basebackup is taking a base backup, the server's pg_stat_progress_basebackup view will report the progress of the backup. See Section 28.4.5 for details.

Options

The following command-line options control the location and format of the output:

-D directory
--pgdata=directory

Sets the target directory to write the output to. pg_basebackup will create this directory (and any missing parent directories) if it does not exist. If it already exists, it must be empty.

When the backup is in tar format, the target directory may be specified as - (dash), causing the tar file to be written to stdout.

This option is required.

-F format
--format=format

Selects the format for the output. format can be one of the following:

p
plain

Write the output as plain files, with the same layout as the source server's data directory and tablespaces. When the cluster has no additional tablespaces, the whole database will be placed in the target directory. If the cluster contains additional tablespaces, the main data directory will be placed in the target directory, but all other tablespaces will be placed in the same absolute path as they have on the source server. (See --tablespace-mapping to change that.)

This is the default format.

t
tar

Write the output as tar files in the target directory. The main data directory's contents will be written to a file named base.tar, and each other tablespace will be written to a separate tar file named after that tablespace's OID.

If the target directory is specified as - (dash), the tar contents will be written to standard output, suitable for piping to (for example) gzip. This is only allowed if the cluster has no additional tablespaces and WAL streaming is not used.

-R
--write-recovery-conf

Creates a standby.signal file and appends connection settings to the postgresql.auto.conf file in the target directory (or within the base archive file when using tar format). This eases setting up a standby server using the results of the backup.

The postgresql.auto.conf file will record the connection settings and, if specified, the replication slot that pg_basebackup is using, so that streaming replication will use the same settings later on.

-T olddir=newdir
--tablespace-mapping=olddir=newdir

Relocates the tablespace in directory olddir to newdir during the backup. To be effective, olddir must exactly match the path specification of the tablespace as it is defined on the source server. (But it is not an error if there is no tablespace in olddir on the source server.) Meanwhile newdir is a directory in the receiving host's filesystem. As with the main target directory, newdir need not exist already, but if it does exist it must be empty. Both olddir and newdir must be absolute paths. If either path needs to contain an equal sign (=), precede that with a backslash. This option can be specified multiple times for multiple tablespaces.

If a tablespace is relocated in this way, the symbolic links inside the main data directory are updated to point to the new location. So the new data directory is ready to be used for a new server instance with all tablespaces in the updated locations.

Currently, this option only works with plain output format; it is ignored if tar format is selected.

--waldir=waldir

Sets the directory to write WAL (write-ahead log) files to. By default WAL files will be placed in the pg_wal subdirectory of the target directory, but this option can be used to place them elsewhere. waldir must be an absolute path. As with the main target directory, waldir need not exist already, but if it does exist it must be empty. This option can only be specified when the backup is in plain format.

-X method
--wal-method=method

Includes the required WAL (write-ahead log) files in the backup. This will include all write-ahead logs generated during the backup. Unless the method none is specified, it is possible to start a postmaster in the target directory without the need to consult the log archive, thus making the output a completely standalone backup.

The following methods for collecting the write-ahead logs are supported:

n
none

Don't include write-ahead logs in the backup.

f
fetch

The write-ahead log files are collected at the end of the backup. Therefore, it is necessary for the source server's wal_keep_size parameter to be set high enough that the required log data is not removed before the end of the backup. If the required log data has been recycled before it's time to transfer it, the backup will fail and be unusable.

When tar format is used, the write-ahead log files will be included in the base.tar file.

s
stream

Stream write-ahead log data while the backup is being taken. This method will open a second connection to the server and start streaming the write-ahead log in parallel while running the backup. Therefore, it will require two replication connections not just one. As long as the client can keep up with the write-ahead log data, using this method requires no extra write-ahead logs to be saved on the source server.

When tar format is used, the write-ahead log files will be written to a separate file named pg_wal.tar (if the server is a version earlier than 10, the file will be named pg_xlog.tar).

This value is the default.

-z
--gzip

Enables gzip compression of tar file output, with the default compression level. Compression is only available when using the tar format, and the suffix .gz will automatically be added to all tar filenames.

-Z level
--compress=level

Enables gzip compression of tar file output, and specifies the compression level (0 through 9, 0 being no compression and 9 being best compression). Compression is only available when using the tar format, and the suffix .gz will automatically be added to all tar filenames.

The following command-line options control the generation of the backup and the invocation of the program:

-c fast|spread
--checkpoint=fast|spread

Sets checkpoint mode to fast (immediate) or spread (the default) (see Section 26.3.3).

-C
--create-slot

Specifies that the replication slot named by the --slot option should be created before starting the backup. An error is raised if the slot already exists.

-l label
--label=label

Sets the label for the backup. If none is specified, a default value of “pg_basebackup base backup” will be used.

-n
--no-clean

By default, when pg_basebackup aborts with an error, it removes any directories it might have created before discovering that it cannot finish the job (for example, the target directory and write-ahead log directory). This option inhibits tidying-up and is thus useful for debugging.

Note that tablespace directories are not cleaned up either way.

-N
--no-sync

By default, pg_basebackup will wait for all files to be written safely to disk. This option causes pg_basebackup to return without waiting, which is faster, but means that a subsequent operating system crash can leave the base backup corrupt. Generally, this option is useful for testing but should not be used when creating a production installation.

-P
--progress

Enables progress reporting. Turning this on will deliver an approximate progress report during the backup. Since the database may change during the backup, this is only an approximation and may not end at exactly 100%. In particular, when WAL log is included in the backup, the total amount of data cannot be estimated in advance, and in this case the estimated target size will increase once it passes the total estimate without WAL.

-r rate
--max-rate=rate

Sets the maximum transfer rate at which data is collected from the source server. This can be useful to limit the impact of pg_basebackup on the server. Values are in kilobytes per second. Use a suffix of M to indicate megabytes per second. A suffix of k is also accepted, and has no effect. Valid values are between 32 kilobytes per second and 1024 megabytes per second.

This option always affects transfer of the data directory. Transfer of WAL files is only affected if the collection method is fetch.

-S slotname
--slot=slotname

This option can only be used together with -X stream. It causes WAL streaming to use the specified replication slot. If the base backup is intended to be used as a streaming-replication standby using a replication slot, the standby should then use the same replication slot name as primary_slot_name. This ensures that the primary server does not remove any necessary WAL data in the time between the end of the base backup and the start of streaming replication on the new standby.

The specified replication slot has to exist unless the option -C is also used.

If this option is not specified and the server supports temporary replication slots (version 10 and later), then a temporary replication slot is automatically used for WAL streaming.

-v
--verbose

Enables verbose mode. Will output some extra steps during startup and shutdown, as well as show the exact file name that is currently being processed if progress reporting is also enabled.

--manifest-checksums=algorithm

Specifies the checksum algorithm that should be applied to each file included in the backup manifest. Currently, the available algorithms are NONE, CRC32C, SHA224, SHA256, SHA384, and SHA512. The default is CRC32C.

If NONE is selected, the backup manifest will not contain any checksums. Otherwise, it will contain a checksum of each file in the backup using the specified algorithm. In addition, the manifest will always contain a SHA256 checksum of its own contents. The SHA algorithms are significantly more CPU-intensive than CRC32C, so selecting one of them may increase the time required to complete the backup.

Using a SHA hash function provides a cryptographically secure digest of each file for users who wish to verify that the backup has not been tampered with, while the CRC32C algorithm provides a checksum that is much faster to calculate; it is good at catching errors due to accidental changes but is not resistant to malicious modifications. Note that, to be useful against an adversary who has access to the backup, the backup manifest would need to be stored securely elsewhere or otherwise verified not to have been modified since the backup was taken.

pg_verifybackup can be used to check the integrity of a backup against the backup manifest.

--manifest-force-encode

Forces all filenames in the backup manifest to be hex-encoded. If this option is not specified, only non-UTF8 filenames are hex-encoded. This option is mostly intended to test that tools which read a backup manifest file properly handle this case.

--no-estimate-size

Prevents the server from estimating the total amount of backup data that will be streamed, resulting in the backup_total column in the pg_stat_progress_basebackup view always being NULL.

Without this option, the backup will start by enumerating the size of the entire database, and then go back and send the actual contents. This may make the backup take slightly longer, and in particular it will take longer before the first data is sent. This option is useful to avoid such estimation time if it's too long.

This option is not allowed when using --progress.

--no-manifest

Disables generation of a backup manifest. If this option is not specified, the server will generate and send a backup manifest which can be verified using pg_verifybackup. The manifest is a list of every file present in the backup with the exception of any WAL files that may be included. It also stores the size, last modification time, and an optional checksum for each file.

--no-slot

Prevents the creation of a temporary replication slot for the backup.

By default, if log streaming is selected but no slot name is given with the -S option, then a temporary replication slot is created (if supported by the source server).

The main purpose of this option is to allow taking a base backup when the server has no free replication slots. Using a replication slot is almost always preferred, because it prevents needed WAL from being removed by the server during the backup.

--no-verify-checksums

Disables verification of checksums, if they are enabled on the server the base backup is taken from.

By default, checksums are verified and checksum failures will result in a non-zero exit status. However, the base backup will not be removed in such a case, as if the --no-clean option had been used. Checksum verification failures will also be reported in the pg_stat_database view.

The following command-line options control the connection to the source server:

-d connstr
--dbname=connstr

Specifies parameters used to connect to the server, as a connection string; these will override any conflicting command line options.

The option is called --dbname for consistency with other client applications, but because pg_basebackup doesn't connect to any particular database in the cluster, any database name in the connection string will be ignored.

-h host
--host=host

Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for a Unix domain socket. The default is taken from the PGHOST environment variable, if set, else a Unix domain socket connection is attempted.

-p port
--port=port

Specifies the TCP port or local Unix domain socket file extension on which the server is listening for connections. Defaults to the PGPORT environment variable, if set, or a compiled-in default.

-s interval
--status-interval=interval

Specifies the number of seconds between status packets sent back to the source server. Smaller values allow more accurate monitoring of backup progress from the server. A value of zero disables periodic status updates completely, although an update will still be sent when requested by the server, to avoid timeout-based disconnects. The default value is 10 seconds.

-U username
--username=username

Specifies the user name to connect as.

-w
--no-password

Prevents issuing a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.

-W
--password

Forces pg_basebackup to prompt for a password before connecting to the source server.

This option is never essential, since pg_basebackup will automatically prompt for a password if the server demands password authentication. However, pg_basebackup will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt.

Other options are also available:

-V
--version

Prints the pg_basebackup version and exits.

-?
--help

Shows help about pg_basebackup command line arguments, and exits.

Environment

This utility, like most other PostgreSQL utilities, uses the environment variables supported by libpq (see Section 34.15).

The environment variable PG_COLOR specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

Notes

At the beginning of the backup, a checkpoint needs to be performed on the source server. This can take some time (especially if the option --checkpoint=fast is not used), during which pg_basebackup will appear to be idle.

The backup will include all files in the data directory and tablespaces, including the configuration files and any additional files placed in the directory by third parties, except certain temporary files managed by PostgreSQL. But only regular files and directories are copied, except that symbolic links used for tablespaces are preserved. Symbolic links pointing to certain directories known to PostgreSQL are copied as empty directories. Other symbolic links and special device files are skipped. See Section 53.4 for the precise details.

In plain format, tablespaces will be backed up to the same path they have on the source server, unless the option --tablespace-mapping is used. Without this option, running a plain format base backup on the same host as the server will not work if tablespaces are in use, because the backup would have to be written to the same directory locations as the original tablespaces.

When tar format is used, it is the user's responsibility to unpack each tar file before starting a PostgreSQL server that uses the data. If there are additional tablespaces, the tar files for them need to be unpacked in the correct locations. In this case the symbolic links for those tablespaces will be created by the server according to the contents of the tablespace_map file that is included in the base.tar file.

pg_basebackup works with servers of the same or an older major version, down to 9.1. However, WAL streaming mode (-X stream) only works with server version 9.3 and later, and tar format (--format=tar) only works with server version 9.5 and later.

pg_basebackup will preserve group permissions for data files if group permissions are enabled on the source cluster.

Examples

To create a base backup of the server at mydbserver and store it in the local directory /usr/local/pgsql/data:

$ pg_basebackup -h mydbserver -D /usr/local/pgsql/data

To create a backup of the local server with one compressed tar file for each tablespace, and store it in the directory backup, showing a progress report while running:

$ pg_basebackup -D backup -Ft -z -P

To create a backup of a single-tablespace local database and compress this with bzip2:

$ pg_basebackup -D - -Ft -X fetch | bzip2 > backup.tar.bz2

(This command will fail if there are multiple tablespaces in the database.)

To create a backup of a local database where the tablespace in /opt/ts is relocated to ./backup/ts:

$ pg_basebackup -D backup/data -T /opt/ts=$(pwd)/backup/ts

See Also

pg_dump, Section 28.4.5
Prev | Up | Next
pg_amcheck | Home | pgbench


pgbench
Prev Up PostgreSQL Client Applications Home Next

pgbench

pgbench — run a benchmark test on PostgreSQL

Synopsis

pgbench -i [option...] [dbname]

pgbench [option...] [dbname]

Description

pgbench is a simple program for running benchmark tests on PostgreSQL. It runs the same sequence of SQL commands over and over, possibly in multiple concurrent database sessions, and then calculates the average transaction rate (transactions per second). By default, pgbench tests a scenario that is loosely based on TPC-B, involving five SELECT, UPDATE, and INSERT commands per transaction. However, it is easy to test other cases by writing your own transaction script files.

Typical output from pgbench looks like:

transaction type: <builtin: TPC-B (sort of)>
scaling factor: 10
query mode: simple
number of clients: 10
number of threads: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
latency average = 11.013 ms
latency stddev = 7.351 ms
initial connection time = 45.758 ms
tps = 896.967014 (without initial connection time)

The first six lines report some of the most important parameter settings. The next line reports the number of transactions completed and intended (the latter being just the product of number of clients and number of transactions per client); these will be equal unless the run failed before completion. (In -T mode, only the actual number of transactions is printed.) The last line reports the number of transactions per second.

The default TPC-B-like transaction test requires specific tables to be set up beforehand. pgbench should be invoked with the -i (initialize) option to create and populate these tables. (When you are testing a custom script, you don't need this step, but will instead need to do whatever setup your test needs.) Initialization looks like:

pgbench -i [ other-options ] dbname

where dbname is the name of the already-created database to test in. (You may also need -h, -p, and/or -U options to specify how to connect to the database server.)

Caution

pgbench -i creates four tables pgbench_accounts, pgbench_branches, pgbench_history, and pgbench_tellers, destroying any existing tables of these names. Be very careful to use another database if you have tables having these names!

At the default “scale factor” of 1, the tables initially contain this many rows:

table                   # of rows
---------------------------------
pgbench_branches        1
pgbench_tellers         10
pgbench_accounts        100000
pgbench_history         0

You can (and, for most purposes, probably should) increase the number of rows by using the -s (scale factor) option. The -F (fillfactor) option might also be used at this point.

Once you have done the necessary setup, you can run your benchmark with a command that doesn't include -i, that is

pgbench [ options ] dbname

In nearly all cases, you'll need some options to make a useful test. The most important options are -c (number of clients), -t (number of transactions), -T (time limit), and -f (specify a custom script file). See below for a full list.

Options

The following is divided into three subsections. Different options are used during database initialization and while running benchmarks, but some options are useful in both cases.

Initialization Options

pgbench accepts the following command-line initialization arguments:

dbname

Specifies the name of the database to test in. If this is not specified, the environment variable PGDATABASE is used. If that is not set, the user name specified for the connection is used.

-i
--initialize

Required to invoke initialization mode.

-I init_steps
--init-steps=init_steps

Perform just a selected set of the normal initialization steps. init_steps specifies the initialization steps to be performed, using one character per step. Each step is invoked in the specified order. The default is dtgvp. The available steps are:

d (Drop)

Drop any existing pgbench tables.

t (create Tables)

Create the tables used by the standard pgbench scenario, namely pgbench_accounts, pgbench_branches, pgbench_history, and pgbench_tellers.

g or G (Generate data, client-side or server-side)

Generate data and load it into the standard tables, replacing any data already present.

With g (client-side data generation), data is generated in pgbench client and then sent to the server. This uses the client/server bandwidth extensively through a COPY. Using g causes logging to print one message every 100,000 rows while generating data for the pgbench_accounts table.

With G (server-side data generation), only small queries are sent from the pgbench client and then data is actually generated in the server. No significant bandwidth is required for this variant, but the server will do more work. Using G causes logging not to print any progress message while generating data.

The default initialization behavior uses client-side data generation (equivalent to g).

v (Vacuum)

Invoke VACUUM on the standard tables.

p (create Primary keys)

Create primary key indexes on the standard tables.

f (create Foreign keys)

Create foreign key constraints between the standard tables. (Note that this step is not performed by default.)

-F fillfactor
--fillfactor=fillfactor

Create the pgbench_accounts, pgbench_tellers and pgbench_branches tables with the given fillfactor. Default is 100.

-n
--no-vacuum

Perform no vacuuming during initialization. (This option suppresses the v initialization step, even if it was specified in -I.)

-q
--quiet

Switch logging to quiet mode, producing only one progress message per 5 seconds. The default logging prints one message each 100,000 rows, which often outputs many lines per second (especially on good hardware).

This setting has no effect if G is specified in -I.

-s scale_factor
--scale=scale_factor

Multiply the number of rows generated by the scale factor. For example, -s 100 will create 10,000,000 rows in the pgbench_accounts table. Default is 1. When the scale is 20,000 or larger, the columns used to hold account identifiers (aid columns) will switch to using larger integers (bigint), in order to be big enough to hold the range of account identifiers.

--foreign-keys

Create foreign key constraints between the standard tables. (This option adds the f step to the initialization step sequence, if it is not already present.)

--index-tablespace=index_tablespace

Create indexes in the specified tablespace, rather than the default tablespace.

--partition-method=NAME

Create a partitioned pgbench_accounts table with NAME method. Expected values are range or hash. This option requires that --partitions is set to non-zero. If unspecified, default is range.

--partitions=NUM

Create a partitioned pgbench_accounts table with NUM partitions of nearly equal size for the scaled number of accounts. Default is 0, meaning no partitioning.

--tablespace=tablespace

Create tables in the specified tablespace, rather than the default tablespace.

--unlogged-tables

Create all tables as unlogged tables, rather than permanent tables.

Benchmarking Options

pgbench accepts the following command-line benchmarking arguments:

-b scriptname[@weight]
--builtin=scriptname[@weight]

Add the specified built-in script to the list of scripts to be executed. Available built-in scripts are: tpcb-like, simple-update and select-only. Unambiguous prefixes of built-in names are accepted. With the special name list, show the list of built-in scripts and exit immediately.

Optionally, write an integer weight after @ to adjust the probability of selecting this script versus other ones. The default weight is 1. See below for details.

-c clients
--client=clients

Number of clients simulated, that is, number of concurrent database sessions. Default is 1.

-C
--connect

Establish a new connection for each transaction, rather than doing it just once per client session. This is useful to measure the connection overhead.

-d
--debug

Print debugging output.

-D varname=value
--define=varname=value

Define a variable for use by a custom script (see below). Multiple -D options are allowed.

-f filename[@weight]
--file=filename[@weight]

Add a transaction script read from filename to the list of scripts to be executed.

Optionally, write an integer weight after @ to adjust the probability of selecting this script versus other ones. The default weight is 1. (To use a script file name that includes an @ character, append a weight so that there is no ambiguity, for example filen@me@1.) See below for details.

-j threads
--jobs=threads

Number of worker threads within pgbench. Using more than one thread can be helpful on multi-CPU machines. Clients are distributed as evenly as possible among available threads. Default is 1.

-l
--log

Write information about each transaction to a log file. See below for details.

-L limit
--latency-limit=limit

Transactions that last more than limit milliseconds are counted and reported separately, as late.

When throttling is used (--rate=...), transactions that lag behind schedule by more than limit ms, and thus have no hope of meeting the latency limit, are not sent to the server at all. They are counted and reported separately as skipped.

-M querymode
--protocol=querymode

Protocol to use for submitting queries to the server:

  • simple: use simple query protocol.

  • extended: use extended query protocol.

  • prepared: use extended query protocol with prepared statements.

In the prepared mode, pgbench reuses the parse analysis result starting from the second query iteration, so pgbench runs faster than in other modes.

The default is simple query protocol. (See Chapter 53 for more information.)

-n
--no-vacuum

Perform no vacuuming before running the test. This option is necessary if you are running a custom test scenario that does not include the standard tables pgbench_accounts, pgbench_branches, pgbench_history, and pgbench_tellers.

-N
--skip-some-updates

Run built-in simple-update script. Shorthand for -b simple-update.

-P sec
--progress=sec

Show progress report every sec seconds. The report includes the time since the beginning of the run, the TPS since the last report, and the transaction latency average and standard deviation since the last report. Under throttling (-R), the latency is computed with respect to the transaction scheduled start time, not the actual transaction beginning time, thus it also includes the average schedule lag time.

-r
--report-latencies

Report the average per-statement latency (execution time from the perspective of the client) of each command after the benchmark finishes. See below for details.

-R rate
--rate=rate

Execute transactions targeting the specified rate instead of running as fast as possible (the default). The rate is given in transactions per second. If the targeted rate is above the maximum possible rate, the rate limit won't impact the results.

The rate is targeted by starting transactions along a Poisson-distributed schedule time line. The expected start time schedule moves forward based on when the client first started, not when the previous transaction ended. That approach means that when transactions go past their original scheduled end time, it is possible for later ones to catch up again.

When throttling is active, the transaction latency reported at the end of the run is calculated from the scheduled start times, so it includes the time each transaction had to wait for the previous transaction to finish. The wait time is called the schedule lag time, and its average and maximum are also reported separately. The transaction latency with respect to the actual transaction start time, i.e., the time spent executing the transaction in the database, can be computed by subtracting the schedule lag time from the reported latency.

If --latency-limit is used together with --rate, a transaction can lag behind so much that it is already over the latency limit when the previous transaction ends, because the latency is calculated from the scheduled start time. Such transactions are not sent to the server, but are skipped altogether and counted separately.

A high schedule lag time is an indication that the system cannot process transactions at the specified rate, with the chosen number of clients and threads. When the average transaction execution time is longer than the scheduled interval between each transaction, each successive transaction will fall further behind, and the schedule lag time will keep increasing the longer the test run is. When that happens, you will have to reduce the specified transaction rate.

-s scale_factor
--scale=scale_factor

Report the specified scale factor in pgbench's output. With the built-in tests, this is not necessary; the correct scale factor will be detected by counting the number of rows in the pgbench_branches table. However, when testing only custom benchmarks (-f option), the scale factor will be reported as 1 unless this option is used.

-S
--select-only

Run built-in select-only script. Shorthand for -b select-only.

-t transactions
--transactions=transactions

Number of transactions each client runs. Default is 10.

-T seconds
--time=seconds

Run the test for this many seconds, rather than a fixed number of transactions per client. -t and -T are mutually exclusive.

-v
--vacuum-all

Vacuum all four standard tables before running the test. With neither -n nor -v, pgbench will vacuum the pgbench_tellers and pgbench_branches tables, and will truncate pgbench_history.

--aggregate-interval=seconds

Length of aggregation interval (in seconds). May be used only with -l option. With this option, the log contains per-interval summary data, as described below.

--log-prefix=prefix

Set the filename prefix for the log files created by --log. The default is pgbench_log.

--progress-timestamp

When showing progress (option -P), use a timestamp (Unix epoch) instead of the number of seconds since the beginning of the run. The unit is in seconds, with millisecond precision after the dot. This helps compare logs generated by various tools.

--random-seed=seed

Set random generator seed. Seeds the system random number generator, which then produces a sequence of initial generator states, one for each thread. Values for seed may be: time (the default, the seed is based on the current time), rand (use a strong random source, failing if none is available), or an unsigned decimal integer value. The random generator is invoked explicitly from a pgbench script (random... functions) or implicitly (for instance option --rate uses it to schedule transactions). When explicitly set, the value used for seeding is shown on the terminal. Any value allowed for seed may also be provided through the environment variable PGBENCH_RANDOM_SEED. To ensure that the provided seed impacts all possible uses, put this option first or use the environment variable.

Setting the seed explicitly allows to reproduce a pgbench run exactly, as far as random numbers are concerned. As the random state is managed per thread, this means the exact same pgbench run for an identical invocation if there is one client per thread and there are no external or data dependencies. From a statistical viewpoint reproducing runs exactly is a bad idea because it can hide the performance variability or improve performance unduly, e.g., by hitting the same pages as a previous run. However, it may also be of great help for debugging, for instance re-running a tricky case which leads to an error. Use wisely.

--sampling-rate=rate

Sampling rate, used when writing data into the log, to reduce the amount of log generated. If this option is given, only the specified fraction of transactions are logged. 1.0 means all transactions will be logged, 0.05 means only 5% of the transactions will be logged.

Remember to take the sampling rate into account when processing the log file. For example, when computing TPS values, you need to multiply the numbers accordingly (e.g., with 0.01 sample rate, you'll only get 1/100 of the actual TPS).

--show-script=scriptname

Show the actual code of builtin script scriptname on stderr, and exit immediately.

Common Options

pgbench also accepts the following common command-line arguments for connection parameters:

-h hostname
--host=hostname

The database server's host name

-p port
--port=port

The database server's port number

-U login
--username=login

The user name to connect as

-V
--version

Print the pgbench version and exit.

-?
--help

Show help about pgbench command line arguments, and exit.

Exit Status

A successful run will exit with status 0. Exit status 1 indicates static problems such as invalid command-line options. Errors during the run such as database errors or problems in the script will result in exit status 2. In the latter case, pgbench will print partial results.

Environment

PGDATABASE
PGHOST
PGPORT
PGUSER

Default connection parameters.

This utility, like most other PostgreSQL utilities, uses the environment variables supported by libpq (see Section 34.15).

The environment variable PG_COLOR specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

Notes

What Is the “Transaction” Actually Performed in pgbench?

pgbench executes test scripts chosen randomly from a specified list. The scripts may include built-in scripts specified with -b and user-provided scripts specified with -f. Each script may be given a relative weight specified after an @ so as to change its selection probability. The default weight is 1. Scripts with a weight of 0 are ignored.

The default built-in transaction script (also invoked with -b tpcb-like) issues seven commands per transaction over randomly chosen aid, tid, bid and delta. The scenario is inspired by the TPC-B benchmark, but is not actually TPC-B, hence the name.

  1. BEGIN;

  2. UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;

  3. SELECT abalance FROM pgbench_accounts WHERE aid = :aid;

  4. UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;

  5. UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;

  6. INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);

  7. END;

If you select the simple-update built-in (also -N), steps 4 and 5 aren't included in the transaction. This will avoid update contention on these tables, but it makes the test case even less like TPC-B.

If you select the select-only built-in (also -S), only the SELECT is issued.

Custom Scripts

pgbench has support for running custom benchmark scenarios by replacing the default transaction script (described above) with a transaction script read from a file (-f option). In this case a “transaction” counts as one execution of a script file.

A script file contains one or more SQL commands terminated by semicolons. Empty lines and lines beginning with -- are ignored. Script files can also contain “meta commands”, which are interpreted by pgbench itself, as described below.

Note

Before PostgreSQL 9.6, SQL commands in script files were terminated by newlines, and so they could not be continued across lines. Now a semicolon is required to separate consecutive SQL commands (though an SQL command does not need one if it is followed by a meta command). If you need to create a script file that works with both old and new versions of pgbench, be sure to write each SQL command on a single line ending with a semicolon.

There is a simple variable-substitution facility for script files. Variable names must consist of letters (including non-Latin letters), digits, and underscores, with the first character not being a digit. Variables can be set by the command-line -D option, explained above, or by the meta commands explained below. In addition to any variables preset by -D command-line options, there are a few variables that are preset automatically, listed in Table 282. A value specified for these variables using -D takes precedence over the automatic presets. Once set, a variable's value can be inserted into an SQL command by writing :variablename. When running more than one client session, each session has its own set of variables. pgbench supports up to 255 variable uses in one statement.

Table 282. pgbench Automatic Variables

Variable | Description
client_id | unique number identifying the client session (starts from zero)
default_seed | seed used in hash and pseudorandom permutation functions by default
random_seed | random generator seed (unless overwritten with -D)
scale | current scale factor

Script file meta commands begin with a backslash (\) and normally extend to the end of the line, although they can be continued to additional lines by writing backslash-return. Arguments to a meta command are separated by white space. These meta commands are supported:

\gset [prefix] \aset [prefix]

These commands may be used to end SQL queries, taking the place of the terminating semicolon (;).

When the \gset command is used, the preceding SQL query is expected to return one row, the columns of which are stored into variables named after column names, and prefixed with prefix if provided.

When the \aset command is used, all combined SQL queries (separated by \;) have their columns stored into variables named after column names, and prefixed with prefix if provided. If a query returns no row, no assignment is made and the variable can be tested for existence to detect this. If a query returns more than one row, the last value is kept.

\gset and \aset cannot be used in pipeline mode, since the query results are not yet available by the time the commands would need them.

The following example puts the final account balance from the first query into variable abalance, and fills variables p_two and p_three with integers from the third query. The result of the second query is discarded. The result of the two last combined queries are stored in variables four and five.

UPDATE pgbench_accounts
  SET abalance = abalance + :delta
  WHERE aid = :aid
  RETURNING abalance \gset
-- compound of two queries
SELECT 1 \;
SELECT 2 AS two, 3 AS three \gset p_
SELECT 4 AS four \; SELECT 5 AS five \aset
\if expression
\elif expression
\else
\endif

This group of commands implements nestable conditional blocks, similarly to psql's \if expression. Conditional expressions are identical to those with \set, with non-zero values interpreted as true.

\set varname expression

Sets variable varname to a value calculated from expression. The expression may contain the NULL constant, Boolean constants TRUE and FALSE, integer constants such as 5432, double constants such as 3.14159, references to variables :variablename, operators with their usual SQL precedence and associativity, function calls, SQL CASE generic conditional expressions and parentheses.

Functions and most operators return NULL on NULL input.

For conditional purposes, non zero numerical values are TRUE, zero numerical values and NULL are FALSE.

Too large or small integer and double constants, as well as integer arithmetic operators (+, -, * and /) raise errors on overflows.

When no final ELSE clause is provided to a CASE, the default value is NULL.

Examples:

\set ntellers 10 * :scale
\set aid (1021 * random(1, 100000 * :scale)) % \
           (100000 * :scale) + 1
\set divx CASE WHEN :x <> 0 THEN :y/:x ELSE NULL END
\sleep number [ us | ms | s ]

Causes script execution to sleep for the specified duration in microseconds (us), milliseconds (ms) or seconds (s). If the unit is omitted then seconds are the default. number can be either an integer constant or a :variablename reference to a variable having an integer value.

Example:

\sleep 10 ms
\setshell varname command [ argument ... ]

Sets variable varname to the result of the shell command command with the given argument(s). The command must return an integer value through its standard output.

command and each argument can be either a text constant or a :variablename reference to a variable. If you want to use an argument starting with a colon, write an additional colon at the beginning of argument.

Example:

\setshell variable_to_be_assigned command literal_argument :variable ::literal_starting_with_colon
\shell command [ argument ... ]

Same as \setshell, but the result of the command is discarded.

Example:

\shell command literal_argument :variable ::literal_starting_with_colon
\startpipeline
\endpipeline

These commands delimit the start and end of a pipeline of SQL statements. In pipeline mode, statements are sent to the server without waiting for the results of previous statements. See Section 34.5 for more details. Pipeline mode requires the use of extended query protocol.

Built-in Operators

The arithmetic, bitwise, comparison and logical operators listed in Table 283 are built into pgbench and may be used in expressions appearing in \set. The operators are listed in increasing precedence order. Except as noted, operators taking two numeric inputs will produce a double value if either input is double, otherwise they produce an integer result.

Table 283. pgbench Operators

Operator

Description

Example(s)

boolean OR boolean → boolean

Logical OR

5 or 0 → TRUE

boolean AND boolean → boolean

Logical AND

3 and 0 → FALSE

NOT boolean → boolean

Logical NOT

not false → TRUE

boolean IS [NOT] (NULL|TRUE|FALSE) → boolean

Boolean value tests

1 is null → FALSE

value ISNULL|NOTNULL → boolean

Nullness tests

1 notnull → TRUE

number = number → boolean

Equal

5 = 4 → FALSE

number <> number → boolean

Not equal

5 <> 4 → TRUE

number != number → boolean

Not equal

5 != 5 → FALSE

number < number → boolean

Less than

5 < 4 → FALSE

number <= number → boolean

Less than or equal to

5 <= 4 → FALSE

number > number → boolean

Greater than

5 > 4 → TRUE

number >= number → boolean

Greater than or equal to

5 >= 4 → TRUE

integer | integer → integer

Bitwise OR

1 | 2 → 3

integer # integer → integer

Bitwise XOR

1 # 3 → 2

integer & integer → integer

Bitwise AND

1 & 3 → 1

~ integer → integer

Bitwise NOT

~ 1 → -2

integer << integer → integer

Bitwise shift left

1 << 2 → 4

integer >> integer → integer

Bitwise shift right

8 >> 2 → 2

number + number → number

Addition

5 + 4 → 9

number - number → number

Subtraction

3 - 2.0 → 1.0

number * number → number

Multiplication

5 * 4 → 20

number / number → number

Division (truncates the result towards zero if both inputs are integers)

5 / 3 → 1

integer % integer → integer

Modulo (remainder)

3 % 2 → 1

- number → number

Negation

- 2.0 → -2.0

Built-In Functions

The functions listed in Table 284 are built into pgbench and may be used in expressions appearing in \set.

Table 284. pgbench Functions

Function

Description

Example(s)

abs ( number ) → same type as input

Absolute value

abs(-17) → 17

debug ( number ) → same type as input

Prints the argument to stderr, and returns the argument.

debug(5432.1) → 5432.1

double ( number ) → double

Casts to double.

double(5432) → 5432.0

exp ( number ) → double

Exponential (e raised to the given power)

exp(1.0) → 2.718281828459045

greatest ( number [, ... ] ) → double if any argument is double, else integer

Selects the largest value among the arguments.

greatest(5, 4, 3, 2) → 5

hash ( value [, seed ] ) → integer

This is an alias for hash_murmur2.

hash(10, 5432) → -5817877081768721676

hash_fnv1a ( value [, seed ] ) → integer

Computes FNV-1a hash.

hash_fnv1a(10, 5432) → -7793829335365542153

hash_murmur2 ( value [, seed ] ) → integer

Computes MurmurHash2 hash.

hash_murmur2(10, 5432) → -5817877081768721676

int ( number ) → integer

Casts to integer.

int(5.4 + 3.8) → 9

least ( number [, ... ] ) → double if any argument is double, else integer

Selects the smallest value among the arguments.

least(5, 4, 3, 2.1) → 2.1

ln ( number ) → double

Natural logarithm

ln(2.718281828459045) → 1.0

mod ( integer, integer ) → integer

Modulo (remainder)

mod(54, 32) → 22

permute ( i, size [, seed ] ) → integer

Permuted value of i, in the range [0, size). This is the new position of i (modulo size) in a pseudorandom permutation of the integers 0...size-1, parameterized by seed, see below.

permute(0, 4) → an integer between 0 and 3

pi () → double

Approximate value of π

pi() → 3.14159265358979323846

pow ( x, y ) → double

power ( x, y ) → double

x raised to the power of y

pow(2.0, 10) → 1024.0

random ( lb, ub ) → integer

Computes a uniformly-distributed random integer in [lb, ub].

random(1, 10) → an integer between 1 and 10

random_exponential ( lb, ub, parameter ) → integer

Computes an exponentially-distributed random integer in [lb, ub], see below.

random_exponential(1, 10, 3.0) → an integer between 1 and 10

random_gaussian ( lb, ub, parameter ) → integer

Computes a Gaussian-distributed random integer in [lb, ub], see below.

random_gaussian(1, 10, 2.5) → an integer between 1 and 10

random_zipfian ( lb, ub, parameter ) → integer

Computes a Zipfian-distributed random integer in [lb, ub], see below.

random_zipfian(1, 10, 1.5) → an integer between 1 and 10

sqrt ( number ) → double

Square root

sqrt(2.0) → 1.414213562

The random function generates values using a uniform distribution, that is all the values are drawn within the specified range with equal probability. The random_exponential, random_gaussian and random_zipfian functions require an additional double parameter which determines the precise shape of the distribution.

  • For an exponential distribution, parameter controls the distribution by truncating a quickly-decreasing exponential distribution at parameter, and then projecting onto integers between the bounds. To be precise, with

    f(x) = exp(-parameter * (x - min) / (max - min + 1)) / (1 - exp(-parameter))

    Then value i between min and max inclusive is drawn with probability: f(i) - f(i + 1).

    Intuitively, the larger the parameter, the more frequently values close to min are accessed, and the less frequently values close to max are accessed. The closer to 0 parameter is, the flatter (more uniform) the access distribution. A crude approximation of the distribution is that the most frequent 1% values in the range, close to min, are drawn parameter% of the time. The parameter value must be strictly positive.

  • For a Gaussian distribution, the interval is mapped onto a standard normal distribution (the classical bell-shaped Gaussian curve) truncated at -parameter on the left and +parameter on the right. Values in the middle of the interval are more likely to be drawn. To be precise, if PHI(x) is the cumulative distribution function of the standard normal distribution, with mean mu defined as (max + min) / 2.0, with

    f(x) = PHI(2.0 * parameter * (x - mu) / (max - min + 1)) /
    (2.0 * PHI(parameter) - 1)

    then value i between min and max inclusive is drawn with probability: f(i + 0.5) - f(i - 0.5). Intuitively, the larger the parameter, the more frequently values close to the middle of the interval are drawn, and the less frequently values close to the min and max bounds. About 67% of values are drawn from the middle 1.0 / parameter, that is a relative 0.5 / parameter around the mean, and 95% in the middle 2.0 / parameter, that is a relative 1.0 / parameter around the mean; for instance, if parameter is 4.0, 67% of values are drawn from the middle quarter (1.0 / 4.0) of the interval (i.e., from 3.0 / 8.0 to 5.0 / 8.0) and 95% from the middle half (2.0 / 4.0) of the interval (second and third quartiles). The minimum allowed parameter value is 2.0.

  • random_zipfian generates a bounded Zipfian distribution. parameter defines how skewed the distribution is. The larger the parameter, the more frequently values closer to the beginning of the interval are drawn. The distribution is such that, assuming the range starts from 1, the ratio of the probability of drawing k versus drawing k+1 is ((k+1)/k)**parameter. For example, random_zipfian(1, ..., 2.5) produces the value 1 about (2/1)**2.5 = 5.66 times more frequently than 2, which itself is produced (3/2)**2.5 = 2.76 times more frequently than 3, and so on.

    pgbench's implementation is based on "Non-Uniform Random Variate Generation", Luc Devroye, p. 550-551, Springer 1986. Due to limitations of that algorithm, the parameter value is restricted to the range [1.001, 1000].

Note

When designing a benchmark which selects rows non-uniformly, be aware that the rows chosen may be correlated with other data such as IDs from a sequence or the physical row ordering, which may skew performance measurements.

To avoid this, you may wish to use the permute function, or some other additional step with similar effect, to shuffle the selected rows and remove such correlations.

Hash functions hash, hash_murmur2 and hash_fnv1a accept an input value and an optional seed parameter. In case the seed isn't provided the value of :default_seed is used, which is initialized randomly unless set by the command-line -D option.

permute accepts an input value, a size, and an optional seed parameter. It generates a pseudorandom permutation of integers in the range [0, size), and returns the index of the input value in the permuted values. The permutation chosen is parameterized by the seed, which defaults to :default_seed, if not specified. Unlike the hash functions, permute ensures that there are no collisions or holes in the output values. Input values outside the interval are interpreted modulo the size. The function raises an error if the size is not positive. permute can be used to scatter the distribution of non-uniform random functions such as random_zipfian or random_exponential so that values drawn more often are not trivially correlated. For instance, the following pgbench script simulates a possible real world workload typical for social media and blogging platforms where a few accounts generate excessive load:

\set size 1000000
\set r random_zipfian(1, :size, 1.07)
\set k 1 + permute(:r, :size)

In some cases several distinct distributions are needed which don't correlate with each other and this is when the optional seed parameter comes in handy:

\set k1 1 + permute(:r, :size, :default_seed + 123)
\set k2 1 + permute(:r, :size, :default_seed + 321)

A similar behavior can also be approximated with hash:

\set size 1000000
\set r random_zipfian(1, 100 * :size, 1.07)
\set k 1 + abs(hash(:r)) % :size

However, since hash generates collisions, some values will not be reachable and others will be more frequent than expected from the original distribution.

As an example, the full definition of the built-in TPC-B-like transaction is:

\set aid random(1, 100000 * :scale)
\set bid random(1, 1 * :scale)
\set tid random(1, 10 * :scale)
\set delta random(-5000, 5000)
BEGIN;
UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
END;

This script allows each iteration of the transaction to reference different, randomly-chosen rows. (This example also shows why it's important for each client session to have its own variables — otherwise they'd not be independently touching different rows.)

Per-Transaction Logging

With the -l option (but without the --aggregate-interval option), pgbench writes information about each transaction to a log file. The log file will be named prefix.nnn, where prefix defaults to pgbench_log, and nnn is the PID of the pgbench process. The prefix can be changed by using the --log-prefix option. If the -j option is 2 or higher, so that there are multiple worker threads, each will have its own log file. The first worker will use the same name for its log file as in the standard single worker case. The additional log files for the other workers will be named prefix.nnn.mmm, where mmm is a sequential number for each worker starting with 1.

The format of the log is:

client_id transaction_no time script_no time_epoch time_us [ schedule_lag ]

where client_id indicates which client session ran the transaction, transaction_no counts how many transactions have been run by that session, time is the total elapsed transaction time in microseconds, script_no identifies which script file was used (useful when multiple scripts were specified with -f or -b), and time_epoch/time_us are a Unix-epoch time stamp and an offset in microseconds (suitable for creating an ISO 8601 time stamp with fractional seconds) showing when the transaction completed. The schedule_lag field is the difference between the transaction's scheduled start time, and the time it actually started, in microseconds. It is only present when the --rate option is used. When both --rate and --latency-limit are used, the time for a skipped transaction will be reported as skipped.

Here is a snippet of a log file generated in a single-client run:

0 199 2241 0 1175850568 995598
0 200 2465 0 1175850568 998079
0 201 2513 0 1175850569 608
0 202 2038 0 1175850569 2663

Another example with --rate=100 and --latency-limit=5 (note the additional schedule_lag column):

0 81 4621 0 1412881037 912698 3005
0 82 6173 0 1412881037 914578 4304
0 83 skipped 0 1412881037 914578 5217
0 83 skipped 0 1412881037 914578 5099
0 83 4722 0 1412881037 916203 3108
0 84 4142 0 1412881037 918023 2333
0 85 2465 0 1412881037 919759 740

In this example, transaction 82 was late, because its latency (6.173 ms) was over the 5 ms limit. The next two transactions were skipped, because they were already late before they were even started.

When running a long test on hardware that can handle a lot of transactions, the log files can become very large. The --sampling-rate option can be used to log only a random sample of transactions.

Aggregated Logging

With the --aggregate-interval option, a different format is used for the log files:

interval_start num_transactions​ sum_latency sum_latency_2 min_latency max_latency​ [ sum_lag sum_lag_2 min_lag max_lag [ skipped ] ]

where interval_start is the start of the interval (as a Unix epoch time stamp), num_transactions is the number of transactions within the interval, sum_latency is the sum of the transaction latencies within the interval, sum_latency_2 is the sum of squares of the transaction latencies within the interval, min_latency is the minimum latency within the interval, and max_latency is the maximum latency within the interval. The next fields, sum_lag, sum_lag_2, min_lag, and max_lag, are only present if the --rate option is used. They provide statistics about the time each transaction had to wait for the previous one to finish, i.e., the difference between each transaction's scheduled start time and the time it actually started. The very last field, skipped, is only present if the --latency-limit option is used, too. It counts the number of transactions skipped because they would have started too late. Each transaction is counted in the interval when it was committed.

Here is some example output:

1345828501 5601 1542744 483552416 61 2573
1345828503 7884 1979812 565806736 60 1479
1345828505 7208 1979422 567277552 59 1391
1345828507 7685 1980268 569784714 60 1398
1345828509 7073 1979779 573489941 236 1411

Notice that while the plain (unaggregated) log file shows which script was used for each transaction, the aggregated log does not. Therefore if you need per-script data, you need to aggregate the data on your own.

Per-Statement Latencies

With the -r option, pgbench collects the elapsed transaction time of each statement executed by every client. It then reports an average of those values, referred to as the latency for each statement, after the benchmark has finished.

For the default script, the output will look similar to this:

starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 1
query mode: simple
number of clients: 10
number of threads: 1
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
latency average = 10.870 ms
latency stddev = 7.341 ms
initial connection time = 30.954 ms
tps = 907.949122 (without initial connection time)
statement latencies in milliseconds:
    0.001  \set aid random(1, 100000 * :scale)
    0.001  \set bid random(1, 1 * :scale)
    0.001  \set tid random(1, 10 * :scale)
    0.000  \set delta random(-5000, 5000)
    0.046  BEGIN;
    0.151  UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
    0.107  SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
    4.241  UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
    5.245  UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
    0.102  INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
    0.974  END;

If multiple script files are specified, the averages are reported separately for each script file.

Note that collecting the additional timing information needed for per-statement latency computation adds some overhead. This will slow average execution speed and lower the computed TPS. The amount of slowdown varies significantly depending on platform and hardware. Comparing average TPS values with and without latency reporting enabled is a good way to measure if the timing overhead is significant.

Good Practices

It is very easy to use pgbench to produce completely meaningless numbers. Here are some guidelines to help you get useful results.

In the first place, never believe any test that runs for only a few seconds. Use the -t or -T option to make the run last at least a few minutes, so as to average out noise. In some cases you could need hours to get numbers that are reproducible. It's a good idea to try the test run a few times, to find out if your numbers are reproducible or not.

For the default TPC-B-like test scenario, the initialization scale factor (-s) should be at least as large as the largest number of clients you intend to test (-c); else you'll mostly be measuring update contention. There are only -s rows in the pgbench_branches table, and every transaction wants to update one of them, so -c values in excess of -s will undoubtedly result in lots of transactions blocked waiting for other transactions.

The default test scenario is also quite sensitive to how long it's been since the tables were initialized: accumulation of dead rows and dead space in the tables changes the results. To understand the results you must keep track of the total number of updates and when vacuuming happens. If autovacuum is enabled it can result in unpredictable changes in measured performance.

A limitation of pgbench is that it can itself become the bottleneck when trying to test a large number of client sessions. This can be alleviated by running pgbench on a different machine from the database server, although low network latency will be essential. It might even be useful to run several pgbench instances concurrently, on several client machines, against the same database server.

Security

If untrusted users have access to a database that has not adopted a secure schema usage pattern, do not run pgbench in that database. pgbench uses unqualified names and does not manipulate the search path.

Prev | Up | Next
pg_basebackup | Home | pg_config


pg_config
Prev Up PostgreSQL Client Applications Home Next

pg_config

pg_config — retrieve information about the installed version of PostgreSQL

Synopsis

pg_config [option...]

Description

The pg_config utility prints configuration parameters of the currently installed version of PostgreSQL. It is intended, for example, to be used by software packages that want to interface to PostgreSQL to facilitate finding the required header files and libraries.

Options

To use pg_config, supply one or more of the following options:

--bindir

Print the location of user executables. Use this, for example, to find the psql program. This is normally also the location where the pg_config program resides.

--docdir

Print the location of documentation files.

--htmldir

Print the location of HTML documentation files.

--includedir

Print the location of C header files of the client interfaces.

--pkgincludedir

Print the location of other C header files.

--includedir-server

Print the location of C header files for server programming.

--libdir

Print the location of object code libraries.

--pkglibdir

Print the location of dynamically loadable modules, or where the server would search for them. (Other architecture-dependent data files might also be installed in this directory.)

--localedir

Print the location of locale support files. (This will be an empty string if locale support was not configured when PostgreSQL was built.)

--mandir

Print the location of manual pages.

--sharedir

Print the location of architecture-independent support files.

--sysconfdir

Print the location of system-wide configuration files.

--pgxs

Print the location of extension makefiles.

--configure

Print the options that were given to the configure script when PostgreSQL was configured for building. This can be used to reproduce the identical configuration, or to find out with what options a binary package was built. (Note however that binary packages often contain vendor-specific custom patches.) See also the examples below.

--cc

Print the value of the CC variable that was used for building PostgreSQL. This shows the C compiler used.

--cppflags

Print the value of the CPPFLAGS variable that was used for building PostgreSQL. This shows C compiler switches needed at preprocessing time (typically, -I switches).

--cflags

Print the value of the CFLAGS variable that was used for building PostgreSQL. This shows C compiler switches.

--cflags_sl

Print the value of the CFLAGS_SL variable that was used for building PostgreSQL. This shows extra C compiler switches used for building shared libraries.

--ldflags

Print the value of the LDFLAGS variable that was used for building PostgreSQL. This shows linker switches.

--ldflags_ex

Print the value of the LDFLAGS_EX variable that was used for building PostgreSQL. This shows linker switches used for building executables only.

--ldflags_sl

Print the value of the LDFLAGS_SL variable that was used for building PostgreSQL. This shows linker switches used for building shared libraries only.

--libs

Print the value of the LIBS variable that was used for building PostgreSQL. This normally contains -l switches for external libraries linked into PostgreSQL.

--version

Print the version of PostgreSQL.

-?
--help

Show help about pg_config command line arguments, and exit.

If more than one option is given, the information is printed in that order, one item per line. If no options are given, all available information is printed, with labels.

Notes

The options --docdir, --pkgincludedir, --localedir, --mandir, --sharedir, --sysconfdir, --cc, --cppflags, --cflags, --cflags_sl, --ldflags, --ldflags_sl, and --libs were added in PostgreSQL 8.1. The option --htmldir was added in PostgreSQL 8.4. The option --ldflags_ex was added in PostgreSQL 9.0.

Example

To reproduce the build configuration of the current PostgreSQL installation, run the following command:

eval ./configure `pg_config --configure`

The output of pg_config --configure contains shell quotation marks so arguments with spaces are represented correctly. Therefore, using eval is required for proper results.

Prev | Up | Next
pgbench | Home | pg_dump


pg_dump
Prev Up PostgreSQL Client Applications Home Next

pg_dump

pg_dump — extract a PostgreSQL database into a script file or other archive file

Synopsis

pg_dump [connection-option...] [option...] [dbname]

Description

pg_dump is a utility for backing up a PostgreSQL database. It makes consistent backups even if the database is being used concurrently. pg_dump does not block other users accessing the database (readers or writers).

pg_dump only dumps a single database. To back up an entire cluster, or to back up global objects that are common to all databases in a cluster (such as roles and tablespaces), use pg_dumpall.

Dumps can be output in script or archive file formats. Script dumps are plain-text files containing the SQL commands required to reconstruct the database to the state it was in at the time it was saved. To restore from such a script, feed it to psql. Script files can be used to reconstruct the database even on other machines and other architectures; with some modifications, even on other SQL database products.

The alternative archive file formats must be used with pg_restore to rebuild the database. They allow pg_restore to be selective about what is restored, or even to reorder the items prior to being restored. The archive file formats are designed to be portable across architectures.

When used with one of the archive file formats and combined with pg_restore, pg_dump provides a flexible archival and transfer mechanism. pg_dump can be used to backup an entire database, then pg_restore can be used to examine the archive and/or select which parts of the database are to be restored. The most flexible output file formats are the “custom” format (-Fc) and the “directory” format (-Fd). They allow for selection and reordering of all archived items, support parallel restoration, and are compressed by default. The “directory” format is the only format that supports parallel dumps.

While running pg_dump, one should examine the output for any warnings (printed on standard error), especially in light of the limitations listed below.

Options

The following command-line options control the content and format of the output.

dbname

Specifies the name of the database to be dumped. If this is not specified, the environment variable PGDATABASE is used. If that is not set, the user name specified for the connection is used.

-a
--data-only

Dump only the data, not the schema (data definitions). Table data, large objects, and sequence values are dumped.

This option is similar to, but for historical reasons not identical to, specifying --section=data.

-b
--blobs

Include large objects in the dump. This is the default behavior except when --schema, --table, or --schema-only is specified. The -b switch is therefore only useful to add large objects to dumps where a specific schema or table has been requested. Note that blobs are considered data and therefore will be included when --data-only is used, but not when --schema-only is.

-B
--no-blobs

Exclude large objects in the dump.

When both -b and -B are given, the behavior is to output large objects, when data is being dumped, see the -b documentation.

-c
--clean

Output commands to clean (drop) database objects prior to outputting the commands for creating them. (Unless --if-exists is also specified, restore might generate some harmless error messages, if any objects were not present in the destination database.)

This option is ignored when emitting an archive (non-text) output file. For the archive formats, you can specify the option when you call pg_restore.

-C
--create

Begin the output with a command to create the database itself and reconnect to the created database. (With a script of this form, it doesn't matter which database in the destination installation you connect to before running the script.) If --clean is also specified, the script drops and recreates the target database before reconnecting to it.

With --create, the output also includes the database's comment if any, and any configuration variable settings that are specific to this database, that is, any ALTER DATABASE ... SET ... and ALTER ROLE ... IN DATABASE ... SET ... commands that mention this database. Access privileges for the database itself are also dumped, unless --no-acl is specified.

This option is ignored when emitting an archive (non-text) output file. For the archive formats, you can specify the option when you call pg_restore.

-e pattern
--extension=pattern

Dump only extensions matching pattern. When this option is not specified, all non-system extensions in the target database will be dumped. Multiple extensions can be selected by writing multiple -e switches. The pattern parameter is interpreted as a pattern according to the same rules used by psql's \d commands (see Patterns), so multiple extensions can also be selected by writing wildcard characters in the pattern. When using wildcards, be careful to quote the pattern if needed to prevent the shell from expanding the wildcards.

Any configuration relation registered by pg_extension_config_dump is included in the dump if its extension is specified by --extension.

Note

When -e is specified, pg_dump makes no attempt to dump any other database objects that the selected extension(s) might depend upon. Therefore, there is no guarantee that the results of a specific-extension dump can be successfully restored by themselves into a clean database.

-E encoding
--encoding=encoding

Create the dump in the specified character set encoding. By default, the dump is created in the database encoding. (Another way to get the same result is to set the PGCLIENTENCODING environment variable to the desired dump encoding.) The supported encodings are described in Section 24.3.1.

-f file
--file=file

Send output to the specified file. This parameter can be omitted for file based output formats, in which case the standard output is used. It must be given for the directory output format however, where it specifies the target directory instead of a file. In this case the directory is created by pg_dump and must not exist before.

-F format
--format=format

Selects the format of the output. format can be one of the following:

p
plain

Output a plain-text SQL script file (the default).

c
custom

Output a custom-format archive suitable for input into pg_restore. Together with the directory output format, this is the most flexible output format in that it allows manual selection and reordering of archived items during restore. This format is also compressed by default.

d
directory

Output a directory-format archive suitable for input into pg_restore. This will create a directory with one file for each table and blob being dumped, plus a so-called Table of Contents file describing the dumped objects in a machine-readable format that pg_restore can read. A directory format archive can be manipulated with standard Unix tools; for example, files in an uncompressed archive can be compressed with the gzip tool. This format is compressed by default and also supports parallel dumps.

t
tar

Output a tar-format archive suitable for input into pg_restore. The tar format is compatible with the directory format: extracting a tar-format archive produces a valid directory-format archive. However, the tar format does not support compression. Also, when using tar format the relative order of table data items cannot be changed during restore.

-j njobs
--jobs=njobs

Run the dump in parallel by dumping njobs tables simultaneously. This option may reduce the time needed to perform the dump but it also increases the load on the database server. You can only use this option with the directory output format because this is the only output format where multiple processes can write their data at the same time.

pg_dump will open njobs + 1 connections to the database, so make sure your max_connections setting is high enough to accommodate all connections.

Requesting exclusive locks on database objects while running a parallel dump could cause the dump to fail. The reason is that the pg_dump leader process requests shared locks on the objects that the worker processes are going to dump later in order to make sure that nobody deletes them and makes them go away while the dump is running. If another client then requests an exclusive lock on a table, that lock will not be granted but will be queued waiting for the shared lock of the leader process to be released. Consequently any other access to the table will not be granted either and will queue after the exclusive lock request. This includes the worker process trying to dump the table. Without any precautions this would be a classic deadlock situation. To detect this conflict, the pg_dump worker process requests another shared lock using the NOWAIT option. If the worker process is not granted this shared lock, somebody else must have requested an exclusive lock in the meantime and there is no way to continue with the dump, so pg_dump has no choice but to abort the dump.

For a consistent backup, the database server needs to support synchronized snapshots, a feature that was introduced in PostgreSQL 9.2 for primary servers and 10 for standbys. With this feature, database clients can ensure they see the same data set even though they use different connections. pg_dump -j uses multiple database connections; it connects to the database once with the leader process and once again for each worker job. Without the synchronized snapshot feature, the different worker jobs wouldn't be guaranteed to see the same data in each connection, which could lead to an inconsistent backup.

If you want to run a parallel dump of a pre-9.2 server, you need to make sure that the database content doesn't change from between the time the leader connects to the database until the last worker job has connected to the database. The easiest way to do this is to halt any data modifying processes (DDL and DML) accessing the database before starting the backup. You also need to specify the --no-synchronized-snapshots parameter when running pg_dump -j against a pre-9.2 PostgreSQL server.

-n pattern
--schema=pattern

Dump only schemas matching pattern; this selects both the schema itself, and all its contained objects. When this option is not specified, all non-system schemas in the target database will be dumped. Multiple schemas can be selected by writing multiple -n switches. The pattern parameter is interpreted as a pattern according to the same rules used by psql's \d commands (see Patterns below), so multiple schemas can also be selected by writing wildcard characters in the pattern. When using wildcards, be careful to quote the pattern if needed to prevent the shell from expanding the wildcards; see Examples below.

Note

When -n is specified, pg_dump makes no attempt to dump any other database objects that the selected schema(s) might depend upon. Therefore, there is no guarantee that the results of a specific-schema dump can be successfully restored by themselves into a clean database.

Note

Non-schema objects such as blobs are not dumped when -n is specified. You can add blobs back to the dump with the --blobs switch.

-N pattern
--exclude-schema=pattern

Do not dump any schemas matching pattern. The pattern is interpreted according to the same rules as for -n. -N can be given more than once to exclude schemas matching any of several patterns.

When both -n and -N are given, the behavior is to dump just the schemas that match at least one -n switch but no -N switches. If -N appears without -n, then schemas matching -N are excluded from what is otherwise a normal dump.

-O
--no-owner

Do not output commands to set ownership of objects to match the original database. By default, pg_dump issues ALTER OWNER or SET SESSION AUTHORIZATION statements to set ownership of created database objects. These statements will fail when the script is run unless it is started by a superuser (or the same user that owns all of the objects in the script). To make a script that can be restored by any user, but will give that user ownership of all the objects, specify -O.

This option is ignored when emitting an archive (non-text) output file. For the archive formats, you can specify the option when you call pg_restore.

-R
--no-reconnect

This option is obsolete but still accepted for backwards compatibility.

-s
--schema-only

Dump only the object definitions (schema), not data.

This option is the inverse of --data-only. It is similar to, but for historical reasons not identical to, specifying --section=pre-data --section=post-data.

(Do not confuse this with the --schema option, which uses the word “schema” in a different meaning.)

To exclude table data for only a subset of tables in the database, see --exclude-table-data.

-S username
--superuser=username

Specify the superuser user name to use when disabling triggers. This is relevant only if --disable-triggers is used. (Usually, it's better to leave this out, and instead start the resulting script as superuser.)

-t pattern
--table=pattern

Dump only tables with names matching pattern. Multiple tables can be selected by writing multiple -t switches. The pattern parameter is interpreted as a pattern according to the same rules used by psql's \d commands (see Patterns below), so multiple tables can also be selected by writing wildcard characters in the pattern. When using wildcards, be careful to quote the pattern if needed to prevent the shell from expanding the wildcards; see Examples below.

As well as tables, this option can be used to dump the definition of matching views, materialized views, foreign tables, and sequences. It will not dump the contents of views or materialized views, and the contents of foreign tables will only be dumped if the corresponding foreign server is specified with --include-foreign-data.

The -n and -N switches have no effect when -t is used, because tables selected by -t will be dumped regardless of those switches, and non-table objects will not be dumped.

Note

When -t is specified, pg_dump makes no attempt to dump any other database objects that the selected table(s) might depend upon. Therefore, there is no guarantee that the results of a specific-table dump can be successfully restored by themselves into a clean database.

-T pattern
--exclude-table=pattern

Do not dump any tables matching pattern. The pattern is interpreted according to the same rules as for -t. -T can be given more than once to exclude tables matching any of several patterns.

When both -t and -T are given, the behavior is to dump just the tables that match at least one -t switch but no -T switches. If -T appears without -t, then tables matching -T are excluded from what is otherwise a normal dump.

-v
--verbose

Specifies verbose mode. This will cause pg_dump to output detailed object comments and start/stop times to the dump file, and progress messages to standard error. Repeating the option causes additional debug-level messages to appear on standard error.

-V
--version

Print the pg_dump version and exit.

-x
--no-privileges
--no-acl

Prevent dumping of access privileges (grant/revoke commands).

-Z 0..9
--compress=0..9

Specify the compression level to use. Zero means no compression. For the custom and directory archive formats, this specifies compression of individual table-data segments, and the default is to compress at a moderate level. For plain text output, setting a nonzero compression level causes the entire output file to be compressed, as though it had been fed through gzip; but the default is not to compress. The tar archive format currently does not support compression at all.

--binary-upgrade

This option is for use by in-place upgrade utilities. Its use for other purposes is not recommended or supported. The behavior of the option may change in future releases without notice.

--column-inserts
--attribute-inserts

Dump data as INSERT commands with explicit column names (INSERT INTO table (column, ...) VALUES ...). This will make restoration very slow; it is mainly useful for making dumps that can be loaded into non-PostgreSQL databases. Any error during reloading will cause only rows that are part of the problematic INSERT to be lost, rather than the entire table contents.

--disable-dollar-quoting

This option disables the use of dollar quoting for function bodies, and forces them to be quoted using SQL standard string syntax.

--disable-triggers

This option is relevant only when creating a data-only dump. It instructs pg_dump to include commands to temporarily disable triggers on the target tables while the data is reloaded. Use this if you have referential integrity checks or other triggers on the tables that you do not want to invoke during data reload.

Presently, the commands emitted for --disable-triggers must be done as superuser. So, you should also specify a superuser name with -S, or preferably be careful to start the resulting script as a superuser.

This option is ignored when emitting an archive (non-text) output file. For the archive formats, you can specify the option when you call pg_restore.

--enable-row-security

This option is relevant only when dumping the contents of a table which has row security. By default, pg_dump will set row_security to off, to ensure that all data is dumped from the table. If the user does not have sufficient privileges to bypass row security, then an error is thrown. This parameter instructs pg_dump to set row_security to on instead, allowing the user to dump the parts of the contents of the table that they have access to.

Note that if you use this option currently, you probably also want the dump be in INSERT format, as the COPY FROM during restore does not support row security.

--exclude-table-data=pattern

Do not dump data for any tables matching pattern. The pattern is interpreted according to the same rules as for -t. --exclude-table-data can be given more than once to exclude tables matching any of several patterns. This option is useful when you need the definition of a particular table even though you do not need the data in it.

To exclude data for all tables in the database, see --schema-only.

--extra-float-digits=ndigits

Use the specified value of extra_float_digits when dumping floating-point data, instead of the maximum available precision. Routine dumps made for backup purposes should not use this option.

--if-exists

Use conditional commands (i.e., add an IF EXISTS clause) when cleaning database objects. This option is not valid unless --clean is also specified.

--include-foreign-data=foreignserver

Dump the data for any foreign table with a foreign server matching foreignserver pattern. Multiple foreign servers can be selected by writing multiple --include-foreign-data switches. Also, the foreignserver parameter is interpreted as a pattern according to the same rules used by psql's \d commands (see Patterns below), so multiple foreign servers can also be selected by writing wildcard characters in the pattern. When using wildcards, be careful to quote the pattern if needed to prevent the shell from expanding the wildcards; see Examples below. The only exception is that an empty pattern is disallowed.

Note

When --include-foreign-data is specified, pg_dump does not check that the foreign table is writable. Therefore, there is no guarantee that the results of a foreign table dump can be successfully restored.

--inserts

Dump data as INSERT commands (rather than COPY). This will make restoration very slow; it is mainly useful for making dumps that can be loaded into non-PostgreSQL databases. Any error during reloading will cause only rows that are part of the problematic INSERT to be lost, rather than the entire table contents. Note that the restore might fail altogether if you have rearranged column order. The --column-inserts option is safe against column order changes, though even slower.

--load-via-partition-root

When dumping data for a table partition, make the COPY or INSERT statements target the root of the partitioning hierarchy that contains it, rather than the partition itself. This causes the appropriate partition to be re-determined for each row when the data is loaded. This may be useful when reloading data on a server where rows do not always fall into the same partitions as they did on the original server. That could happen, for example, if the partitioning column is of type text and the two systems have different definitions of the collation used to sort the partitioning column.

It is best not to use parallelism when restoring from an archive made with this option, because pg_restore will not know exactly which partition(s) a given archive data item will load data into. This could result in inefficiency due to lock conflicts between parallel jobs, or perhaps even reload failures due to foreign key constraints being set up before all the relevant data is loaded.

--lock-wait-timeout=timeout

Do not wait forever to acquire shared table locks at the beginning of the dump. Instead fail if unable to lock a table within the specified timeout. The timeout may be specified in any of the formats accepted by SET statement_timeout. (Allowed formats vary depending on the server version you are dumping from, but an integer number of milliseconds is accepted by all versions.)

--no-comments

Do not dump comments.

--no-publications

Do not dump publications.

--no-security-labels

Do not dump security labels.

--no-subscriptions

Do not dump subscriptions.

--no-sync

By default, pg_dump will wait for all files to be written safely to disk. This option causes pg_dump to return without waiting, which is faster, but means that a subsequent operating system crash can leave the dump corrupt. Generally, this option is useful for testing but should not be used when dumping data from production installation.

--no-synchronized-snapshots

This option allows running pg_dump -j against a pre-9.2 server, see the documentation of the -j parameter for more details.

--no-tablespaces

Do not output commands to select tablespaces. With this option, all objects will be created in whichever tablespace is the default during restore.

This option is ignored when emitting an archive (non-text) output file. For the archive formats, you can specify the option when you call pg_restore.

--no-toast-compression

Do not output commands to set TOAST compression methods. With this option, all columns will be restored with the default compression setting.

--no-unlogged-table-data

Do not dump the contents of unlogged tables. This option has no effect on whether or not the table definitions (schema) are dumped; it only suppresses dumping the table data. Data in unlogged tables is always excluded when dumping from a standby server.

--on-conflict-do-nothing

Add ON CONFLICT DO NOTHING to INSERT commands. This option is not valid unless --inserts, --column-inserts or --rows-per-insert is also specified.

--quote-all-identifiers

Force quoting of all identifiers. This option is recommended when dumping a database from a server whose PostgreSQL major version is different from pg_dump's, or when the output is intended to be loaded into a server of a different major version. By default, pg_dump quotes only identifiers that are reserved words in its own major version. This sometimes results in compatibility issues when dealing with servers of other versions that may have slightly different sets of reserved words. Using --quote-all-identifiers prevents such issues, at the price of a harder-to-read dump script.

--rows-per-insert=nrows

Dump data as INSERT commands (rather than COPY). Controls the maximum number of rows per INSERT command. The value specified must be a number greater than zero. Any error during reloading will cause only rows that are part of the problematic INSERT to be lost, rather than the entire table contents.

--section=sectionname

Only dump the named section. The section name can be pre-data, data, or post-data. This option can be specified more than once to select multiple sections. The default is to dump all sections.

The data section contains actual table data, large-object contents, and sequence values. Post-data items include definitions of indexes, triggers, rules, and constraints other than validated check constraints. Pre-data items include all other data definition items.

--serializable-deferrable

Use a serializable transaction for the dump, to ensure that the snapshot used is consistent with later database states; but do this by waiting for a point in the transaction stream at which no anomalies can be present, so that there isn't a risk of the dump failing or causing other transactions to roll back with a serialization_failure. See Chapter 13 for more information about transaction isolation and concurrency control.

This option is not beneficial for a dump which is intended only for disaster recovery. It could be useful for a dump used to load a copy of the database for reporting or other read-only load sharing while the original database continues to be updated. Without it the dump may reflect a state which is not consistent with any serial execution of the transactions eventually committed. For example, if batch processing techniques are used, a batch may show as closed in the dump without all of the items which are in the batch appearing.

This option will make no difference if there are no read-write transactions active when pg_dump is started. If read-write transactions are active, the start of the dump may be delayed for an indeterminate length of time. Once running, performance with or without the switch is the same.

--snapshot=snapshotname

Use the specified synchronized snapshot when making a dump of the database (see Table 9.90 for more details).

This option is useful when needing to synchronize the dump with a logical replication slot (see Chapter 49) or with a concurrent session.

In the case of a parallel dump, the snapshot name defined by this option is used rather than taking a new snapshot.

--strict-names

Require that each extension (-e/--extension), schema (-n/--schema) and table (-t/--table) qualifier match at least one extension/schema/table in the database to be dumped. Note that if none of the extension/schema/table qualifiers find matches, pg_dump will generate an error even without --strict-names.

This option has no effect on -N/--exclude-schema, -T/--exclude-table, or --exclude-table-data. An exclude pattern failing to match any objects is not considered an error.

--use-set-session-authorization

Output SQL-standard SET SESSION AUTHORIZATION commands instead of ALTER OWNER commands to determine object ownership. This makes the dump more standards-compatible, but depending on the history of the objects in the dump, might not restore properly. Also, a dump using SET SESSION AUTHORIZATION will certainly require superuser privileges to restore correctly, whereas ALTER OWNER requires lesser privileges.

-?
--help

Show help about pg_dump command line arguments, and exit.

The following command-line options control the database connection parameters.

-d dbname
--dbname=dbname

Specifies the name of the database to connect to. This is equivalent to specifying dbname as the first non-option argument on the command line. The dbname can be a connection string. If so, connection string parameters will override any conflicting command line options.

-h host
--host=host

Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for the Unix domain socket. The default is taken from the PGHOST environment variable, if set, else a Unix domain socket connection is attempted.

-p port
--port=port

Specifies the TCP port or local Unix domain socket file extension on which the server is listening for connections. Defaults to the PGPORT environment variable, if set, or a compiled-in default.

-U username
--username=username

User name to connect as.

-w
--no-password

Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.

-W
--password

Force pg_dump to prompt for a password before connecting to a database.

This option is never essential, since pg_dump will automatically prompt for a password if the server demands password authentication. However, pg_dump will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt.

--role=rolename

Specifies a role name to be used to create the dump. This option causes pg_dump to issue a SET ROLE rolename command after connecting to the database. It is useful when the authenticated user (specified by -U) lacks privileges needed by pg_dump, but can switch to a role with the required rights. Some installations have a policy against logging in directly as a superuser, and use of this option allows dumps to be made without violating the policy.

Environment

PGDATABASE
PGHOST
PGOPTIONS
PGPORT
PGUSER

Default connection parameters.

PG_COLOR

Specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

This utility, like most other PostgreSQL utilities, also uses the environment variables supported by libpq (see Section 34.15).

Diagnostics

pg_dump internally executes SELECT statements. If you have problems running pg_dump, make sure you are able to select information from the database using, for example, psql. Also, any default connection settings and environment variables used by the libpq front-end library will apply.

The database activity of pg_dump is normally collected by the statistics collector. If this is undesirable, you can set parameter track_counts to false via PGOPTIONS or the ALTER USER command.

Notes

If your database cluster has any local additions to the template1 database, be careful to restore the output of pg_dump into a truly empty database; otherwise you are likely to get errors due to duplicate definitions of the added objects. To make an empty database without any local additions, copy from template0 not template1, for example:

CREATE DATABASE foo WITH TEMPLATE template0;

When a data-only dump is chosen and the option --disable-triggers is used, pg_dump emits commands to disable triggers on user tables before inserting the data, and then commands to re-enable them after the data has been inserted. If the restore is stopped in the middle, the system catalogs might be left in the wrong state.

The dump file produced by pg_dump does not contain the statistics used by the optimizer to make query planning decisions. Therefore, it is wise to run ANALYZE after restoring from a dump file to ensure optimal performance; see Section 25.1.3 and Section 25.1.6 for more information.

Because pg_dump is used to transfer data to newer versions of PostgreSQL, the output of pg_dump can be expected to load into PostgreSQL server versions newer than pg_dump's version. pg_dump can also dump from PostgreSQL servers older than its own version. (Currently, servers back to version 8.0 are supported.) However, pg_dump cannot dump from PostgreSQL servers newer than its own major version; it will refuse to even try, rather than risk making an invalid dump. Also, it is not guaranteed that pg_dump's output can be loaded into a server of an older major version — not even if the dump was taken from a server of that version. Loading a dump file into an older server may require manual editing of the dump file to remove syntax not understood by the older server. Use of the --quote-all-identifiers option is recommended in cross-version cases, as it can prevent problems arising from varying reserved-word lists in different PostgreSQL versions.

When dumping logical replication subscriptions, pg_dump will generate CREATE SUBSCRIPTION commands that use the connect = false option, so that restoring the subscription does not make remote connections for creating a replication slot or for initial table copy. That way, the dump can be restored without requiring network access to the remote servers. It is then up to the user to reactivate the subscriptions in a suitable way. If the involved hosts have changed, the connection information might have to be changed. It might also be appropriate to truncate the target tables before initiating a new full table copy.

Examples

To dump a database called mydb into an SQL-script file:

$ pg_dump mydb > db.sql

To reload such a script into a (freshly created) database named newdb:

$ psql -d newdb -f db.sql

To dump a database into a custom-format archive file:

$ pg_dump -Fc mydb > db.dump

To dump a database into a directory-format archive:

$ pg_dump -Fd mydb -f dumpdir

To dump a database into a directory-format archive in parallel with 5 worker jobs:

$ pg_dump -Fd mydb -j 5 -f dumpdir

To reload an archive file into a (freshly created) database named newdb:

$ pg_restore -d newdb db.dump

To reload an archive file into the same database it was dumped from, discarding the current contents of that database:

$ pg_restore -d postgres --clean --create db.dump

To dump a single table named mytab:

$ pg_dump -t mytab mydb > db.sql

To dump all tables whose names start with emp in the detroit schema, except for the table named employee_log:

$ pg_dump -t 'detroit.emp*' -T detroit.employee_log mydb > db.sql

To dump all schemas whose names start with east or west and end in gsm, excluding any schemas whose names contain the word test:

$ pg_dump -n 'east*gsm' -n 'west*gsm' -N '*test*' mydb > db.sql

The same, using regular expression notation to consolidate the switches:

$ pg_dump -n '(east|west)*gsm' -N '*test*' mydb > db.sql

To dump all database objects except for tables whose names begin with ts_:

$ pg_dump -T 'ts_*' mydb > db.sql

To specify an upper-case or mixed-case name in -t and related switches, you need to double-quote the name; else it will be folded to lower case (see Patterns below). But double quotes are special to the shell, so in turn they must be quoted. Thus, to dump a single table with a mixed-case name, you need something like

$ pg_dump -t "\"MixedCaseName\"" mydb > mytab.sql

See Also

pg_dumpall, pg_restore, psql
Prev | Up | Next
pg_config | Home | pg_dumpall


pg_dumpall
Prev Up PostgreSQL Client Applications Home Next

pg_dumpall

pg_dumpall — extract a PostgreSQL database cluster into a script file

Synopsis

pg_dumpall [connection-option...] [option...]

Description

pg_dumpall is a utility for writing out (“dumping”) all PostgreSQL databases of a cluster into one script file. The script file contains SQL commands that can be used as input to psql to restore the databases. It does this by calling pg_dump for each database in the cluster. pg_dumpall also dumps global objects that are common to all databases, that is, database roles and tablespaces. (pg_dump does not save these objects.)

Since pg_dumpall reads tables from all databases you will most likely have to connect as a database superuser in order to produce a complete dump. Also you will need superuser privileges to execute the saved script in order to be allowed to add roles and create databases.

The SQL script will be written to the standard output. Use the -f/--file option or shell operators to redirect it into a file.

pg_dumpall needs to connect several times to the PostgreSQL server (once per database). If you use password authentication it will ask for a password each time. It is convenient to have a ~/.pgpass file in such cases. See Section 34.16 for more information.

Options

The following command-line options control the content and format of the output.

-a
--data-only

Dump only the data, not the schema (data definitions).

-c
--clean

Include SQL commands to clean (drop) databases before recreating them. DROP commands for roles and tablespaces are added as well.

-E encoding
--encoding=encoding

Create the dump in the specified character set encoding. By default, the dump is created in the database encoding. (Another way to get the same result is to set the PGCLIENTENCODING environment variable to the desired dump encoding.)

-f filename
--file=filename

Send output to the specified file. If this is omitted, the standard output is used.

-g
--globals-only

Dump only global objects (roles and tablespaces), no databases.

-O
--no-owner

Do not output commands to set ownership of objects to match the original database. By default, pg_dumpall issues ALTER OWNER or SET SESSION AUTHORIZATION statements to set ownership of created schema elements. These statements will fail when the script is run unless it is started by a superuser (or the same user that owns all of the objects in the script). To make a script that can be restored by any user, but will give that user ownership of all the objects, specify -O.

-r
--roles-only

Dump only roles, no databases or tablespaces.

-s
--schema-only

Dump only the object definitions (schema), not data.

-S username
--superuser=username

Specify the superuser user name to use when disabling triggers. This is relevant only if --disable-triggers is used. (Usually, it's better to leave this out, and instead start the resulting script as superuser.)

-t
--tablespaces-only

Dump only tablespaces, no databases or roles.

-v
--verbose

Specifies verbose mode. This will cause pg_dumpall to output start/stop times to the dump file, and progress messages to standard error. Repeating the option causes additional debug-level messages to appear on standard error. The option is also passed down to pg_dump.

-V
--version

Print the pg_dumpall version and exit.

-x
--no-privileges
--no-acl

Prevent dumping of access privileges (grant/revoke commands).

--binary-upgrade

This option is for use by in-place upgrade utilities. Its use for other purposes is not recommended or supported. The behavior of the option may change in future releases without notice.

--column-inserts
--attribute-inserts

Dump data as INSERT commands with explicit column names (INSERT INTO table (column, ...) VALUES ...). This will make restoration very slow; it is mainly useful for making dumps that can be loaded into non-PostgreSQL databases.

--disable-dollar-quoting

This option disables the use of dollar quoting for function bodies, and forces them to be quoted using SQL standard string syntax.

--disable-triggers

This option is relevant only when creating a data-only dump. It instructs pg_dumpall to include commands to temporarily disable triggers on the target tables while the data is reloaded. Use this if you have referential integrity checks or other triggers on the tables that you do not want to invoke during data reload.

Presently, the commands emitted for --disable-triggers must be done as superuser. So, you should also specify a superuser name with -S, or preferably be careful to start the resulting script as a superuser.

--exclude-database=pattern

Do not dump databases whose name matches pattern. Multiple patterns can be excluded by writing multiple --exclude-database switches. The pattern parameter is interpreted as a pattern according to the same rules used by psql's \d commands (see Patterns below), so multiple databases can also be excluded by writing wildcard characters in the pattern. When using wildcards, be careful to quote the pattern if needed to prevent shell wildcard expansion.

--extra-float-digits=ndigits

Use the specified value of extra_float_digits when dumping floating-point data, instead of the maximum available precision. Routine dumps made for backup purposes should not use this option.

--if-exists

Use conditional commands (i.e., add an IF EXISTS clause) to drop databases and other objects. This option is not valid unless --clean is also specified.

--inserts

Dump data as INSERT commands (rather than COPY). This will make restoration very slow; it is mainly useful for making dumps that can be loaded into non-PostgreSQL databases. Note that the restore might fail altogether if you have rearranged column order. The --column-inserts option is safer, though even slower.

--load-via-partition-root

When dumping data for a table partition, make the COPY or INSERT statements target the root of the partitioning hierarchy that contains it, rather than the partition itself. This causes the appropriate partition to be re-determined for each row when the data is loaded. This may be useful when reloading data on a server where rows do not always fall into the same partitions as they did on the original server. That could happen, for example, if the partitioning column is of type text and the two systems have different definitions of the collation used to sort the partitioning column.

--lock-wait-timeout=timeout

Do not wait forever to acquire shared table locks at the beginning of the dump. Instead, fail if unable to lock a table within the specified timeout. The timeout may be specified in any of the formats accepted by SET statement_timeout.

--no-comments

Do not dump comments.

--no-publications

Do not dump publications.

--no-role-passwords

Do not dump passwords for roles. When restored, roles will have a null password, and password authentication will always fail until the password is set. Since password values aren't needed when this option is specified, the role information is read from the catalog view pg_roles instead of pg_authid. Therefore, this option also helps if access to pg_authid is restricted by some security policy.

--no-security-labels

Do not dump security labels.

--no-subscriptions

Do not dump subscriptions.

--no-sync

By default, pg_dumpall will wait for all files to be written safely to disk. This option causes pg_dumpall to return without waiting, which is faster, but means that a subsequent operating system crash can leave the dump corrupt. Generally, this option is useful for testing but should not be used when dumping data from production installation.

--no-tablespaces

Do not output commands to create tablespaces nor select tablespaces for objects. With this option, all objects will be created in whichever tablespace is the default during restore.

--no-toast-compression

Do not output commands to set TOAST compression methods. With this option, all columns will be restored with the default compression setting.

--no-unlogged-table-data

Do not dump the contents of unlogged tables. This option has no effect on whether or not the table definitions (schema) are dumped; it only suppresses dumping the table data.

--on-conflict-do-nothing

Add ON CONFLICT DO NOTHING to INSERT commands. This option is not valid unless --inserts or --column-inserts is also specified.

--quote-all-identifiers

Force quoting of all identifiers. This option is recommended when dumping a database from a server whose PostgreSQL major version is different from pg_dumpall's, or when the output is intended to be loaded into a server of a different major version. By default, pg_dumpall quotes only identifiers that are reserved words in its own major version. This sometimes results in compatibility issues when dealing with servers of other versions that may have slightly different sets of reserved words. Using --quote-all-identifiers prevents such issues, at the price of a harder-to-read dump script.

--rows-per-insert=nrows

Dump data as INSERT commands (rather than COPY). Controls the maximum number of rows per INSERT command. The value specified must be a number greater than zero. Any error during reloading will cause only rows that are part of the problematic INSERT to be lost, rather than the entire table contents.

--use-set-session-authorization

Output SQL-standard SET SESSION AUTHORIZATION commands instead of ALTER OWNER commands to determine object ownership. This makes the dump more standards compatible, but depending on the history of the objects in the dump, might not restore properly.

-?
--help

Show help about pg_dumpall command line arguments, and exit.

The following command-line options control the database connection parameters.

-d connstr
--dbname=connstr

Specifies parameters used to connect to the server, as a connection string; these will override any conflicting command line options.

The option is called --dbname for consistency with other client applications, but because pg_dumpall needs to connect to many databases, the database name in the connection string will be ignored. Use the -l option to specify the name of the database used for the initial connection, which will dump global objects and discover what other databases should be dumped.

-h host
--host=host

Specifies the host name of the machine on which the database server is running. If the value begins with a slash, it is used as the directory for the Unix domain socket. The default is taken from the PGHOST environment variable, if set, else a Unix domain socket connection is attempted.

-l dbname
--database=dbname

Specifies the name of the database to connect to for dumping global objects and discovering what other databases should be dumped. If not specified, the postgres database will be used, and if that does not exist, template1 will be used.

-p port
--port=port

Specifies the TCP port or local Unix domain socket file extension on which the server is listening for connections. Defaults to the PGPORT environment variable, if set, or a compiled-in default.

-U username
--username=username

User name to connect as.

-w
--no-password

Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.

-W
--password

Force pg_dumpall to prompt for a password before connecting to a database.

This option is never essential, since pg_dumpall will automatically prompt for a password if the server demands password authentication. However, pg_dumpall will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt.

Note that the password prompt will occur again for each database to be dumped. Usually, it's better to set up a ~/.pgpass file than to rely on manual password entry.

--role=rolename

Specifies a role name to be used to create the dump. This option causes pg_dumpall to issue a SET ROLE rolename command after connecting to the database. It is useful when the authenticated user (specified by -U) lacks privileges needed by pg_dumpall, but can switch to a role with the required rights. Some installations have a policy against logging in directly as a superuser, and use of this option allows dumps to be made without violating the policy.

Environment

PGHOST
PGOPTIONS
PGPORT
PGUSER

Default connection parameters

PG_COLOR

Specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

This utility, like most other PostgreSQL utilities, also uses the environment variables supported by libpq (see Section 34.15).

Notes

Since pg_dumpall calls pg_dump internally, some diagnostic messages will refer to pg_dump.

The --clean option can be useful even when your intention is to restore the dump script into a fresh cluster. Use of --clean authorizes the script to drop and re-create the built-in postgres and template1 databases, ensuring that those databases will retain the same properties (for instance, locale and encoding) that they had in the source cluster. Without the option, those databases will retain their existing database-level properties, as well as any pre-existing contents.

Once restored, it is wise to run ANALYZE on each database so the optimizer has useful statistics. You can also run vacuumdb -a -z to analyze all databases.

The dump script should not be expected to run completely without errors. In particular, because the script will issue CREATE ROLE for every role existing in the source cluster, it is certain to get a “role already exists” error for the bootstrap superuser, unless the destination cluster was initialized with a different bootstrap superuser name. This error is harmless and should be ignored. Use of the --clean option is likely to produce additional harmless error messages about non-existent objects, although you can minimize those by adding --if-exists.

pg_dumpall requires all needed tablespace directories to exist before the restore; otherwise, database creation will fail for databases in non-default locations.

Examples

To dump all databases:

$ pg_dumpall > db.out

To reload database(s) from this file, you can use:

$ psql -f db.out postgres

It is not important to which database you connect here since the script file created by pg_dumpall will contain the appropriate commands to create and connect to the saved databases. An exception is that if you specified --clean, you must connect to the postgres database initially; the script will attempt to drop other databases immediately, and that will fail for the database you are connected to.

See Also

Check pg_dump for details on possible error conditions.

Prev | Up | Next
pg_dump | Home | pg_isready


pg_isready
Prev Up PostgreSQL Client Applications Home Next

pg_isready

pg_isready — check the connection status of a PostgreSQL server

Synopsis

pg_isready [connection-option...] [option...]

Description

pg_isready is a utility for checking the connection status of a PostgreSQL database server. The exit status specifies the result of the connection check.

Options

-d dbname
--dbname=dbname

Specifies the name of the database to connect to. The dbname can be a connection string. If so, connection string parameters will override any conflicting command line options.

-h hostname
--host=hostname

Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for the Unix-domain socket.

-p port
--port=port

Specifies the TCP port or the local Unix-domain socket file extension on which the server is listening for connections. Defaults to the value of the PGPORT environment variable or, if not set, to the port specified at compile time, usually 5432.

-q
--quiet

Do not display status message. This is useful when scripting.

-t seconds
--timeout=seconds

The maximum number of seconds to wait when attempting connection before returning that the server is not responding. Setting to 0 disables. The default is 3 seconds.

-U username
--username=username

Connect to the database as the user username instead of the default.

-V
--version

Print the pg_isready version and exit.

-?
--help

Show help about pg_isready command line arguments, and exit.

Exit Status

pg_isready returns 0 to the shell if the server is accepting connections normally, 1 if the server is rejecting connections (for example during startup), 2 if there was no response to the connection attempt, and 3 if no attempt was made (for example due to invalid parameters).

Environment

pg_isready, like most other PostgreSQL utilities, also uses the environment variables supported by libpq (see Section 34.15).

The environment variable PG_COLOR specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

Notes

It is not necessary to supply correct user name, password, or database name values to obtain the server status; however, if incorrect values are provided, the server will log a failed connection attempt.

Examples

Standard Usage:

$ pg_isready
/tmp:5432 - accepting connections
$ echo $?
0

Running with connection parameters to a PostgreSQL cluster in startup:

$ pg_isready -h localhost -p 5433
localhost:5433 - rejecting connections
$ echo $?
1

Running with connection parameters to a non-responsive PostgreSQL cluster:

$ pg_isready -h someremotehost
someremotehost:5432 - no response
$ echo $?
2

Prev | Up | Next
pg_dumpall | Home | pg_receivewal


pg_receivewal
Prev Up PostgreSQL Client Applications Home Next

pg_receivewal

pg_receivewal — stream write-ahead logs from a PostgreSQL server

Synopsis

pg_receivewal [option...]

Description

pg_receivewal is used to stream the write-ahead log from a running PostgreSQL cluster. The write-ahead log is streamed using the streaming replication protocol, and is written to a local directory of files. This directory can be used as the archive location for doing a restore using point-in-time recovery (see Section 26.3).

pg_receivewal streams the write-ahead log in real time as it's being generated on the server, and does not wait for segments to complete like archive_command does. For this reason, it is not necessary to set archive_timeout when using pg_receivewal.

Unlike the WAL receiver of a PostgreSQL standby server, pg_receivewal by default flushes WAL data only when a WAL file is closed. The option --synchronous must be specified to flush WAL data in real time. Since pg_receivewal does not apply WAL, you should not allow it to become a synchronous standby when synchronous_commit equals remote_apply. If it does, it will appear to be a standby that never catches up, and will cause transaction commits to block. To avoid this, you should either configure an appropriate value for synchronous_standby_names, or specify application_name for pg_receivewal that does not match it, or change the value of synchronous_commit to something other than remote_apply.

The write-ahead log is streamed over a regular PostgreSQL connection and uses the replication protocol. The connection must be made with a user having REPLICATION permissions (see Section 22.2) or a superuser, and pg_hba.conf must permit the replication connection. The server must also be configured with max_wal_senders set high enough to leave at least one session available for the stream.

The starting point of the write-ahead log streaming is calculated when pg_receivewal starts:

  1. First, scan the directory where the WAL segment files are written and find the newest completed segment file, using as the starting point the beginning of the next WAL segment file.

  2. If a starting point cannot be calculated with the previous method, the latest WAL flush location is used as reported by the server from an IDENTIFY_SYSTEM command.

If the connection is lost, or if it cannot be initially established, with a non-fatal error, pg_receivewal will retry the connection indefinitely, and reestablish streaming as soon as possible. To avoid this behavior, use the -n parameter.

In the absence of fatal errors, pg_receivewal will run until terminated by the SIGINT signal (Control+C).

Options

-D directory
--directory=directory

Directory to write the output to.

This parameter is required.

-E lsn
--endpos=lsn

Automatically stop replication and exit with normal exit status 0 when receiving reaches the specified LSN.

If there is a record with LSN exactly equal to lsn, the record will be processed.

--if-not-exists

Do not error out when --create-slot is specified and a slot with the specified name already exists.

-n
--no-loop

Don't loop on connection errors. Instead, exit right away with an error.

--no-sync

This option causes pg_receivewal to not force WAL data to be flushed to disk. This is faster, but means that a subsequent operating system crash can leave the WAL segments corrupt. Generally, this option is useful for testing but should not be used when doing WAL archiving on a production deployment.

This option is incompatible with --synchronous.

-s interval
--status-interval=interval

Specifies the number of seconds between status packets sent back to the server. This allows for easier monitoring of the progress from server. A value of zero disables the periodic status updates completely, although an update will still be sent when requested by the server, to avoid timeout disconnect. The default value is 10 seconds.

-S slotname
--slot=slotname

Require pg_receivewal to use an existing replication slot (see Section 27.2.6). When this option is used, pg_receivewal will report a flush position to the server, indicating when each segment has been synchronized to disk so that the server can remove that segment if it is not otherwise needed.

When the replication client of pg_receivewal is configured on the server as a synchronous standby, then using a replication slot will report the flush position to the server, but only when a WAL file is closed. Therefore, that configuration will cause transactions on the primary to wait for a long time and effectively not work satisfactorily. The option --synchronous (see below) must be specified in addition to make this work correctly.

--synchronous

Flush the WAL data to disk immediately after it has been received. Also send a status packet back to the server immediately after flushing, regardless of --status-interval.

This option should be specified if the replication client of pg_receivewal is configured on the server as a synchronous standby, to ensure that timely feedback is sent to the server.

-v
--verbose

Enables verbose mode.

-Z level
--compress=level

Enables gzip compression of write-ahead logs, and specifies the compression level (0 through 9, 0 being no compression and 9 being best compression). The suffix .gz will automatically be added to all filenames.

The following command-line options control the database connection parameters.

-d connstr
--dbname=connstr

Specifies parameters used to connect to the server, as a connection string; these will override any conflicting command line options.

The option is called --dbname for consistency with other client applications, but because pg_receivewal doesn't connect to any particular database in the cluster, database name in the connection string will be ignored.

-h host
--host=host

Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for the Unix domain socket. The default is taken from the PGHOST environment variable, if set, else a Unix domain socket connection is attempted.

-p port
--port=port

Specifies the TCP port or local Unix domain socket file extension on which the server is listening for connections. Defaults to the PGPORT environment variable, if set, or a compiled-in default.

-U username
--username=username

User name to connect as.

-w
--no-password

Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.

-W
--password

Force pg_receivewal to prompt for a password before connecting to a database.

This option is never essential, since pg_receivewal will automatically prompt for a password if the server demands password authentication. However, pg_receivewal will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt.

pg_receivewal can perform one of the two following actions in order to control physical replication slots:

--create-slot

Create a new physical replication slot with the name specified in --slot, then exit.

--drop-slot

Drop the replication slot with the name specified in --slot, then exit.

Other options are also available:

-V
--version

Print the pg_receivewal version and exit.

-?
--help

Show help about pg_receivewal command line arguments, and exit.

Exit Status

pg_receivewal will exit with status 0 when terminated by the SIGINT signal. (That is the normal way to end it. Hence it is not an error.) For fatal errors or other signals, the exit status will be nonzero.

Environment

This utility, like most other PostgreSQL utilities, uses the environment variables supported by libpq (see Section 34.15).

The environment variable PG_COLOR specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

Notes

When using pg_receivewal instead of archive_command as the main WAL backup method, it is strongly recommended to use replication slots. Otherwise, the server is free to recycle or remove write-ahead log files before they are backed up, because it does not have any information, either from archive_command or the replication slots, about how far the WAL stream has been archived. Note, however, that a replication slot will fill up the server's disk space if the receiver does not keep up with fetching the WAL data.

pg_receivewal will preserve group permissions on the received WAL files if group permissions are enabled on the source cluster.

Examples

To stream the write-ahead log from the server at mydbserver and store it in the local directory /usr/local/pgsql/archive:

$ pg_receivewal -h mydbserver -D /usr/local/pgsql/archive

See Also

pg_basebackup
Prev | Up | Next
pg_isready | Home | pg_recvlogical


pg_recvlogical
Prev Up PostgreSQL Client Applications Home Next

pg_recvlogical

pg_recvlogical — control PostgreSQL logical decoding streams

Synopsis

pg_recvlogical [option...]

Description

pg_recvlogical controls logical decoding replication slots and streams data from such replication slots.

It creates a replication-mode connection, so it is subject to the same constraints as pg_receivewal, plus those for logical replication (see Chapter 49).

pg_recvlogical has no equivalent to the logical decoding SQL interface's peek and get modes. It sends replay confirmations for data lazily as it receives it and on clean exit. To examine pending data on a slot without consuming it, use pg_logical_slot_peek_changes.

Options

At least one of the following options must be specified to select an action:

--create-slot

Create a new logical replication slot with the name specified by --slot, using the output plugin specified by --plugin, for the database specified by --dbname.

--drop-slot

Drop the replication slot with the name specified by --slot, then exit.

--start

Begin streaming changes from the logical replication slot specified by --slot, continuing until terminated by a signal. If the server side change stream ends with a server shutdown or disconnect, retry in a loop unless --no-loop is specified.

The stream format is determined by the output plugin specified when the slot was created.

The connection must be to the same database used to create the slot.

--create-slot and --start can be specified together. --drop-slot cannot be combined with another action.

The following command-line options control the location and format of the output and other replication behavior:

-E lsn
--endpos=lsn

In --start mode, automatically stop replication and exit with normal exit status 0 when receiving reaches the specified LSN. If specified when not in --start mode, an error is raised.

If there's a record with LSN exactly equal to lsn, the record will be output.

The --endpos option is not aware of transaction boundaries and may truncate output partway through a transaction. Any partially output transaction will not be consumed and will be replayed again when the slot is next read from. Individual messages are never truncated.

-f filename
--file=filename

Write received and decoded transaction data into this file. Use - for stdout.

-F interval_seconds
--fsync-interval=interval_seconds

Specifies how often pg_recvlogical should issue fsync() calls to ensure the output file is safely flushed to disk.

The server will occasionally request the client to perform a flush and report the flush position to the server. This setting is in addition to that, to perform flushes more frequently.

Specifying an interval of 0 disables issuing fsync() calls altogether, while still reporting progress to the server. In this case, data could be lost in the event of a crash.

-I lsn
--startpos=lsn

In --start mode, start replication from the given LSN. For details on the effect of this, see the documentation in Chapter 49 and Section 53.4. Ignored in other modes.

--if-not-exists

Do not error out when --create-slot is specified and a slot with the specified name already exists.

-n
--no-loop

When the connection to the server is lost, do not retry in a loop, just exit.

-o name[=value]
--option=name[=value]

Pass the option name to the output plugin with, if specified, the option value value. Which options exist and their effects depends on the used output plugin.

-P plugin
--plugin=plugin

When creating a slot, use the specified logical decoding output plugin. See Chapter 49. This option has no effect if the slot already exists.

-s interval_seconds
--status-interval=interval_seconds

This option has the same effect as the option of the same name in pg_receivewal. See the description there.

-S slot_name
--slot=slot_name

In --start mode, use the existing logical replication slot named slot_name. In --create-slot mode, create the slot with this name. In --drop-slot mode, delete the slot with this name.

-v
--verbose

Enables verbose mode.

The following command-line options control the database connection parameters.

-d dbname
--dbname=dbname

The database to connect to. See the description of the actions for what this means in detail. The dbname can be a connection string. If so, connection string parameters will override any conflicting command line options. Defaults to the user name.

-h hostname-or-ip
--host=hostname-or-ip

Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for the Unix domain socket. The default is taken from the PGHOST environment variable, if set, else a Unix domain socket connection is attempted.

-p port
--port=port

Specifies the TCP port or local Unix domain socket file extension on which the server is listening for connections. Defaults to the PGPORT environment variable, if set, or a compiled-in default.

-U user
--username=user

User name to connect as. Defaults to current operating system user name.

-w
--no-password

Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.

-W
--password

Force pg_recvlogical to prompt for a password before connecting to a database.

This option is never essential, since pg_recvlogical will automatically prompt for a password if the server demands password authentication. However, pg_recvlogical will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt.

The following additional options are available:

-V
--version

Print the pg_recvlogical version and exit.

-?
--help

Show help about pg_recvlogical command line arguments, and exit.

Environment

This utility, like most other PostgreSQL utilities, uses the environment variables supported by libpq (see Section 34.15).

The environment variable PG_COLOR specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

Notes

pg_recvlogical will preserve group permissions on the received WAL files if group permissions are enabled on the source cluster.

Examples

See Section 49.1 for an example.

See Also

pg_receivewal
Prev | Up | Next
pg_receivewal | Home | pg_restore


pg_restore
Prev Up PostgreSQL Client Applications Home Next

pg_restore

pg_restore — restore a PostgreSQL database from an archive file created by pg_dump

Synopsis

pg_restore [connection-option...] [option...] [filename]

Description

pg_restore is a utility for restoring a PostgreSQL database from an archive created by pg_dump in one of the non-plain-text formats. It will issue the commands necessary to reconstruct the database to the state it was in at the time it was saved. The archive files also allow pg_restore to be selective about what is restored, or even to reorder the items prior to being restored. The archive files are designed to be portable across architectures.

pg_restore can operate in two modes. If a database name is specified, pg_restore connects to that database and restores archive contents directly into the database. Otherwise, a script containing the SQL commands necessary to rebuild the database is created and written to a file or standard output. This script output is equivalent to the plain text output format of pg_dump. Some of the options controlling the output are therefore analogous to pg_dump options.

Obviously, pg_restore cannot restore information that is not present in the archive file. For instance, if the archive was made using the “dump data as INSERT commands” option, pg_restore will not be able to load the data using COPY statements.

Options

pg_restore accepts the following command line arguments.

filename

Specifies the location of the archive file (or directory, for a directory-format archive) to be restored. If not specified, the standard input is used.

-a
--data-only

Restore only the data, not the schema (data definitions). Table data, large objects, and sequence values are restored, if present in the archive.

This option is similar to, but for historical reasons not identical to, specifying --section=data.

-c
--clean

Clean (drop) database objects before recreating them. (Unless --if-exists is used, this might generate some harmless error messages, if any objects were not present in the destination database.)

-C
--create

Create the database before restoring into it. If --clean is also specified, drop and recreate the target database before connecting to it.

With --create, pg_restore also restores the database's comment if any, and any configuration variable settings that are specific to this database, that is, any ALTER DATABASE ... SET ... and ALTER ROLE ... IN DATABASE ... SET ... commands that mention this database. Access privileges for the database itself are also restored, unless --no-acl is specified.

When this option is used, the database named with -d is used only to issue the initial DROP DATABASE and CREATE DATABASE commands. All data is restored into the database name that appears in the archive.

-d dbname
--dbname=dbname

Connect to database dbname and restore directly into the database. The dbname can be a connection string. If so, connection string parameters will override any conflicting command line options.

-e
--exit-on-error

Exit if an error is encountered while sending SQL commands to the database. The default is to continue and to display a count of errors at the end of the restoration.

-f filename
--file=filename

Specify output file for generated script, or for the listing when used with -l. Use - for stdout.

-F format
--format=format

Specify format of the archive. It is not necessary to specify the format, since pg_restore will determine the format automatically. If specified, it can be one of the following:

c
custom

The archive is in the custom format of pg_dump.

d
directory

The archive is a directory archive.

t
tar

The archive is a tar archive.

-I index
--index=index

Restore definition of named index only. Multiple indexes may be specified with multiple -I switches.

-j number-of-jobs
--jobs=number-of-jobs

Run the most time-consuming steps of pg_restore — those that load data, create indexes, or create constraints — concurrently, using up to number-of-jobs concurrent sessions. This option can dramatically reduce the time to restore a large database to a server running on a multiprocessor machine. This option is ignored when emitting a script rather than connecting directly to a database server.

Each job is one process or one thread, depending on the operating system, and uses a separate connection to the server.

The optimal value for this option depends on the hardware setup of the server, of the client, and of the network. Factors include the number of CPU cores and the disk setup. A good place to start is the number of CPU cores on the server, but values larger than that can also lead to faster restore times in many cases. Of course, values that are too high will lead to decreased performance because of thrashing.

Only the custom and directory archive formats are supported with this option. The input must be a regular file or directory (not, for example, a pipe or standard input). Also, multiple jobs cannot be used together with the option --single-transaction.

-l
--list

List the table of contents of the archive. The output of this operation can be used as input to the -L option. Note that if filtering switches such as -n or -t are used with -l, they will restrict the items listed.

-L list-file
--use-list=list-file

Restore only those archive elements that are listed in list-file, and restore them in the order they appear in the file. Note that if filtering switches such as -n or -t are used with -L, they will further restrict the items restored.

list-file is normally created by editing the output of a previous -l operation. Lines can be moved or removed, and can also be commented out by placing a semicolon (;) at the start of the line. See below for examples.

-n schema
--schema=schema

Restore only objects that are in the named schema. Multiple schemas may be specified with multiple -n switches. This can be combined with the -t option to restore just a specific table.

-N schema
--exclude-schema=schema

Do not restore objects that are in the named schema. Multiple schemas to be excluded may be specified with multiple -N switches.

When both -n and -N are given for the same schema name, the -N switch wins and the schema is excluded.

-O
--no-owner

Do not output commands to set ownership of objects to match the original database. By default, pg_restore issues ALTER OWNER or SET SESSION AUTHORIZATION statements to set ownership of created schema elements. These statements will fail unless the initial connection to the database is made by a superuser (or the same user that owns all of the objects in the script). With -O, any user name can be used for the initial connection, and this user will own all the created objects.

-P function-name(argtype [, ...])
--function=function-name(argtype [, ...])

Restore the named function only. Be careful to spell the function name and arguments exactly as they appear in the dump file's table of contents. Multiple functions may be specified with multiple -P switches.

-R
--no-reconnect

This option is obsolete but still accepted for backwards compatibility.

-s
--schema-only

Restore only the schema (data definitions), not data, to the extent that schema entries are present in the archive.

This option is the inverse of --data-only. It is similar to, but for historical reasons not identical to, specifying --section=pre-data --section=post-data.

(Do not confuse this with the --schema option, which uses the word “schema” in a different meaning.)

-S username
--superuser=username

Specify the superuser user name to use when disabling triggers. This is relevant only if --disable-triggers is used.

-t table
--table=table

Restore definition and/or data of only the named table. For this purpose, “table” includes views, materialized views, sequences, and foreign tables. Multiple tables can be selected by writing multiple -t switches. This option can be combined with the -n option to specify table(s) in a particular schema.

Note

When -t is specified, pg_restore makes no attempt to restore any other database objects that the selected table(s) might depend upon. Therefore, there is no guarantee that a specific-table restore into a clean database will succeed.

Note

This flag does not behave identically to the -t flag of pg_dump. There is not currently any provision for wild-card matching in pg_restore, nor can you include a schema name within its -t. And, while pg_dump's -t flag will also dump subsidiary objects (such as indexes) of the selected table(s), pg_restore's -t flag does not include such subsidiary objects.

Note

In versions prior to PostgreSQL 9.6, this flag matched only tables, not any other type of relation.

-T trigger
--trigger=trigger

Restore named trigger only. Multiple triggers may be specified with multiple -T switches.

-v
--verbose

Specifies verbose mode. This will cause pg_restore to output detailed object comments and start/stop times to the output file, and progress messages to standard error. Repeating the option causes additional debug-level messages to appear on standard error.

-V
--version

Print the pg_restore version and exit.

-x
--no-privileges
--no-acl

Prevent restoration of access privileges (grant/revoke commands).

-1
--single-transaction

Execute the restore as a single transaction (that is, wrap the emitted commands in BEGIN/COMMIT). This ensures that either all the commands complete successfully, or no changes are applied. This option implies --exit-on-error.

--disable-triggers

This option is relevant only when performing a data-only restore. It instructs pg_restore to execute commands to temporarily disable triggers on the target tables while the data is reloaded. Use this if you have referential integrity checks or other triggers on the tables that you do not want to invoke during data reload.

Presently, the commands emitted for --disable-triggers must be done as superuser. So you should also specify a superuser name with -S or, preferably, run pg_restore as a PostgreSQL superuser.

--enable-row-security

This option is relevant only when restoring the contents of a table which has row security. By default, pg_restore will set row_security to off, to ensure that all data is restored in to the table. If the user does not have sufficient privileges to bypass row security, then an error is thrown. This parameter instructs pg_restore to set row_security to on instead, allowing the user to attempt to restore the contents of the table with row security enabled. This might still fail if the user does not have the right to insert the rows from the dump into the table.

Note that this option currently also requires the dump be in INSERT format, as COPY FROM does not support row security.

--if-exists

Use conditional commands (i.e., add an IF EXISTS clause) to drop database objects. This option is not valid unless --clean is also specified.

--no-comments

Do not output commands to restore comments, even if the archive contains them.

--no-data-for-failed-tables

By default, table data is restored even if the creation command for the table failed (e.g., because it already exists). With this option, data for such a table is skipped. This behavior is useful if the target database already contains the desired table contents. For example, auxiliary tables for PostgreSQL extensions such as PostGIS might already be loaded in the target database; specifying this option prevents duplicate or obsolete data from being loaded into them.

This option is effective only when restoring directly into a database, not when producing SQL script output.

--no-publications

Do not output commands to restore publications, even if the archive contains them.

--no-security-labels

Do not output commands to restore security labels, even if the archive contains them.

--no-subscriptions

Do not output commands to restore subscriptions, even if the archive contains them.

--no-tablespaces

Do not output commands to select tablespaces. With this option, all objects will be created in whichever tablespace is the default during restore.

--section=sectionname

Only restore the named section. The section name can be pre-data, data, or post-data. This option can be specified more than once to select multiple sections. The default is to restore all sections.

The data section contains actual table data as well as large-object definitions. Post-data items consist of definitions of indexes, triggers, rules and constraints other than validated check constraints. Pre-data items consist of all other data definition items.

--strict-names

Require that each schema (-n/--schema) and table (-t/--table) qualifier match at least one schema/table in the backup file.

--use-set-session-authorization

Output SQL-standard SET SESSION AUTHORIZATION commands instead of ALTER OWNER commands to determine object ownership. This makes the dump more standards-compatible, but depending on the history of the objects in the dump, might not restore properly.

-?
--help

Show help about pg_restore command line arguments, and exit.

pg_restore also accepts the following command line arguments for connection parameters:

-h host
--host=host

Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for the Unix domain socket. The default is taken from the PGHOST environment variable, if set, else a Unix domain socket connection is attempted.

-p port
--port=port

Specifies the TCP port or local Unix domain socket file extension on which the server is listening for connections. Defaults to the PGPORT environment variable, if set, or a compiled-in default.

-U username
--username=username

User name to connect as.

-w
--no-password

Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.

-W
--password

Force pg_restore to prompt for a password before connecting to a database.

This option is never essential, since pg_restore will automatically prompt for a password if the server demands password authentication. However, pg_restore will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt.

--role=rolename

Specifies a role name to be used to perform the restore. This option causes pg_restore to issue a SET ROLE rolename command after connecting to the database. It is useful when the authenticated user (specified by -U) lacks privileges needed by pg_restore, but can switch to a role with the required rights. Some installations have a policy against logging in directly as a superuser, and use of this option allows restores to be performed without violating the policy.

Environment

PGHOST
PGOPTIONS
PGPORT
PGUSER

Default connection parameters

PG_COLOR

Specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

This utility, like most other PostgreSQL utilities, also uses the environment variables supported by libpq (see Section 34.15). However, it does not read PGDATABASE when a database name is not supplied.

Diagnostics

When a direct database connection is specified using the -d option, pg_restore internally executes SQL statements. If you have problems running pg_restore, make sure you are able to select information from the database using, for example, psql. Also, any default connection settings and environment variables used by the libpq front-end library will apply.

Notes

If your installation has any local additions to the template1 database, be careful to load the output of pg_restore into a truly empty database; otherwise you are likely to get errors due to duplicate definitions of the added objects. To make an empty database without any local additions, copy from template0 not template1, for example:

CREATE DATABASE foo WITH TEMPLATE template0;

The limitations of pg_restore are detailed below.

  • When restoring data to a pre-existing table and the option --disable-triggers is used, pg_restore emits commands to disable triggers on user tables before inserting the data, then emits commands to re-enable them after the data has been inserted. If the restore is stopped in the middle, the system catalogs might be left in the wrong state.

  • pg_restore cannot restore large objects selectively; for instance, only those for a specific table. If an archive contains large objects, then all large objects will be restored, or none of them if they are excluded via -L, -t, or other options.

See also the pg_dump documentation for details on limitations of pg_dump.

Once restored, it is wise to run ANALYZE on each restored table so the optimizer has useful statistics; see Section 25.1.3 and Section 25.1.6 for more information.

Examples

Assume we have dumped a database called mydb into a custom-format dump file:

$ pg_dump -Fc mydb > db.dump

To drop the database and recreate it from the dump:

$ dropdb mydb
$ pg_restore -C -d postgres db.dump

The database named in the -d switch can be any database existing in the cluster; pg_restore only uses it to issue the CREATE DATABASE command for mydb. With -C, data is always restored into the database name that appears in the dump file.

To reload the dump into a new database called newdb:

$ createdb -T template0 newdb
$ pg_restore -d newdb db.dump

Notice we don't use -C, and instead connect directly to the database to be restored into. Also note that we clone the new database from template0 not template1, to ensure it is initially empty.

To reorder database items, it is first necessary to dump the table of contents of the archive:

$ pg_restore -l db.dump > db.list

The listing file consists of a header and one line for each item, e.g.:

;
; Archive created at Mon Sep 14 13:55:39 2009
;     dbname: DBDEMOS
;     TOC Entries: 81
;     Compression: 9
;     Dump Version: 1.10-0
;     Format: CUSTOM
;     Integer: 4 bytes
;     Offset: 8 bytes
;     Dumped from database version: 8.3.5
;     Dumped by pg_dump version: 8.3.8
;
;
; Selected TOC Entries:
;
3; 2615 2200 SCHEMA - public pasha
1861; 0 0 COMMENT - SCHEMA public pasha
1862; 0 0 ACL - public pasha
317; 1247 17715 TYPE public composite pasha
319; 1247 25899 DOMAIN public domain0 pasha

Semicolons start a comment, and the numbers at the start of lines refer to the internal archive ID assigned to each item.

Lines in the file can be commented out, deleted, and reordered. For example:

10; 145433 TABLE map_resolutions postgres
;2; 145344 TABLE species postgres
;4; 145359 TABLE nt_header postgres
6; 145402 TABLE species_records postgres
;8; 145416 TABLE ss_old postgres

could be used as input to pg_restore and would only restore items 10 and 6, in that order:

$ pg_restore -L db.list db.dump

See Also

pg_dump, pg_dumpall, psql
Prev | Up | Next
pg_recvlogical | Home | pg_verifybackup


pg_verifybackup
Prev Up PostgreSQL Client Applications Home Next

pg_verifybackup

pg_verifybackup — verify the integrity of a base backup of a PostgreSQL cluster

Synopsis

pg_verifybackup [option...]

Description

pg_verifybackup is used to check the integrity of a database cluster backup taken using pg_basebackup against a backup_manifest generated by the server at the time of the backup. The backup must be stored in the "plain" format; a "tar" format backup can be checked after extracting it.

It is important to note that the validation which is performed by pg_verifybackup does not and cannot include every check which will be performed by a running server when attempting to make use of the backup. Even if you use this tool, you should still perform test restores and verify that the resulting databases work as expected and that they appear to contain the correct data. However, pg_verifybackup can detect many problems that commonly occur due to storage problems or user error.

Backup verification proceeds in four stages. First, pg_verifybackup reads the backup_manifest file. If that file does not exist, cannot be read, is malformed, or fails verification against its own internal checksum, pg_verifybackup will terminate with a fatal error.

Second, pg_verifybackup will attempt to verify that the data files currently stored on disk are exactly the same as the data files which the server intended to send, with some exceptions that are described below. Extra and missing files will be detected, with a few exceptions. This step will ignore the presence or absence of, or any modifications to, postgresql.auto.conf, standby.signal, and recovery.signal, because it is expected that these files may have been created or modified as part of the process of taking the backup. It also won't complain about a backup_manifest file in the target directory or about anything inside pg_wal, even though these files won't be listed in the backup manifest. Only files are checked; the presence or absence of directories is not verified, except indirectly: if a directory is missing, any files it should have contained will necessarily also be missing.

Next, pg_verifybackup will checksum all the files, compare the checksums against the values in the manifest, and emit errors for any files for which the computed checksum does not match the checksum stored in the manifest. This step is not performed for any files which produced errors in the previous step, since they are already known to have problems. Files which were ignored in the previous step are also ignored in this step.

Finally, pg_verifybackup will use the manifest to verify that the write-ahead log records which will be needed to recover the backup are present and that they can be read and parsed. The backup_manifest contains information about which write-ahead log records will be needed, and pg_verifybackup will use that information to invoke pg_waldump to parse those write-ahead log records. The --quiet flag will be used, so that pg_waldump will only report errors, without producing any other output. While this level of verification is sufficient to detect obvious problems such as a missing file or one whose internal checksums do not match, they aren't extensive enough to detect every possible problem that might occur when attempting to recover. For instance, a server bug that produces write-ahead log records that have the correct checksums but specify nonsensical actions can't be detected by this method.

Note that if extra WAL files which are not required to recover the backup are present, they will not be checked by this tool, although a separate invocation of pg_waldump could be used for that purpose. Also note that WAL verification is version-specific: you must use the version of pg_verifybackup, and thus of pg_waldump, which pertains to the backup being checked. In contrast, the data file integrity checks should work with any version of the server that generates a backup_manifest file.

Options

pg_verifybackup accepts the following command-line arguments:

-e
--exit-on-error

Exit as soon as a problem with the backup is detected. If this option is not specified, pg_verifybackup will continue checking the backup even after a problem has been detected, and will report all problems detected as errors.

-i path
--ignore=path

Ignore the specified file or directory, which should be expressed as a relative path name, when comparing the list of data files actually present in the backup to those listed in the backup_manifest file. If a directory is specified, this option affects the entire subtree rooted at that location. Complaints about extra files, missing files, file size differences, or checksum mismatches will be suppressed if the relative path name matches the specified path name. This option can be specified multiple times.

-m path
--manifest-path=path

Use the manifest file at the specified path, rather than one located in the root of the backup directory.

-n
--no-parse-wal

Don't attempt to parse write-ahead log data that will be needed to recover from this backup.

-q
--quiet

Don't print anything when a backup is successfully verified.

-s
--skip-checksums

Do not verify data file checksums. The presence or absence of files and the sizes of those files will still be checked. This is much faster, because the files themselves do not need to be read.

-w path
--wal-directory=path

Try to parse WAL files stored in the specified directory, rather than in pg_wal. This may be useful if the backup is stored in a separate location from the WAL archive.

Other options are also available:

-V
--version

Print the pg_verifybackup version and exit.

-?
--help

Show help about pg_verifybackup command line arguments, and exit.

Examples

To create a base backup of the server at mydbserver and verify the integrity of the backup:

$ pg_basebackup -h mydbserver -D /usr/local/pgsql/data
$ pg_verifybackup /usr/local/pgsql/data

To create a base backup of the server at mydbserver, move the manifest somewhere outside the backup directory, and verify the backup:

$ pg_basebackup -h mydbserver -D /usr/local/pgsql/backup1234
$ mv /usr/local/pgsql/backup1234/backup_manifest /my/secure/location/backup_manifest.1234
$ pg_verifybackup -m /my/secure/location/backup_manifest.1234 /usr/local/pgsql/backup1234

To verify a backup while ignoring a file that was added manually to the backup directory, and also skipping checksum verification:

$ pg_basebackup -h mydbserver -D /usr/local/pgsql/data
$ edit /usr/local/pgsql/data/note.to.self
$ pg_verifybackup --ignore=note.to.self --skip-checksums /usr/local/pgsql/data

See Also

pg_basebackup
Prev | Up | Next
pg_restore | Home | psql


psql
Prev Up PostgreSQL Client Applications Home Next

psql

psql — PostgreSQL interactive terminal

Synopsis

psql [option...] [dbname [username]]

Description

psql is a terminal-based front-end to PostgreSQL. It enables you to type in queries interactively, issue them to PostgreSQL, and see the query results. Alternatively, input can be from a file or from command line arguments. In addition, psql provides a number of meta-commands and various shell-like features to facilitate writing scripts and automating a wide variety of tasks.

Options

-a
--echo-all

Print all nonempty input lines to standard output as they are read. (This does not apply to lines read interactively.) This is equivalent to setting the variable ECHO to all.

-A
--no-align

Switches to unaligned output mode. (The default output mode is aligned.) This is equivalent to \pset format unaligned.

-b
--echo-errors

Print failed SQL commands to standard error output. This is equivalent to setting the variable ECHO to errors.

-c command
--command=command

Specifies that psql is to execute the given command string, command. This option can be repeated and combined in any order with the -f option. When either -c or -f is specified, psql does not read commands from standard input; instead it terminates after processing all the -c and -f options in sequence.

command must be either a command string that is completely parsable by the server (i.e., it contains no psql-specific features), or a single backslash command. Thus you cannot mix SQL and psql meta-commands within a -c option. To achieve that, you could use repeated -c options or pipe the string into psql, for example:

psql -c '\x' -c 'SELECT * FROM foo;'

or

echo '\x \\ SELECT * FROM foo;' | psql

(\\ is the separator meta-command.)

Each SQL command string passed to -c is sent to the server as a single request. Because of this, the server executes it as a single transaction even if the string contains multiple SQL commands, unless there are explicit BEGIN/COMMIT commands included in the string to divide it into multiple transactions. (See Section 53.2.2.1 for more details about how the server handles multi-query strings.) Also, psql only prints the result of the last SQL command in the string. This is different from the behavior when the same string is read from a file or fed to psql's standard input, because then psql sends each SQL command separately.

Because of this behavior, putting more than one SQL command in a single -c string often has unexpected results. It's better to use repeated -c commands or feed multiple commands to psql's standard input, either using echo as illustrated above, or via a shell here-document, for example:

psql <<EOF
\x
SELECT * FROM foo;
EOF
--csv

Switches to CSV (Comma-Separated Values) output mode. This is equivalent to \pset format csv.

-d dbname
--dbname=dbname

Specifies the name of the database to connect to. This is equivalent to specifying dbname as the first non-option argument on the command line. The dbname can be a connection string. If so, connection string parameters will override any conflicting command line options.

-e
--echo-queries

Copy all SQL commands sent to the server to standard output as well. This is equivalent to setting the variable ECHO to queries.

-E
--echo-hidden

Echo the actual queries generated by \d and other backslash commands. You can use this to study psql's internal operations. This is equivalent to setting the variable ECHO_HIDDEN to on.

-f filename
--file=filename

Read commands from the file filename, rather than standard input. This option can be repeated and combined in any order with the -c option. When either -c or -f is specified, psql does not read commands from standard input; instead it terminates after processing all the -c and -f options in sequence. Except for that, this option is largely equivalent to the meta-command \i.

If filename is - (hyphen), then standard input is read until an EOF indication or \q meta-command. This can be used to intersperse interactive input with input from files. Note however that Readline is not used in this case (much as if -n had been specified).

Using this option is subtly different from writing psql < filename. In general, both will do what you expect, but using -f enables some nice features such as error messages with line numbers. There is also a slight chance that using this option will reduce the start-up overhead. On the other hand, the variant using the shell's input redirection is (in theory) guaranteed to yield exactly the same output you would have received had you entered everything by hand.

-F separator
--field-separator=separator

Use separator as the field separator for unaligned output. This is equivalent to \pset fieldsep or \f.

-h hostname
--host=hostname

Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for the Unix-domain socket.

-H
--html

Switches to HTML output mode. This is equivalent to \pset format html or the \H command.

-l
--list

List all available databases, then exit. Other non-connection options are ignored. This is similar to the meta-command \list.

When this option is used, psql will connect to the database postgres, unless a different database is named on the command line (option -d or non-option argument, possibly via a service entry, but not via an environment variable).

-L filename
--log-file=filename

Write all query output into file filename, in addition to the normal output destination.

-n
--no-readline

Do not use Readline for line editing and do not use the command history. This can be useful to turn off tab expansion when cutting and pasting.

-o filename
--output=filename

Put all query output into file filename. This is equivalent to the command \o.

-p port
--port=port

Specifies the TCP port or the local Unix-domain socket file extension on which the server is listening for connections. Defaults to the value of the PGPORT environment variable or, if not set, to the port specified at compile time, usually 5432.

-P assignment
--pset=assignment

Specifies printing options, in the style of \pset. Note that here you have to separate name and value with an equal sign instead of a space. For example, to set the output format to LaTeX, you could write -P format=latex.

-q
--quiet

Specifies that psql should do its work quietly. By default, it prints welcome messages and various informational output. If this option is used, none of this happens. This is useful with the -c option. This is equivalent to setting the variable QUIET to on.

-R separator
--record-separator=separator

Use separator as the record separator for unaligned output. This is equivalent to \pset recordsep.

-s
--single-step

Run in single-step mode. That means the user is prompted before each command is sent to the server, with the option to cancel execution as well. Use this to debug scripts.

-S
--single-line

Runs in single-line mode where a newline terminates an SQL command, as a semicolon does.

Note

This mode is provided for those who insist on it, but you are not necessarily encouraged to use it. In particular, if you mix SQL and meta-commands on a line the order of execution might not always be clear to the inexperienced user.

-t
--tuples-only

Turn off printing of column names and result row count footers, etc. This is equivalent to \t or \pset tuples_only.

-T table_options
--table-attr=table_options

Specifies options to be placed within the HTML table tag. See \pset tableattr for details.

-U username
--username=username

Connect to the database as the user username instead of the default. (You must have permission to do so, of course.)

-v assignment
--set=assignment
--variable=assignment

Perform a variable assignment, like the \set meta-command. Note that you must separate name and value, if any, by an equal sign on the command line. To unset a variable, leave off the equal sign. To set a variable with an empty value, use the equal sign but leave off the value. These assignments are done during command line processing, so variables that reflect connection state will get overwritten later.

-V
--version

Print the psql version and exit.

-w
--no-password

Never issue a password prompt. If the server requires password authentication and a password is not available from other sources such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.

Note that this option will remain set for the entire session, and so it affects uses of the meta-command \connect as well as the initial connection attempt.

-W
--password

Force psql to prompt for a password before connecting to a database, even if the password will not be used.

If the server requires password authentication and a password is not available from other sources such as a .pgpass file, psql will prompt for a password in any case. However, psql will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt.

Note that this option will remain set for the entire session, and so it affects uses of the meta-command \connect as well as the initial connection attempt.

-x
--expanded

Turn on the expanded table formatting mode. This is equivalent to \x or \pset expanded.

-X,
--no-psqlrc

Do not read the start-up file (neither the system-wide psqlrc file nor the user's ~/.psqlrc file).

-z
--field-separator-zero

Set the field separator for unaligned output to a zero byte. This is equivalent to \pset fieldsep_zero.

-0
--record-separator-zero

Set the record separator for unaligned output to a zero byte. This is useful for interfacing, for example, with xargs -0. This is equivalent to \pset recordsep_zero.

-1
--single-transaction

This option can only be used in combination with one or more -c and/or -f options. It causes psql to issue a BEGIN command before the first such option and a COMMIT command after the last one, thereby wrapping all the commands into a single transaction. This ensures that either all the commands complete successfully, or no changes are applied.

If the commands themselves contain BEGIN, COMMIT, or ROLLBACK, this option will not have the desired effects. Also, if an individual command cannot be executed inside a transaction block, specifying this option will cause the whole transaction to fail.

-?
--help[=topic]

Show help about psql and exit. The optional topic parameter (defaulting to options) selects which part of psql is explained: commands describes psql's backslash commands; options describes the command-line options that can be passed to psql; and variables shows help about psql configuration variables.

Exit Status

psql returns 0 to the shell if it finished normally, 1 if a fatal error of its own occurs (e.g., out of memory, file not found), 2 if the connection to the server went bad and the session was not interactive, and 3 if an error occurred in a script and the variable ON_ERROR_STOP was set.

Usage

Connecting to a Database

psql is a regular PostgreSQL client application. In order to connect to a database you need to know the name of your target database, the host name and port number of the server, and what user name you want to connect as. psql can be told about those parameters via command line options, namely -d, -h, -p, and -U respectively. If an argument is found that does not belong to any option it will be interpreted as the database name (or the user name, if the database name is already given). Not all of these options are required; there are useful defaults. If you omit the host name, psql will connect via a Unix-domain socket to a server on the local host, or via TCP/IP to localhost on machines that don't have Unix-domain sockets. The default port number is determined at compile time. Since the database server uses the same default, you will not have to specify the port in most cases. The default user name is your operating-system user name, as is the default database name. Note that you cannot just connect to any database under any user name. Your database administrator should have informed you about your access rights.

When the defaults aren't quite right, you can save yourself some typing by setting the environment variables PGDATABASE, PGHOST, PGPORT and/or PGUSER to appropriate values. (For additional environment variables, see Section 34.15.) It is also convenient to have a ~/.pgpass file to avoid regularly having to type in passwords. See Section 34.16 for more information.

An alternative way to specify connection parameters is in a conninfo string or a URI, which is used instead of a database name. This mechanism give you very wide control over the connection. For example:

$ psql "service=myservice sslmode=require"
$ psql postgresql://dbmaster:5433/mydb?sslmode=require

This way you can also use LDAP for connection parameter lookup as described in Section 34.18. See Section 34.1.2 for more information on all the available connection options.

If the connection could not be made for any reason (e.g., insufficient privileges, server is not running on the targeted host, etc.), psql will return an error and terminate.

If both standard input and standard output are a terminal, then psql sets the client encoding to “auto”, which will detect the appropriate client encoding from the locale settings (LC_CTYPE environment variable on Unix systems). If this doesn't work out as expected, the client encoding can be overridden using the environment variable PGCLIENTENCODING.

Entering SQL Commands

In normal operation, psql provides a prompt with the name of the database to which psql is currently connected, followed by the string =>. For example:

$ psql testdb
psql (14.4)
Type "help" for help.

testdb=>

At the prompt, the user can type in SQL commands. Ordinarily, input lines are sent to the server when a command-terminating semicolon is reached. An end of line does not terminate a command. Thus commands can be spread over several lines for clarity. If the command was sent and executed without error, the results of the command are displayed on the screen.

If untrusted users have access to a database that has not adopted a secure schema usage pattern, begin your session by removing publicly-writable schemas from search_path. One can add options=-csearch_path= to the connection string or issue SELECT pg_catalog.set_config('search_path', '', false) before other SQL commands. This consideration is not specific to psql; it applies to every interface for executing arbitrary SQL commands.

Whenever a command is executed, psql also polls for asynchronous notification events generated by LISTEN and NOTIFY.

While C-style block comments are passed to the server for processing and removal, SQL-standard comments are removed by psql.

Meta-Commands

Anything you enter in psql that begins with an unquoted backslash is a psql meta-command that is processed by psql itself. These commands make psql more useful for administration or scripting. Meta-commands are often called slash or backslash commands.

The format of a psql command is the backslash, followed immediately by a command verb, then any arguments. The arguments are separated from the command verb and each other by any number of whitespace characters.

To include whitespace in an argument you can quote it with single quotes. To include a single quote in an argument, write two single quotes within single-quoted text. Anything contained in single quotes is furthermore subject to C-like substitutions for \n (new line), \t (tab), \b (backspace), \r (carriage return), \f (form feed), \digits (octal), and \x digits (hexadecimal). A backslash preceding any other character within single-quoted text quotes that single character, whatever it is.

If an unquoted colon (:) followed by a psql variable name appears within an argument, it is replaced by the variable's value, as described in SQL Interpolation below. The forms :'variable_name' and :"variable_name" described there work as well. The :{?variable_name} syntax allows testing whether a variable is defined. It is substituted by TRUE or FALSE. Escaping the colon with a backslash protects it from substitution.

Within an argument, text that is enclosed in backquotes (`) is taken as a command line that is passed to the shell. The output of the command (with any trailing newline removed) replaces the backquoted text. Within the text enclosed in backquotes, no special quoting or other processing occurs, except that appearances of :variable_name where variable_name is a psql variable name are replaced by the variable's value. Also, appearances of :'variable_name' are replaced by the variable's value suitably quoted to become a single shell command argument. (The latter form is almost always preferable, unless you are very sure of what is in the variable.) Because carriage return and line feed characters cannot be safely quoted on all platforms, the :'variable_name' form prints an error message and does not substitute the variable value when such characters appear in the value.

Some commands take an SQL identifier (such as a table name) as argument. These arguments follow the syntax rules of SQL: Unquoted letters are forced to lowercase, while double quotes (") protect letters from case conversion and allow incorporation of whitespace into the identifier. Within double quotes, paired double quotes reduce to a single double quote in the resulting name. For example, FOO"BAR"BAZ is interpreted as fooBARbaz, and "A weird"" name" becomes A weird" name.

Parsing for arguments stops at the end of the line, or when another unquoted backslash is found. An unquoted backslash is taken as the beginning of a new meta-command. The special sequence \\ (two backslashes) marks the end of arguments and continues parsing SQL commands, if any. That way SQL and psql commands can be freely mixed on a line. But in any case, the arguments of a meta-command cannot continue beyond the end of the line.

Many of the meta-commands act on the current query buffer. This is simply a buffer holding whatever SQL command text has been typed but not yet sent to the server for execution. This will include previous input lines as well as any text appearing before the meta-command on the same line.

The following meta-commands are defined:

\a

If the current table output format is unaligned, it is switched to aligned. If it is not unaligned, it is set to unaligned. This command is kept for backwards compatibility. See \pset for a more general solution.

\c or \connect [ -reuse-previous=on|off ] [ dbname [ username ] [ host ] [ port ] | conninfo ]

Establishes a new connection to a PostgreSQL server. The connection parameters to use can be specified either using a positional syntax (one or more of database name, user, host, and port), or using a conninfo connection string as detailed in Section 34.1.1. If no arguments are given, a new connection is made using the same parameters as before.

Specifying any of dbname, username, host or port as - is equivalent to omitting that parameter.

The new connection can re-use connection parameters from the previous connection; not only database name, user, host, and port, but other settings such as sslmode. By default, parameters are re-used in the positional syntax, but not when a conninfo string is given. Passing a first argument of -reuse-previous=on or -reuse-previous=off overrides that default. If parameters are re-used, then any parameter not explicitly specified as a positional parameter or in the conninfo string is taken from the existing connection's parameters. An exception is that if the host setting is changed from its previous value using the positional syntax, any hostaddr setting present in the existing connection's parameters is dropped. Also, any password used for the existing connection will be re-used only if the user, host, and port settings are not changed. When the command neither specifies nor reuses a particular parameter, the libpq default is used.

If the new connection is successfully made, the previous connection is closed. If the connection attempt fails (wrong user name, access denied, etc.), the previous connection will be kept if psql is in interactive mode. But when executing a non-interactive script, the old connection is closed and an error is reported. That may or may not terminate the script; if it does not, all database-accessing commands will fail until another \connect command is successfully executed. This distinction was chosen as a user convenience against typos on the one hand, and a safety mechanism that scripts are not accidentally acting on the wrong database on the other hand. Note that whenever a \connect command attempts to re-use parameters, the values re-used are those of the last successful connection, not of any failed attempts made subsequently. However, in the case of a non-interactive \connect failure, no parameters are allowed to be re-used later, since the script would likely be expecting the values from the failed \connect to be re-used.

Examples:

=> \c mydb myuser host.dom 6432
=> \c service=foo
=> \c "host=localhost port=5432 dbname=mydb connect_timeout=10 sslmode=disable"
=> \c -reuse-previous=on sslmode=require    -- changes only sslmode
=> \c postgresql://tom@localhost/mydb?application_name=myapp
\C [ title ]

Sets the title of any tables being printed as the result of a query or unset any such title. This command is equivalent to \pset title title. (The name of this command derives from “caption”, as it was previously only used to set the caption in an HTML table.)

\cd [ directory ]

Changes the current working directory to directory. Without argument, changes to the current user's home directory.

Tip

To print your current working directory, use \! pwd.

\conninfo

Outputs information about the current database connection.

\copy { table [ ( column_list ) ] } from { 'filename' | program 'command' | stdin | pstdin } [ [ with ] ( option [, ...] ) ] [ where condition ]
\copy { table [ ( column_list ) ] | ( query ) } to { 'filename' | program 'command' | stdout | pstdout } [ [ with ] ( option [, ...] ) ]

Performs a frontend (client) copy. This is an operation that runs an SQL COPY command, but instead of the server reading or writing the specified file, psql reads or writes the file and routes the data between the server and the local file system. This means that file accessibility and privileges are those of the local user, not the server, and no SQL superuser privileges are required.

When program is specified, command is executed by psql and the data passed from or to command is routed between the server and the client. Again, the execution privileges are those of the local user, not the server, and no SQL superuser privileges are required.

For \copy ... from stdin, data rows are read from the same source that issued the command, continuing until \. is read or the stream reaches EOF. This option is useful for populating tables in-line within an SQL script file. For \copy ... to stdout, output is sent to the same place as psql command output, and the COPY count command status is not printed (since it might be confused with a data row). To read/write psql's standard input or output regardless of the current command source or \o option, write from pstdin or to pstdout.

The syntax of this command is similar to that of the SQL COPY command. All options other than the data source/destination are as specified for COPY. Because of this, special parsing rules apply to the \copy meta-command. Unlike most other meta-commands, the entire remainder of the line is always taken to be the arguments of \copy, and neither variable interpolation nor backquote expansion are performed in the arguments.

Tip

Another way to obtain the same result as \copy ... to is to use the SQL COPY ... TO STDOUT command and terminate it with \g filename or \g |program. Unlike \copy, this method allows the command to span multiple lines; also, variable interpolation and backquote expansion can be used.

Tip

These operations are not as efficient as the SQL COPY command with a file or program data source or destination, because all data must pass through the client/server connection. For large amounts of data the SQL command might be preferable.

\copyright

Shows the copyright and distribution terms of PostgreSQL.

\crosstabview [ colV [ colH [ colD [ sortcolH ] ] ] ]

Executes the current query buffer (like \g) and shows the results in a crosstab grid. The query must return at least three columns. The output column identified by colV becomes a vertical header and the output column identified by colH becomes a horizontal header. colD identifies the output column to display within the grid. sortcolH identifies an optional sort column for the horizontal header.

Each column specification can be a column number (starting at 1) or a column name. The usual SQL case folding and quoting rules apply to column names. If omitted, colV is taken as column 1 and colH as column 2. colH must differ from colV. If colD is not specified, then there must be exactly three columns in the query result, and the column that is neither colV nor colH is taken to be colD.

The vertical header, displayed as the leftmost column, contains the values found in column colV, in the same order as in the query results, but with duplicates removed.

The horizontal header, displayed as the first row, contains the values found in column colH, with duplicates removed. By default, these appear in the same order as in the query results. But if the optional sortcolH argument is given, it identifies a column whose values must be integer numbers, and the values from colH will appear in the horizontal header sorted according to the corresponding sortcolH values.

Inside the crosstab grid, for each distinct value x of colH and each distinct value y of colV, the cell located at the intersection (x,y) contains the value of the colD column in the query result row for which the value of colH is x and the value of colV is y. If there is no such row, the cell is empty. If there are multiple such rows, an error is reported.

\d[S+] [ pattern ]

For each relation (table, view, materialized view, index, sequence, or foreign table) or composite type matching the pattern, show all columns, their types, the tablespace (if not the default) and any special attributes such as NOT NULL or defaults. Associated indexes, constraints, rules, and triggers are also shown. For foreign tables, the associated foreign server is shown as well. (“Matching the pattern” is defined in Patterns below.)

For some types of relation, \d shows additional information for each column: column values for sequences, indexed expressions for indexes, and foreign data wrapper options for foreign tables.

The command form \d+ is identical, except that more information is displayed: any comments associated with the columns of the table are shown, as is the presence of OIDs in the table, the view definition if the relation is a view, a non-default replica identity setting and the access method name if the relation has an access method.

By default, only user-created objects are shown; supply a pattern or the S modifier to include system objects.

Note

If \d is used without a pattern argument, it is equivalent to \dtvmsE which will show a list of all visible tables, views, materialized views, sequences and foreign tables. This is purely a convenience measure.

\da[S] [ pattern ]

Lists aggregate functions, together with their return type and the data types they operate on. If pattern is specified, only aggregates whose names match the pattern are shown. By default, only user-created objects are shown; supply a pattern or the S modifier to include system objects.

\dA[+] [ pattern ]

Lists access methods. If pattern is specified, only access methods whose names match the pattern are shown. If + is appended to the command name, each access method is listed with its associated handler function and description.

\dAc[+] [access-method-pattern [input-type-pattern]]

Lists operator classes (see Section 38.16.1). If access-method-pattern is specified, only operator classes associated with access methods whose names match that pattern are listed. If input-type-pattern is specified, only operator classes associated with input types whose names match that pattern are listed. If + is appended to the command name, each operator class is listed with its associated operator family and owner.

\dAf[+] [access-method-pattern [input-type-pattern]]

Lists operator families (see Section 38.16.5). If access-method-pattern is specified, only operator families associated with access methods whose names match that pattern are listed. If input-type-pattern is specified, only operator families associated with input types whose names match that pattern are listed. If + is appended to the command name, each operator family is listed with its owner.

\dAo[+] [access-method-pattern [operator-family-pattern]]

Lists operators associated with operator families (see Section 38.16.2). If access-method-pattern is specified, only members of operator families associated with access methods whose names match that pattern are listed. If operator-family-pattern is specified, only members of operator families whose names match that pattern are listed. If + is appended to the command name, each operator is listed with its sort operator family (if it is an ordering operator).

\dAp[+] [access-method-pattern [operator-family-pattern]]

Lists support functions associated with operator families (see Section 38.16.3). If access-method-pattern is specified, only functions of operator families associated with access methods whose names match that pattern are listed. If operator-family-pattern is specified, only functions of operator families whose names match that pattern are listed. If + is appended to the command name, functions are displayed verbosely, with their actual parameter lists.

\db[+] [ pattern ]

Lists tablespaces. If pattern is specified, only tablespaces whose names match the pattern are shown. If + is appended to the command name, each tablespace is listed with its associated options, on-disk size, permissions and description.

\dc[S+] [ pattern ]

Lists conversions between character-set encodings. If pattern is specified, only conversions whose names match the pattern are listed. By default, only user-created objects are shown; supply a pattern or the S modifier to include system objects. If + is appended to the command name, each object is listed with its associated description.

\dC[+] [ pattern ]

Lists type casts. If pattern is specified, only casts whose source or target types match the pattern are listed. If + is appended to the command name, each object is listed with its associated description.

\dd[S] [ pattern ]

Shows the descriptions of objects of type constraint, operator class, operator family, rule, and trigger. All other comments may be viewed by the respective backslash commands for those object types.

\dd displays descriptions for objects matching the pattern, or of visible objects of the appropriate type if no argument is given. But in either case, only objects that have a description are listed. By default, only user-created objects are shown; supply a pattern or the S modifier to include system objects.

Descriptions for objects can be created with the COMMENT SQL command.

\dD[S+] [ pattern ]

Lists domains. If pattern is specified, only domains whose names match the pattern are shown. By default, only user-created objects are shown; supply a pattern or the S modifier to include system objects. If + is appended to the command name, each object is listed with its associated permissions and description.

\ddp [ pattern ]

Lists default access privilege settings. An entry is shown for each role (and schema, if applicable) for which the default privilege settings have been changed from the built-in defaults. If pattern is specified, only entries whose role name or schema name matches the pattern are listed.

The ALTER DEFAULT PRIVILEGES command is used to set default access privileges. The meaning of the privilege display is explained in Section 5.7.

\dE[S+] [ pattern ]
\di[S+] [ pattern ]
\dm[S+] [ pattern ]
\ds[S+] [ pattern ]
\dt[S+] [ pattern ]
\dv[S+] [ pattern ]

In this group of commands, the letters E, i, m, s, t, and v stand for foreign table, index, materialized view, sequence, table, and view, respectively. You can specify any or all of these letters, in any order, to obtain a listing of objects of these types. For example, \dti lists tables and indexes. If + is appended to the command name, each object is listed with its persistence status (permanent, temporary, or unlogged), physical size on disk, and associated description if any. If pattern is specified, only objects whose names match the pattern are listed. By default, only user-created objects are shown; supply a pattern or the S modifier to include system objects.

\des[+] [ pattern ]

Lists foreign servers (mnemonic: “external servers”). If pattern is specified, only those servers whose name matches the pattern are listed. If the form \des+ is used, a full description of each server is shown, including the server's access privileges, type, version, options, and description.

\det[+] [ pattern ]

Lists foreign tables (mnemonic: “external tables”). If pattern is specified, only entries whose table name or schema name matches the pattern are listed. If the form \det+ is used, generic options and the foreign table description are also displayed.

\deu[+] [ pattern ]

Lists user mappings (mnemonic: “external users”). If pattern is specified, only those mappings whose user names match the pattern are listed. If the form \deu+ is used, additional information about each mapping is shown.

Caution

\deu+ might also display the user name and password of the remote user, so care should be taken not to disclose them.

\dew[+] [ pattern ]

Lists foreign-data wrappers (mnemonic: “external wrappers”). If pattern is specified, only those foreign-data wrappers whose name matches the pattern are listed. If the form \dew+ is used, the access privileges, options, and description of the foreign-data wrapper are also shown.

\df[anptwS+] [ pattern [ arg_pattern ... ] ]

Lists functions, together with their result data types, argument data types, and function types, which are classified as “agg” (aggregate), “normal”, “procedure”, “trigger”, or “window”. To display only functions of specific type(s), add the corresponding letters a, n, p, t, or w to the command. If pattern is specified, only functions whose names match the pattern are shown. Any additional arguments are type-name patterns, which are matched to the type names of the first, second, and so on arguments of the function. (Matching functions can have more arguments than what you specify. To prevent that, write a dash - as the last arg_pattern.) By default, only user-created objects are shown; supply a pattern or the S modifier to include system objects. If the form \df+ is used, additional information about each function is shown, including volatility, parallel safety, owner, security classification, access privileges, language, source code and description.

\dF[+] [ pattern ]

Lists text search configurations. If pattern is specified, only configurations whose names match the pattern are shown. If the form \dF+ is used, a full description of each configuration is shown, including the underlying text search parser and the dictionary list for each parser token type.

\dFd[+] [ pattern ]

Lists text search dictionaries. If pattern is specified, only dictionaries whose names match the pattern are shown. If the form \dFd+ is used, additional information is shown about each selected dictionary, including the underlying text search template and the option values.

\dFp[+] [ pattern ]

Lists text search parsers. If pattern is specified, only parsers whose names match the pattern are shown. If the form \dFp+ is used, a full description of each parser is shown, including the underlying functions and the list of recognized token types.

\dFt[+] [ pattern ]

Lists text search templates. If pattern is specified, only templates whose names match the pattern are shown. If the form \dFt+ is used, additional information is shown about each template, including the underlying function names.

\dg[S+] [ pattern ]

Lists database roles. (Since the concepts of “users” and “groups” have been unified into “roles”, this command is now equivalent to \du.) By default, only user-created roles are shown; supply the S modifier to include system roles. If pattern is specified, only those roles whose names match the pattern are listed. If the form \dg+ is used, additional information is shown about each role; currently this adds the comment for each role.

\dl

This is an alias for \lo_list, which shows a list of large objects.

\dL[S+] [ pattern ]

Lists procedural languages. If pattern is specified, only languages whose names match the pattern are listed. By default, only user-created languages are shown; supply the S modifier to include system objects. If + is appended to the command name, each language is listed with its call handler, validator, access privileges, and whether it is a system object.

\dn[S+] [ pattern ]

Lists schemas (namespaces). If pattern is specified, only schemas whose names match the pattern are listed. By default, only user-created objects are shown; supply a pattern or the S modifier to include system objects. If + is appended to the command name, each object is listed with its associated permissions and description, if any.

\do[S+] [ pattern [ arg_pattern [ arg_pattern ] ] ]

Lists operators with their operand and result types. If pattern is specified, only operators whose names match the pattern are listed. If one arg_pattern is specified, only prefix operators whose right argument's type name matches that pattern are listed. If two arg_patterns are specified, only binary operators whose argument type names match those patterns are listed. (Alternatively, write - for the unused argument of a unary operator.) By default, only user-created objects are shown; supply a pattern or the S modifier to include system objects. If + is appended to the command name, additional information about each operator is shown, currently just the name of the underlying function.

\dO[S+] [ pattern ]

Lists collations. If pattern is specified, only collations whose names match the pattern are listed. By default, only user-created objects are shown; supply a pattern or the S modifier to include system objects. If + is appended to the command name, each collation is listed with its associated description, if any. Note that only collations usable with the current database's encoding are shown, so the results may vary in different databases of the same installation.

\dp [ pattern ]

Lists tables, views and sequences with their associated access privileges. If pattern is specified, only tables, views and sequences whose names match the pattern are listed.

The GRANT and REVOKE commands are used to set access privileges. The meaning of the privilege display is explained in Section 5.7.

\dP[itn+] [ pattern ]

Lists partitioned relations. If pattern is specified, only entries whose name matches the pattern are listed. The modifiers t (tables) and i (indexes) can be appended to the command, filtering the kind of relations to list. By default, partitioned tables and indexes are listed.

If the modifier n (“nested”) is used, or a pattern is specified, then non-root partitioned relations are included, and a column is shown displaying the parent of each partitioned relation.

If + is appended to the command name, the sum of the sizes of each relation's partitions is also displayed, along with the relation's description. If n is combined with +, two sizes are shown: one including the total size of directly-attached leaf partitions, and another showing the total size of all partitions, including indirectly attached sub-partitions.

\drds [ role-pattern [ database-pattern ] ]

Lists defined configuration settings. These settings can be role-specific, database-specific, or both. role-pattern and database-pattern are used to select specific roles and databases to list, respectively. If omitted, or if * is specified, all settings are listed, including those not role-specific or database-specific, respectively.

The ALTER ROLE and ALTER DATABASE commands are used to define per-role and per-database configuration settings.

\dRp[+] [ pattern ]

Lists replication publications. If pattern is specified, only those publications whose names match the pattern are listed. If + is appended to the command name, the tables associated with each publication are shown as well.

\dRs[+] [ pattern ]

Lists replication subscriptions. If pattern is specified, only those subscriptions whose names match the pattern are listed. If + is appended to the command name, additional properties of the subscriptions are shown.

\dT[S+] [ pattern ]

Lists data types. If pattern is specified, only types whose names match the pattern are listed. If + is appended to the command name, each type is listed with its internal name and size, its allowed values if it is an enum type, and its associated permissions. By default, only user-created objects are shown; supply a pattern or the S modifier to include system objects.

\du[S+] [ pattern ]

Lists database roles. (Since the concepts of “users” and “groups” have been unified into “roles”, this command is now equivalent to \dg.) By default, only user-created roles are shown; supply the S modifier to include system roles. If pattern is specified, only those roles whose names match the pattern are listed. If the form \du+ is used, additional information is shown about each role; currently this adds the comment for each role.

\dx[+] [ pattern ]

Lists installed extensions. If pattern is specified, only those extensions whose names match the pattern are listed. If the form \dx+ is used, all the objects belonging to each matching extension are listed.

\dX [ pattern ]

Lists extended statistics. If pattern is specified, only those extended statistics whose names match the pattern are listed.

The status of each kind of extended statistics is shown in a column named after its statistic kind (e.g. Ndistinct). defined means that it was requested when creating the statistics, and NULL means it wasn't requested. You can use pg_stats_ext if you'd like to know whether ANALYZE was run and statistics are available to the planner.

\dy[+] [ pattern ]

Lists event triggers. If pattern is specified, only those event triggers whose names match the pattern are listed. If + is appended to the command name, each object is listed with its associated description.

\e or \edit [ filename ] [ line_number ]

If filename is specified, the file is edited; after the editor exits, the file's content is copied into the current query buffer. If no filename is given, the current query buffer is copied to a temporary file which is then edited in the same fashion. Or, if the current query buffer is empty, the most recently executed query is copied to a temporary file and edited in the same fashion.

If you edit a file or the previous query, and you quit the editor without modifying the file, the query buffer is cleared. Otherwise, the new contents of the query buffer are re-parsed according to the normal rules of psql, treating the whole buffer as a single line. Any complete queries are immediately executed; that is, if the query buffer contains or ends with a semicolon, everything up to that point is executed and removed from the query buffer. Whatever remains in the query buffer is redisplayed. Type semicolon or \g to send it, or \r to cancel it by clearing the query buffer.

Treating the buffer as a single line primarily affects meta-commands: whatever is in the buffer after a meta-command will be taken as argument(s) to the meta-command, even if it spans multiple lines. (Thus you cannot make meta-command-using scripts this way. Use \i for that.)

If a line number is specified, psql will position the cursor on the specified line of the file or query buffer. Note that if a single all-digits argument is given, psql assumes it is a line number, not a file name.

Tip

See Environment, below, for how to configure and customize your editor.

\echo text [ ... ]

Prints the evaluated arguments to standard output, separated by spaces and followed by a newline. This can be useful to intersperse information in the output of scripts. For example:

=> \echo `date`
Tue Oct 26 21:40:57 CEST 1999

If the first argument is an unquoted -n the trailing newline is not written (nor is the first argument).

Tip

If you use the \o command to redirect your query output you might wish to use \qecho instead of this command. See also \warn.

\ef [ function_description [ line_number ] ]

This command fetches and edits the definition of the named function or procedure, in the form of a CREATE OR REPLACE FUNCTION or CREATE OR REPLACE PROCEDURE command. Editing is done in the same way as for \edit. If you quit the editor without saving, the statement is discarded. If you save and exit the editor, the updated command is executed immediately if you added a semicolon to it. Otherwise it is redisplayed; type semicolon or \g to send it, or \r to cancel.

The target function can be specified by name alone, or by name and arguments, for example foo(integer, text). The argument types must be given if there is more than one function of the same name.

If no function is specified, a blank CREATE FUNCTION template is presented for editing.

If a line number is specified, psql will position the cursor on the specified line of the function body. (Note that the function body typically does not begin on the first line of the file.)

Unlike most other meta-commands, the entire remainder of the line is always taken to be the argument(s) of \ef, and neither variable interpolation nor backquote expansion are performed in the arguments.

Tip

See Environment, below, for how to configure and customize your editor.

\encoding [ encoding ]

Sets the client character set encoding. Without an argument, this command shows the current encoding.

\errverbose

Repeats the most recent server error message at maximum verbosity, as though VERBOSITY were set to verbose and SHOW_CONTEXT were set to always.

\ev [ view_name [ line_number ] ]

This command fetches and edits the definition of the named view, in the form of a CREATE OR REPLACE VIEW command. Editing is done in the same way as for \edit. If you quit the editor without saving, the statement is discarded. If you save and exit the editor, the updated command is executed immediately if you added a semicolon to it. Otherwise it is redisplayed; type semicolon or \g to send it, or \r to cancel.

If no view is specified, a blank CREATE VIEW template is presented for editing.

If a line number is specified, psql will position the cursor on the specified line of the view definition.

Unlike most other meta-commands, the entire remainder of the line is always taken to be the argument(s) of \ev, and neither variable interpolation nor backquote expansion are performed in the arguments.

\f [ string ]

Sets the field separator for unaligned query output. The default is the vertical bar (|). It is equivalent to \pset fieldsep.

\g [ (option=value [...]) ] [ filename ]
\g [ (option=value [...]) ] [ |command ]

Sends the current query buffer to the server for execution.

If parentheses appear after \g, they surround a space-separated list of option=value formatting-option clauses, which are interpreted in the same way as \pset option value commands, but take effect only for the duration of this query. In this list, spaces are not allowed around = signs, but are required between option clauses. If =value is omitted, the named option is changed in the same way as for \pset option with no explicit value.

If a filename or |command argument is given, the query's output is written to the named file or piped to the given shell command, instead of displaying it as usual. The file or command is written to only if the query successfully returns zero or more tuples, not if the query fails or is a non-data-returning SQL command.

If the current query buffer is empty, the most recently sent query is re-executed instead. Except for that behavior, \g without any arguments is essentially equivalent to a semicolon. With arguments, \g provides a “one-shot” alternative to the \o command, and additionally allows one-shot adjustments of the output formatting options normally set by \pset.

When the last argument begins with |, the entire remainder of the line is taken to be the command to execute, and neither variable interpolation nor backquote expansion are performed in it. The rest of the line is simply passed literally to the shell.

\gdesc

Shows the description (that is, the column names and data types) of the result of the current query buffer. The query is not actually executed; however, if it contains some type of syntax error, that error will be reported in the normal way.

If the current query buffer is empty, the most recently sent query is described instead.

\gexec

Sends the current query buffer to the server, then treats each column of each row of the query's output (if any) as an SQL statement to be executed. For example, to create an index on each column of my_table:

=> SELECT format('create index on my_table(%I)', attname)
-> FROM pg_attribute
-> WHERE attrelid = 'my_table'::regclass AND attnum > 0
-> ORDER BY attnum
-> \gexec
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX

The generated queries are executed in the order in which the rows are returned, and left-to-right within each row if there is more than one column. NULL fields are ignored. The generated queries are sent literally to the server for processing, so they cannot be psql meta-commands nor contain psql variable references. If any individual query fails, execution of the remaining queries continues unless ON_ERROR_STOP is set. Execution of each query is subject to ECHO processing. (Setting ECHO to all or queries is often advisable when using \gexec.) Query logging, single-step mode, timing, and other query execution features apply to each generated query as well.

If the current query buffer is empty, the most recently sent query is re-executed instead.

\gset [ prefix ]

Sends the current query buffer to the server and stores the query's output into psql variables (see Variables below). The query to be executed must return exactly one row. Each column of the row is stored into a separate variable, named the same as the column. For example:

=> SELECT 'hello' AS var1, 10 AS var2
-> \gset
=> \echo :var1 :var2
hello 10

If you specify a prefix, that string is prepended to the query's column names to create the variable names to use:

=> SELECT 'hello' AS var1, 10 AS var2
-> \gset result_
=> \echo :result_var1 :result_var2
hello 10

If a column result is NULL, the corresponding variable is unset rather than being set.

If the query fails or does not return one row, no variables are changed.

If the current query buffer is empty, the most recently sent query is re-executed instead.

\gx [ (option=value [...]) ] [ filename ]
\gx [ (option=value [...]) ] [ |command ]

\gx is equivalent to \g, except that it forces expanded output mode for this query, as if expanded=on were included in the list of \pset options. See also \x.

\h or \help [ command ]

Gives syntax help on the specified SQL command. If command is not specified, then psql will list all the commands for which syntax help is available. If command is an asterisk (*), then syntax help on all SQL commands is shown.

Unlike most other meta-commands, the entire remainder of the line is always taken to be the argument(s) of \help, and neither variable interpolation nor backquote expansion are performed in the arguments.

Note

To simplify typing, commands that consists of several words do not have to be quoted. Thus it is fine to type \help alter table.

\H or \html

Turns on HTML query output format. If the HTML format is already on, it is switched back to the default aligned text format. This command is for compatibility and convenience, but see \pset about setting other output options.

\i or \include filename

Reads input from the file filename and executes it as though it had been typed on the keyboard.

If filename is - (hyphen), then standard input is read until an EOF indication or \q meta-command. This can be used to intersperse interactive input with input from files. Note that Readline behavior will be used only if it is active at the outermost level.

Note

If you want to see the lines on the screen as they are read you must set the variable ECHO to all.

\if expression
\elif expression
\else
\endif

This group of commands implements nestable conditional blocks. A conditional block must begin with an \if and end with an \endif. In between there may be any number of \elif clauses, which may optionally be followed by a single \else clause. Ordinary queries and other types of backslash commands may (and usually do) appear between the commands forming a conditional block.

The \if and \elif commands read their argument(s) and evaluate them as a Boolean expression. If the expression yields true then processing continues normally; otherwise, lines are skipped until a matching \elif, \else, or \endif is reached. Once an \if or \elif test has succeeded, the arguments of later \elif commands in the same block are not evaluated but are treated as false. Lines following an \else are processed only if no earlier matching \if or \elif succeeded.

The expression argument of an \if or \elif command is subject to variable interpolation and backquote expansion, just like any other backslash command argument. After that it is evaluated like the value of an on/off option variable. So a valid value is any unambiguous case-insensitive match for one of: true, false, 1, 0, on, off, yes, no. For example, t, T, and tR will all be considered to be true.

Expressions that do not properly evaluate to true or false will generate a warning and be treated as false.

Lines being skipped are parsed normally to identify queries and backslash commands, but queries are not sent to the server, and backslash commands other than conditionals (\if, \elif, \else, \endif) are ignored. Conditional commands are checked only for valid nesting. Variable references in skipped lines are not expanded, and backquote expansion is not performed either.

All the backslash commands of a given conditional block must appear in the same source file. If EOF is reached on the main input file or an \include-ed file before all local \if-blocks have been closed, then psql will raise an error.

Here is an example:

-- check for the existence of two separate records in the database and store
-- the results in separate psql variables
SELECT
    EXISTS(SELECT 1 FROM customer WHERE customer_id = 123) as is_customer,
    EXISTS(SELECT 1 FROM employee WHERE employee_id = 456) as is_employee
\gset
\if :is_customer
    SELECT * FROM customer WHERE customer_id = 123;
\elif :is_employee
    \echo 'is not a customer but is an employee'
    SELECT * FROM employee WHERE employee_id = 456;
\else
    \if yes
        \echo 'not a customer or employee'
    \else
        \echo 'this will never print'
    \endif
\endif
\ir or \include_relative filename

The \ir command is similar to \i, but resolves relative file names differently. When executing in interactive mode, the two commands behave identically. However, when invoked from a script, \ir interprets file names relative to the directory in which the script is located, rather than the current working directory.

\l[+] or \list[+] [ pattern ]

List the databases in the server and show their names, owners, character set encodings, and access privileges. If pattern is specified, only databases whose names match the pattern are listed. If + is appended to the command name, database sizes, default tablespaces, and descriptions are also displayed. (Size information is only available for databases that the current user can connect to.)

\lo_export loid filename

Reads the large object with OID loid from the database and writes it to filename. Note that this is subtly different from the server function lo_export, which acts with the permissions of the user that the database server runs as and on the server's file system.

Tip

Use \lo_list to find out the large object's OID.

\lo_import filename [ comment ]

Stores the file into a PostgreSQL large object. Optionally, it associates the given comment with the object. Example:

foo=> \lo_import '/home/peter/pictures/photo.xcf' 'a picture of me'
lo_import 152801

The response indicates that the large object received object ID 152801, which can be used to access the newly-created large object in the future. For the sake of readability, it is recommended to always associate a human-readable comment with every object. Both OIDs and comments can be viewed with the \lo_list command.

Note that this command is subtly different from the server-side lo_import because it acts as the local user on the local file system, rather than the server's user and file system.

\lo_list

Shows a list of all PostgreSQL large objects currently stored in the database, along with any comments provided for them.

\lo_unlink loid

Deletes the large object with OID loid from the database.

Tip

Use \lo_list to find out the large object's OID.

\o or \out [ filename ]
\o or \out [ |command ]

Arranges to save future query results to the file filename or pipe future results to the shell command command. If no argument is specified, the query output is reset to the standard output.

If the argument begins with |, then the entire remainder of the line is taken to be the command to execute, and neither variable interpolation nor backquote expansion are performed in it. The rest of the line is simply passed literally to the shell.

“Query results” includes all tables, command responses, and notices obtained from the database server, as well as output of various backslash commands that query the database (such as \d); but not error messages.

Tip

To intersperse text output in between query results, use \qecho.

\p or \print

Print the current query buffer to the standard output. If the current query buffer is empty, the most recently executed query is printed instead.

\password [ username ]

Changes the password of the specified user (by default, the current user). This command prompts for the new password, encrypts it, and sends it to the server as an ALTER ROLE command. This makes sure that the new password does not appear in cleartext in the command history, the server log, or elsewhere.

\prompt [ text ] name

Prompts the user to supply text, which is assigned to the variable name. An optional prompt string, text, can be specified. (For multiword prompts, surround the text with single quotes.)

By default, \prompt uses the terminal for input and output. However, if the -f command line switch was used, \prompt uses standard input and standard output.

\pset [ option [ value ] ]

This command sets options affecting the output of query result tables. option indicates which option is to be set. The semantics of value vary depending on the selected option. For some options, omitting value causes the option to be toggled or unset, as described under the particular option. If no such behavior is mentioned, then omitting value just results in the current setting being displayed.

\pset without any arguments displays the current status of all printing options.

Adjustable printing options are:

border

The value must be a number. In general, the higher the number the more borders and lines the tables will have, but details depend on the particular format. In HTML format, this will translate directly into the border=... attribute. In most other formats only values 0 (no border), 1 (internal dividing lines), and 2 (table frame) make sense, and values above 2 will be treated the same as border = 2. The latex and latex-longtable formats additionally allow a value of 3 to add dividing lines between data rows.

columns

Sets the target width for the wrapped format, and also the width limit for determining whether output is wide enough to require the pager or switch to the vertical display in expanded auto mode. Zero (the default) causes the target width to be controlled by the environment variable COLUMNS, or the detected screen width if COLUMNS is not set. In addition, if columns is zero then the wrapped format only affects screen output. If columns is nonzero then file and pipe output is wrapped to that width as well.

csv_fieldsep

Specifies the field separator to be used in CSV output format. If the separator character appears in a field's value, that field is output within double quotes, following standard CSV rules. The default is a comma.

expanded (or x)

If value is specified it must be either on or off, which will enable or disable expanded mode, or auto. If value is omitted the command toggles between the on and off settings. When expanded mode is enabled, query results are displayed in two columns, with the column name on the left and the data on the right. This mode is useful if the data wouldn't fit on the screen in the normal “horizontal” mode. In the auto setting, the expanded mode is used whenever the query output has more than one column and is wider than the screen; otherwise, the regular mode is used. The auto setting is only effective in the aligned and wrapped formats. In other formats, it always behaves as if the expanded mode is off.

fieldsep

Specifies the field separator to be used in unaligned output format. That way one can create, for example, tab-separated output, which other programs might prefer. To set a tab as field separator, type \pset fieldsep '\t'. The default field separator is '|' (a vertical bar).

fieldsep_zero

Sets the field separator to use in unaligned output format to a zero byte.

footer

If value is specified it must be either on or off which will enable or disable display of the table footer (the (n rows) count). If value is omitted the command toggles footer display on or off.

format

Sets the output format to one of aligned, asciidoc, csv, html, latex, latex-longtable, troff-ms, unaligned, or wrapped. Unique abbreviations are allowed.

aligned format is the standard, human-readable, nicely formatted text output; this is the default.

unaligned format writes all columns of a row on one line, separated by the currently active field separator. This is useful for creating output that might be intended to be read in by other programs, for example, tab-separated or comma-separated format. However, the field separator character is not treated specially if it appears in a column's value; so CSV format may be better suited for such purposes.

csv format writes column values separated by commas, applying the quoting rules described in RFC 4180. This output is compatible with the CSV format of the server's COPY command. A header line with column names is generated unless the tuples_only parameter is on. Titles and footers are not printed. Each row is terminated by the system-dependent end-of-line character, which is typically a single newline (\n) for Unix-like systems or a carriage return and newline sequence (\r\n) for Microsoft Windows. Field separator characters other than comma can be selected with \pset csv_fieldsep.

wrapped format is like aligned but wraps wide data values across lines to make the output fit in the target column width. The target width is determined as described under the columns option. Note that psql will not attempt to wrap column header titles; therefore, wrapped format behaves the same as aligned if the total width needed for column headers exceeds the target.

The asciidoc, html, latex, latex-longtable, and troff-ms formats put out tables that are intended to be included in documents using the respective mark-up language. They are not complete documents! This might not be necessary in HTML, but in LaTeX you must have a complete document wrapper. The latex format uses LaTeX's tabular environment. The latex-longtable format requires the LaTeX longtable and booktabs packages.

linestyle

Sets the border line drawing style to one of ascii, old-ascii, or unicode. Unique abbreviations are allowed. (That would mean one letter is enough.) The default setting is ascii. This option only affects the aligned and wrapped output formats.

ascii style uses plain ASCII characters. Newlines in data are shown using a + symbol in the right-hand margin. When the wrapped format wraps data from one line to the next without a newline character, a dot (.) is shown in the right-hand margin of the first line, and again in the left-hand margin of the following line.

old-ascii style uses plain ASCII characters, using the formatting style used in PostgreSQL 8.4 and earlier. Newlines in data are shown using a : symbol in place of the left-hand column separator. When the data is wrapped from one line to the next without a newline character, a ; symbol is used in place of the left-hand column separator.

unicode style uses Unicode box-drawing characters. Newlines in data are shown using a carriage return symbol in the right-hand margin. When the data is wrapped from one line to the next without a newline character, an ellipsis symbol is shown in the right-hand margin of the first line, and again in the left-hand margin of the following line.

When the border setting is greater than zero, the linestyle option also determines the characters with which the border lines are drawn. Plain ASCII characters work everywhere, but Unicode characters look nicer on displays that recognize them.

null

Sets the string to be printed in place of a null value. The default is to print nothing, which can easily be mistaken for an empty string. For example, one might prefer \pset null '(null)'.

numericlocale

If value is specified it must be either on or off which will enable or disable display of a locale-specific character to separate groups of digits to the left of the decimal marker. If value is omitted the command toggles between regular and locale-specific numeric output.

pager

Controls use of a pager program for query and psql help output. If the environment variable PSQL_PAGER or PAGER is set, the output is piped to the specified program. Otherwise a platform-dependent default program (such as more) is used.

When the pager option is off, the pager program is not used. When the pager option is on, the pager is used when appropriate, i.e., when the output is to a terminal and will not fit on the screen. The pager option can also be set to always, which causes the pager to be used for all terminal output regardless of whether it fits on the screen. \pset pager without a value toggles pager use on and off.

pager_min_lines

If pager_min_lines is set to a number greater than the page height, the pager program will not be called unless there are at least this many lines of output to show. The default setting is 0.

recordsep

Specifies the record (line) separator to use in unaligned output format. The default is a newline character.

recordsep_zero

Sets the record separator to use in unaligned output format to a zero byte.

tableattr (or T)

In HTML format, this specifies attributes to be placed inside the table tag. This could for example be cellpadding or bgcolor. Note that you probably don't want to specify border here, as that is already taken care of by \pset border. If no value is given, the table attributes are unset.

In latex-longtable format, this controls the proportional width of each column containing a left-aligned data type. It is specified as a whitespace-separated list of values, e.g., '0.2 0.2 0.6'. Unspecified output columns use the last specified value.

title (or C)

Sets the table title for any subsequently printed tables. This can be used to give your output descriptive tags. If no value is given, the title is unset.

tuples_only (or t)

If value is specified it must be either on or off which will enable or disable tuples-only mode. If value is omitted the command toggles between regular and tuples-only output. Regular output includes extra information such as column headers, titles, and various footers. In tuples-only mode, only actual table data is shown.

unicode_border_linestyle

Sets the border drawing style for the unicode line style to one of single or double.

unicode_column_linestyle

Sets the column drawing style for the unicode line style to one of single or double.

unicode_header_linestyle

Sets the header drawing style for the unicode line style to one of single or double.

Illustrations of how these different formats look can be seen in Examples, below.

Tip

There are various shortcut commands for \pset. See \a, \C, \f, \H, \t, \T, and \x.

\q or \quit

Quits the psql program. In a script file, only execution of that script is terminated.

\qecho text [ ... ]

This command is identical to \echo except that the output will be written to the query output channel, as set by \o.

\r or \reset

Resets (clears) the query buffer.

\s [ filename ]

Print psql's command line history to filename. If filename is omitted, the history is written to the standard output (using the pager if appropriate). This command is not available if psql was built without Readline support.

\set [ name [ value [ ... ] ] ]

Sets the psql variable name to value, or if more than one value is given, to the concatenation of all of them. If only one argument is given, the variable is set to an empty-string value. To unset a variable, use the \unset command.

\set without any arguments displays the names and values of all currently-set psql variables.

Valid variable names can contain letters, digits, and underscores. See Variables below for details. Variable names are case-sensitive.

Certain variables are special, in that they control psql's behavior or are automatically set to reflect connection state. These variables are documented in Variables, below.

Note

This command is unrelated to the SQL command SET.

\setenv name [ value ]

Sets the environment variable name to value, or if the value is not supplied, unsets the environment variable. Example:

testdb=> \setenv PAGER less
testdb=> \setenv LESS -imx4F
\sf[+] function_description

This command fetches and shows the definition of the named function or procedure, in the form of a CREATE OR REPLACE FUNCTION or CREATE OR REPLACE PROCEDURE command. The definition is printed to the current query output channel, as set by \o.

The target function can be specified by name alone, or by name and arguments, for example foo(integer, text). The argument types must be given if there is more than one function of the same name.

If + is appended to the command name, then the output lines are numbered, with the first line of the function body being line 1.

Unlike most other meta-commands, the entire remainder of the line is always taken to be the argument(s) of \sf, and neither variable interpolation nor backquote expansion are performed in the arguments.

\sv[+] view_name

This command fetches and shows the definition of the named view, in the form of a CREATE OR REPLACE VIEW command. The definition is printed to the current query output channel, as set by \o.

If + is appended to the command name, then the output lines are numbered from 1.

Unlike most other meta-commands, the entire remainder of the line is always taken to be the argument(s) of \sv, and neither variable interpolation nor backquote expansion are performed in the arguments.

\t

Toggles the display of output column name headings and row count footer. This command is equivalent to \pset tuples_only and is provided for convenience.

\T table_options

Specifies attributes to be placed within the table tag in HTML output format. This command is equivalent to \pset tableattr table_options.

\timing [ on | off ]

With a parameter, turns displaying of how long each SQL statement takes on or off. Without a parameter, toggles the display between on and off. The display is in milliseconds; intervals longer than 1 second are also shown in minutes:seconds format, with hours and days fields added if needed.

\unset name

Unsets (deletes) the psql variable name.

Most variables that control psql's behavior cannot be unset; instead, an \unset command is interpreted as setting them to their default values. See Variables below.

\w or \write filename
\w or \write |command

Writes the current query buffer to the file filename or pipes it to the shell command command. If the current query buffer is empty, the most recently executed query is written instead.

If the argument begins with |, then the entire remainder of the line is taken to be the command to execute, and neither variable interpolation nor backquote expansion are performed in it. The rest of the line is simply passed literally to the shell.

\warn text [ ... ]

This command is identical to \echo except that the output will be written to psql's standard error channel, rather than standard output.

\watch [ seconds ]

Repeatedly execute the current query buffer (as \g does) until interrupted or the query fails. Wait the specified number of seconds (default 2) between executions. Each query result is displayed with a header that includes the \pset title string (if any), the time as of query start, and the delay interval.

If the current query buffer is empty, the most recently sent query is re-executed instead.

\x [ on | off | auto ]

Sets or toggles expanded table formatting mode. As such it is equivalent to \pset expanded.

\z [ pattern ]

Lists tables, views and sequences with their associated access privileges. If a pattern is specified, only tables, views and sequences whose names match the pattern are listed.

This is an alias for \dp (“display privileges”).

\! [ command ]

With no argument, escapes to a sub-shell; psql resumes when the sub-shell exits. With an argument, executes the shell command command.

Unlike most other meta-commands, the entire remainder of the line is always taken to be the argument(s) of \!, and neither variable interpolation nor backquote expansion are performed in the arguments. The rest of the line is simply passed literally to the shell.

\? [ topic ]

Shows help information. The optional topic parameter (defaulting to commands) selects which part of psql is explained: commands describes psql's backslash commands; options describes the command-line options that can be passed to psql; and variables shows help about psql configuration variables.

\;

Backslash-semicolon is not a meta-command in the same way as the preceding commands; rather, it simply causes a semicolon to be added to the query buffer without any further processing.

Normally, psql will dispatch an SQL command to the server as soon as it reaches the command-ending semicolon, even if more input remains on the current line. Thus for example entering

select 1; select 2; select 3;

will result in the three SQL commands being individually sent to the server, with each one's results being displayed before continuing to the next command. However, a semicolon entered as \; will not trigger command processing, so that the command before it and the one after are effectively combined and sent to the server in one request. So for example

select 1\; select 2\; select 3;

results in sending the three SQL commands to the server in a single request, when the non-backslashed semicolon is reached. The server executes such a request as a single transaction, unless there are explicit BEGIN/COMMIT commands included in the string to divide it into multiple transactions. (See Section 53.2.2.1 for more details about how the server handles multi-query strings.) psql prints only the last query result it receives for each request; in this example, although all three SELECTs are indeed executed, psql only prints the 3.

Patterns

The various \d commands accept a pattern parameter to specify the object name(s) to be displayed. In the simplest case, a pattern is just the exact name of the object. The characters within a pattern are normally folded to lower case, just as in SQL names; for example, \dt FOO will display the table named foo. As in SQL names, placing double quotes around a pattern stops folding to lower case. Should you need to include an actual double quote character in a pattern, write it as a pair of double quotes within a double-quote sequence; again this is in accord with the rules for SQL quoted identifiers. For example, \dt "FOO""BAR" will display the table named FOO"BAR (not foo"bar). Unlike the normal rules for SQL names, you can put double quotes around just part of a pattern, for instance \dt FOO"FOO"BAR will display the table named fooFOObar.

Whenever the pattern parameter is omitted completely, the \d commands display all objects that are visible in the current schema search path — this is equivalent to using * as the pattern. (An object is said to be visible if its containing schema is in the search path and no object of the same kind and name appears earlier in the search path. This is equivalent to the statement that the object can be referenced by name without explicit schema qualification.) To see all objects in the database regardless of visibility, use *.* as the pattern.

Within a pattern, * matches any sequence of characters (including no characters) and ? matches any single character. (This notation is comparable to Unix shell file name patterns.) For example, \dt int* displays tables whose names begin with int. But within double quotes, * and ? lose these special meanings and are just matched literally.

A relation pattern that contains a dot (.) is interpreted as a schema name pattern followed by an object name pattern. For example, \dt foo*.*bar* displays all tables whose table name includes bar that are in schemas whose schema name starts with foo. When no dot appears, then the pattern matches only objects that are visible in the current schema search path. Again, a dot within double quotes loses its special meaning and is matched literally. A relation pattern that contains two dots (.) is interpreted as a database name followed by a schema name pattern followed by an object name pattern. The database name portion will not be treated as a pattern and must match the name of the currently connected database, else an error will be raised.

A schema pattern that contains a dot (.) is interpreted as a database name followed by a schema name pattern. For example, \dn mydb.*foo* displays all schemas whose schema name includes foo. The database name portion will not be treated as a pattern and must match the name of the currently connected database, else an error will be raised.

Advanced users can use regular-expression notations such as character classes, for example [0-9] to match any digit. All regular expression special characters work as specified in Section 9.7.3, except for . which is taken as a separator as mentioned above, * which is translated to the regular-expression notation .*, ? which is translated to ., and $ which is matched literally. You can emulate these pattern characters at need by writing ? for ., (R+|) for R*, or (R|) for R?. $ is not needed as a regular-expression character since the pattern must match the whole name, unlike the usual interpretation of regular expressions (in other words, $ is automatically appended to your pattern). Write * at the beginning and/or end if you don't wish the pattern to be anchored. Note that within double quotes, all regular expression special characters lose their special meanings and are matched literally. Also, the regular expression special characters are matched literally in operator name patterns (i.e., the argument of \do).

Advanced Features

Variables

psql provides variable substitution features similar to common Unix command shells. Variables are simply name/value pairs, where the value can be any string of any length. The name must consist of letters (including non-Latin letters), digits, and underscores.

To set a variable, use the psql meta-command \set. For example,

testdb=> \set foo bar

sets the variable foo to the value bar. To retrieve the content of the variable, precede the name with a colon, for example:

testdb=> \echo :foo
bar

This works in both regular SQL commands and meta-commands; there is more detail in SQL Interpolation, below.

If you call \set without a second argument, the variable is set to an empty-string value. To unset (i.e., delete) a variable, use the command \unset. To show the values of all variables, call \set without any argument.

Note

The arguments of \set are subject to the same substitution rules as with other commands. Thus you can construct interesting references such as \set :foo 'something' and get “soft links” or “variable variables” of Perl or PHP fame, respectively. Unfortunately (or fortunately?), there is no way to do anything useful with these constructs. On the other hand, \set bar :foo is a perfectly valid way to copy a variable.

A number of these variables are treated specially by psql. They represent certain option settings that can be changed at run time by altering the value of the variable, or in some cases represent changeable state of psql. By convention, all specially treated variables' names consist of all upper-case ASCII letters (and possibly digits and underscores). To ensure maximum compatibility in the future, avoid using such variable names for your own purposes.

Variables that control psql's behavior generally cannot be unset or set to invalid values. An \unset command is allowed but is interpreted as setting the variable to its default value. A \set command without a second argument is interpreted as setting the variable to on, for control variables that accept that value, and is rejected for others. Also, control variables that accept the values on and off will also accept other common spellings of Boolean values, such as true and false.

The specially treated variables are:

AUTOCOMMIT

When on (the default), each SQL command is automatically committed upon successful completion. To postpone commit in this mode, you must enter a BEGIN or START TRANSACTION SQL command. When off or unset, SQL commands are not committed until you explicitly issue COMMIT or END. The autocommit-off mode works by issuing an implicit BEGIN for you, just before any command that is not already in a transaction block and is not itself a BEGIN or other transaction-control command, nor a command that cannot be executed inside a transaction block (such as VACUUM).

Note

In autocommit-off mode, you must explicitly abandon any failed transaction by entering ABORT or ROLLBACK. Also keep in mind that if you exit the session without committing, your work will be lost.

Note

The autocommit-on mode is PostgreSQL's traditional behavior, but autocommit-off is closer to the SQL spec. If you prefer autocommit-off, you might wish to set it in the system-wide psqlrc file or your ~/.psqlrc file.

COMP_KEYWORD_CASE

Determines which letter case to use when completing an SQL key word. If set to lower or upper, the completed word will be in lower or upper case, respectively. If set to preserve-lower or preserve-upper (the default), the completed word will be in the case of the word already entered, but words being completed without anything entered will be in lower or upper case, respectively.

DBNAME

The name of the database you are currently connected to. This is set every time you connect to a database (including program start-up), but can be changed or unset.

ECHO

If set to all, all nonempty input lines are printed to standard output as they are read. (This does not apply to lines read interactively.) To select this behavior on program start-up, use the switch -a. If set to queries, psql prints each query to standard output as it is sent to the server. The switch to select this behavior is -e. If set to errors, then only failed queries are displayed on standard error output. The switch for this behavior is -b. If set to none (the default), then no queries are displayed.

ECHO_HIDDEN

When this variable is set to on and a backslash command queries the database, the query is first shown. This feature helps you to study PostgreSQL internals and provide similar functionality in your own programs. (To select this behavior on program start-up, use the switch -E.) If you set this variable to the value noexec, the queries are just shown but are not actually sent to the server and executed. The default value is off.

ENCODING

The current client character set encoding. This is set every time you connect to a database (including program start-up), and when you change the encoding with \encoding, but it can be changed or unset.

ERROR

true if the last SQL query failed, false if it succeeded. See also SQLSTATE.

FETCH_COUNT

If this variable is set to an integer value greater than zero, the results of SELECT queries are fetched and displayed in groups of that many rows, rather than the default behavior of collecting the entire result set before display. Therefore only a limited amount of memory is used, regardless of the size of the result set. Settings of 100 to 1000 are commonly used when enabling this feature. Keep in mind that when using this feature, a query might fail after having already displayed some rows.

Tip

Although you can use any output format with this feature, the default aligned format tends to look bad because each group of FETCH_COUNT rows will be formatted separately, leading to varying column widths across the row groups. The other output formats work better.

HIDE_TABLEAM

If this variable is set to true, a table's access method details are not displayed. This is mainly useful for regression tests.

HIDE_TOAST_COMPRESSION

If this variable is set to true, column compression method details are not displayed. This is mainly useful for regression tests.

HISTCONTROL

If this variable is set to ignorespace, lines which begin with a space are not entered into the history list. If set to a value of ignoredups, lines matching the previous history line are not entered. A value of ignoreboth combines the two options. If set to none (the default), all lines read in interactive mode are saved on the history list.

Note

This feature was shamelessly plagiarized from Bash.

HISTFILE

The file name that will be used to store the history list. If unset, the file name is taken from the PSQL_HISTORY environment variable. If that is not set either, the default is ~/.psql_history, or %APPDATA%\postgresql\psql_history on Windows. For example, putting:

\set HISTFILE ~/.psql_history-:DBNAME

in ~/.psqlrc will cause psql to maintain a separate history for each database.

Note

This feature was shamelessly plagiarized from Bash.

HISTSIZE

The maximum number of commands to store in the command history (default 500). If set to a negative value, no limit is applied.

Note

This feature was shamelessly plagiarized from Bash.

HOST

The database server host you are currently connected to. This is set every time you connect to a database (including program start-up), but can be changed or unset.

IGNOREEOF

If set to 1 or less, sending an EOF character (usually Control+D) to an interactive session of psql will terminate the application. If set to a larger numeric value, that many consecutive EOF characters must be typed to make an interactive session terminate. If the variable is set to a non-numeric value, it is interpreted as 10. The default is 0.

Note

This feature was shamelessly plagiarized from Bash.

LASTOID

The value of the last affected OID, as returned from an INSERT or \lo_import command. This variable is only guaranteed to be valid until after the result of the next SQL command has been displayed. PostgreSQL servers since version 12 do not support OID system columns anymore, thus LASTOID will always be 0 following INSERT when targeting such servers.

LAST_ERROR_MESSAGE
LAST_ERROR_SQLSTATE

The primary error message and associated SQLSTATE code for the most recent failed query in the current psql session, or an empty string and 00000 if no error has occurred in the current session.

ON_ERROR_ROLLBACK

When set to on, if a statement in a transaction block generates an error, the error is ignored and the transaction continues. When set to interactive, such errors are only ignored in interactive sessions, and not when reading script files. When set to off (the default), a statement in a transaction block that generates an error aborts the entire transaction. The error rollback mode works by issuing an implicit SAVEPOINT for you, just before each command that is in a transaction block, and then rolling back to the savepoint if the command fails.

ON_ERROR_STOP

By default, command processing continues after an error. When this variable is set to on, processing will instead stop immediately. In interactive mode, psql will return to the command prompt; otherwise, psql will exit, returning error code 3 to distinguish this case from fatal error conditions, which are reported using error code 1. In either case, any currently running scripts (the top-level script, if any, and any other scripts which it may have in invoked) will be terminated immediately. If the top-level command string contained multiple SQL commands, processing will stop with the current command.

PORT

The database server port to which you are currently connected. This is set every time you connect to a database (including program start-up), but can be changed or unset.

PROMPT1
PROMPT2
PROMPT3

These specify what the prompts psql issues should look like. See Prompting below.

QUIET

Setting this variable to on is equivalent to the command line option -q. It is probably not too useful in interactive mode.

ROW_COUNT

The number of rows returned or affected by the last SQL query, or 0 if the query failed or did not report a row count.

SERVER_VERSION_NAME
SERVER_VERSION_NUM

The server's version number as a string, for example 9.6.2, 10.1 or 11beta1, and in numeric form, for example 90602 or 100001. These are set every time you connect to a database (including program start-up), but can be changed or unset.

SHOW_CONTEXT

This variable can be set to the values never, errors, or always to control whether CONTEXT fields are displayed in messages from the server. The default is errors (meaning that context will be shown in error messages, but not in notice or warning messages). This setting has no effect when VERBOSITY is set to terse or sqlstate. (See also \errverbose, for use when you want a verbose version of the error you just got.)

SINGLELINE

Setting this variable to on is equivalent to the command line option -S.

SINGLESTEP

Setting this variable to on is equivalent to the command line option -s.

SQLSTATE

The error code (see Appendix A) associated with the last SQL query's failure, or 00000 if it succeeded.

USER

The database user you are currently connected as. This is set every time you connect to a database (including program start-up), but can be changed or unset.

VERBOSITY

This variable can be set to the values default, verbose, terse, or sqlstate to control the verbosity of error reports. (See also \errverbose, for use when you want a verbose version of the error you just got.)

VERSION
VERSION_NAME
VERSION_NUM

These variables are set at program start-up to reflect psql's version, respectively as a verbose string, a short string (e.g., 9.6.2, 10.1, or 11beta1), and a number (e.g., 90602 or 100001). They can be changed or unset.

SQL Interpolation

A key feature of psql variables is that you can substitute (“interpolate”) them into regular SQL statements, as well as the arguments of meta-commands. Furthermore, psql provides facilities for ensuring that variable values used as SQL literals and identifiers are properly quoted. The syntax for interpolating a value without any quoting is to prepend the variable name with a colon (:). For example,

testdb=> \set foo 'my_table'
testdb=> SELECT * FROM :foo;

would query the table my_table. Note that this may be unsafe: the value of the variable is copied literally, so it can contain unbalanced quotes, or even backslash commands. You must make sure that it makes sense where you put it.

When a value is to be used as an SQL literal or identifier, it is safest to arrange for it to be quoted. To quote the value of a variable as an SQL literal, write a colon followed by the variable name in single quotes. To quote the value as an SQL identifier, write a colon followed by the variable name in double quotes. These constructs deal correctly with quotes and other special characters embedded within the variable value. The previous example would be more safely written this way:

testdb=> \set foo 'my_table'
testdb=> SELECT * FROM :"foo";

Variable interpolation will not be performed within quoted SQL literals and identifiers. Therefore, a construction such as ':foo' doesn't work to produce a quoted literal from a variable's value (and it would be unsafe if it did work, since it wouldn't correctly handle quotes embedded in the value).

One example use of this mechanism is to copy the contents of a file into a table column. First load the file into a variable and then interpolate the variable's value as a quoted string:

testdb=> \set content `cat my_file.txt`
testdb=> INSERT INTO my_table VALUES (:'content');

(Note that this still won't work if my_file.txt contains NUL bytes. psql does not support embedded NUL bytes in variable values.)

Since colons can legally appear in SQL commands, an apparent attempt at interpolation (that is, :name, :'name', or :"name") is not replaced unless the named variable is currently set. In any case, you can escape a colon with a backslash to protect it from substitution.

The :{?name} special syntax returns TRUE or FALSE depending on whether the variable exists or not, and is thus always substituted, unless the colon is backslash-escaped.

The colon syntax for variables is standard SQL for embedded query languages, such as ECPG. The colon syntaxes for array slices and type casts are PostgreSQL extensions, which can sometimes conflict with the standard usage. The colon-quote syntax for escaping a variable's value as an SQL literal or identifier is a psql extension.

Prompting

The prompts psql issues can be customized to your preference. The three variables PROMPT1, PROMPT2, and PROMPT3 contain strings and special escape sequences that describe the appearance of the prompt. Prompt 1 is the normal prompt that is issued when psql requests a new command. Prompt 2 is issued when more input is expected during command entry, for example because the command was not terminated with a semicolon or a quote was not closed. Prompt 3 is issued when you are running an SQL COPY FROM STDIN command and you need to type in a row value on the terminal.

The value of the selected prompt variable is printed literally, except where a percent sign (%) is encountered. Depending on the next character, certain other text is substituted instead. Defined substitutions are:

%M

The full host name (with domain name) of the database server, or [local] if the connection is over a Unix domain socket, or [local:/dir/name], if the Unix domain socket is not at the compiled in default location.

%m

The host name of the database server, truncated at the first dot, or [local] if the connection is over a Unix domain socket.

%>

The port number at which the database server is listening.

%n

The database session user name. (The expansion of this value might change during a database session as the result of the command SET SESSION AUTHORIZATION.)

%/

The name of the current database.

%~

Like %/, but the output is ~ (tilde) if the database is your default database.

%#

If the session user is a database superuser, then a #, otherwise a >. (The expansion of this value might change during a database session as the result of the command SET SESSION AUTHORIZATION.)

%p

The process ID of the backend currently connected to.

%R

In prompt 1 normally =, but @ if the session is in an inactive branch of a conditional block, or ^ if in single-line mode, or ! if the session is disconnected from the database (which can happen if \connect fails). In prompt 2 %R is replaced by a character that depends on why psql expects more input: - if the command simply wasn't terminated yet, but * if there is an unfinished /* ... */ comment, a single quote if there is an unfinished quoted string, a double quote if there is an unfinished quoted identifier, a dollar sign if there is an unfinished dollar-quoted string, or ( if there is an unmatched left parenthesis. In prompt 3 %R doesn't produce anything.

%x

Transaction status: an empty string when not in a transaction block, or * when in a transaction block, or ! when in a failed transaction block, or ? when the transaction state is indeterminate (for example, because there is no connection).

%l

The line number inside the current statement, starting from 1.

%digits

The character with the indicated octal code is substituted.

%:name:

The value of the psql variable name. See Variables, above, for details.

%`command`

The output of command, similar to ordinary “back-tick” substitution.

%[ ... %]

Prompts can contain terminal control characters which, for example, change the color, background, or style of the prompt text, or change the title of the terminal window. In order for the line editing features of Readline to work properly, these non-printing control characters must be designated as invisible by surrounding them with %[ and %]. Multiple pairs of these can occur within the prompt. For example:

testdb=> \set PROMPT1 '%[%033[1;33;40m%]%n@%/%R%[%033[0m%]%# '

results in a boldfaced (1;) yellow-on-black (33;40) prompt on VT100-compatible, color-capable terminals.

%w

Whitespace of the same width as the most recent output of PROMPT1. This can be used as a PROMPT2 setting, so that multi-line statements are aligned with the first line, but there is no visible secondary prompt.

To insert a percent sign into your prompt, write %%. The default prompts are '%/%R%x%# ' for prompts 1 and 2, and '>> ' for prompt 3.

Note

This feature was shamelessly plagiarized from tcsh.

Command-Line Editing

psql supports the Readline library for convenient line editing and retrieval. The command history is automatically saved when psql exits and is reloaded when psql starts up. Tab-completion is also supported, although the completion logic makes no claim to be an SQL parser. The queries generated by tab-completion can also interfere with other SQL commands, e.g., SET TRANSACTION ISOLATION LEVEL. If for some reason you do not like the tab completion, you can turn it off by putting this in a file named .inputrc in your home directory:

$if psql
set disable-completion on
$endif

(This is not a psql but a Readline feature. Read its documentation for further details.)

Environment

COLUMNS

If \pset columns is zero, controls the width for the wrapped format and width for determining if wide output requires the pager or should be switched to the vertical format in expanded auto mode.

PGDATABASE
PGHOST
PGPORT
PGUSER

Default connection parameters (see Section 34.15).

PG_COLOR

Specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

PSQL_EDITOR
EDITOR
VISUAL

Editor used by the \e, \ef, and \ev commands. These variables are examined in the order listed; the first that is set is used. If none of them is set, the default is to use vi on Unix systems or notepad.exe on Windows systems.

PSQL_EDITOR_LINENUMBER_ARG

When \e, \ef, or \ev is used with a line number argument, this variable specifies the command-line argument used to pass the starting line number to the user's editor. For editors such as Emacs or vi, this is a plus sign. Include a trailing space in the value of the variable if there needs to be space between the option name and the line number. Examples:

PSQL_EDITOR_LINENUMBER_ARG='+'
PSQL_EDITOR_LINENUMBER_ARG='--line '

The default is + on Unix systems (corresponding to the default editor vi, and useful for many other common editors); but there is no default on Windows systems.

PSQL_HISTORY

Alternative location for the command history file. Tilde (~) expansion is performed.

PSQL_PAGER
PAGER

If a query's results do not fit on the screen, they are piped through this command. Typical values are more or less. Use of the pager can be disabled by setting PSQL_PAGER or PAGER to an empty string, or by adjusting the pager-related options of the \pset command. These variables are examined in the order listed; the first that is set is used. If none of them is set, the default is to use more on most platforms, but less on Cygwin.

PSQLRC

Alternative location of the user's .psqlrc file. Tilde (~) expansion is performed.

SHELL

Command executed by the \! command.

TMPDIR

Directory for storing temporary files. The default is /tmp.

This utility, like most other PostgreSQL utilities, also uses the environment variables supported by libpq (see Section 34.15).

Files

psqlrc and ~/.psqlrc

Unless it is passed an -X option, psql attempts to read and execute commands from the system-wide startup file (psqlrc) and then the user's personal startup file (~/.psqlrc), after connecting to the database but before accepting normal commands. These files can be used to set up the client and/or the server to taste, typically with \set and SET commands.

The system-wide startup file is named psqlrc and is sought in the installation's “system configuration” directory, which is most reliably identified by running pg_config --sysconfdir. By default this directory will be ../etc/ relative to the directory containing the PostgreSQL executables. The name of this directory can be set explicitly via the PGSYSCONFDIR environment variable.

The user's personal startup file is named .psqlrc and is sought in the invoking user's home directory. On Windows, which lacks such a concept, the personal startup file is named %APPDATA%\postgresql\psqlrc.conf. The location of the user's startup file can be set explicitly via the PSQLRC environment variable.

Both the system-wide startup file and the user's personal startup file can be made psql-version-specific by appending a dash and the PostgreSQL major or minor release number to the file name, for example ~/.psqlrc-9.2 or ~/.psqlrc-9.2.5. The most specific version-matching file will be read in preference to a non-version-specific file.

.psql_history

The command-line history is stored in the file ~/.psql_history, or %APPDATA%\postgresql\psql_history on Windows.

The location of the history file can be set explicitly via the HISTFILE psql variable or the PSQL_HISTORY environment variable.

Notes

  • psql works best with servers of the same or an older major version. Backslash commands are particularly likely to fail if the server is of a newer version than psql itself. However, backslash commands of the \d family should work with servers of versions back to 7.4, though not necessarily with servers newer than psql itself. The general functionality of running SQL commands and displaying query results should also work with servers of a newer major version, but this cannot be guaranteed in all cases.

    If you want to use psql to connect to several servers of different major versions, it is recommended that you use the newest version of psql. Alternatively, you can keep around a copy of psql from each major version and be sure to use the version that matches the respective server. But in practice, this additional complication should not be necessary.

  • Before PostgreSQL 9.6, the -c option implied -X (--no-psqlrc); this is no longer the case.

  • Before PostgreSQL 8.4, psql allowed the first argument of a single-letter backslash command to start directly after the command, without intervening whitespace. Now, some whitespace is required.

Notes for Windows Users

psql is built as a “console application”. Since the Windows console windows use a different encoding than the rest of the system, you must take special care when using 8-bit characters within psql. If psql detects a problematic console code page, it will warn you at startup. To change the console code page, two things are necessary:

  • Set the code page by entering cmd.exe /c chcp 1252. (1252 is a code page that is appropriate for German; replace it with your value.) If you are using Cygwin, you can put this command in /etc/profile.

  • Set the console font to Lucida Console, because the raster font does not work with the ANSI code page.

Examples

The first example shows how to spread a command over several lines of input. Notice the changing prompt:

testdb=> CREATE TABLE my_table (
testdb(>  first integer not null default 0,
testdb(>  second text)
testdb-> ;
CREATE TABLE

Now look at the table definition again:

testdb=> \d my_table
              Table "public.my_table"
 Column |  Type   | Collation | Nullable | Default
--------+---------+-----------+----------+---------
 first  | integer |           | not null | 0
 second | text    |           |          |

Now we change the prompt to something more interesting:

testdb=> \set PROMPT1 '%n@%m %~%R%# '
peter@localhost testdb=>

Let's assume you have filled the table with data and want to take a look at it:

peter@localhost testdb=> SELECT * FROM my_table;
 first | second
-------+--------
     1 | one
     2 | two
     3 | three
     4 | four
(4 rows)

You can display tables in different ways by using the \pset command:

peter@localhost testdb=> \pset border 2
Border style is 2.
peter@localhost testdb=> SELECT * FROM my_table;
+-------+--------+
| first | second |
+-------+--------+
|     1 | one    |
|     2 | two    |
|     3 | three  |
|     4 | four   |
+-------+--------+
(4 rows)

peter@localhost testdb=> \pset border 0
Border style is 0.
peter@localhost testdb=> SELECT * FROM my_table;
first second
----- ------
    1 one
    2 two
    3 three
    4 four
(4 rows)

peter@localhost testdb=> \pset border 1
Border style is 1.
peter@localhost testdb=> \pset format csv
Output format is csv.
peter@localhost testdb=> \pset tuples_only
Tuples only is on.
peter@localhost testdb=> SELECT second, first FROM my_table;
one,1
two,2
three,3
four,4
peter@localhost testdb=> \pset format unaligned
Output format is unaligned.
peter@localhost testdb=> \pset fieldsep '\t'
Field separator is "    ".
peter@localhost testdb=> SELECT second, first FROM my_table;
one     1
two     2
three   3
four    4

Alternatively, use the short commands:

peter@localhost testdb=> \a \t \x
Output format is aligned.
Tuples only is off.
Expanded display is on.
peter@localhost testdb=> SELECT * FROM my_table;
-[ RECORD 1 ]-
first  | 1
second | one
-[ RECORD 2 ]-
first  | 2
second | two
-[ RECORD 3 ]-
first  | 3
second | three
-[ RECORD 4 ]-
first  | 4
second | four

Also, these output format options can be set for just one query by using \g:

peter@localhost testdb=> SELECT * FROM my_table
peter@localhost testdb-> \g (format=aligned tuples_only=off expanded=on)
-[ RECORD 1 ]-
first  | 1
second | one
-[ RECORD 2 ]-
first  | 2
second | two
-[ RECORD 3 ]-
first  | 3
second | three
-[ RECORD 4 ]-
first  | 4
second | four

Here is an example of using the \df command to find only functions with names matching int*pl and whose second argument is of type bigint:

testdb=> \df int*pl * bigint
                          List of functions
   Schema   |  Name   | Result data type | Argument data types | Type
------------+---------+------------------+---------------------+------
 pg_catalog | int28pl | bigint           | smallint, bigint    | func
 pg_catalog | int48pl | bigint           | integer, bigint     | func
 pg_catalog | int8pl  | bigint           | bigint, bigint      | func
(3 rows)

When suitable, query results can be shown in a crosstab representation with the \crosstabview command:

testdb=> SELECT first, second, first > 2 AS gt2 FROM my_table;
 first | second | gt2
-------+--------+-----
     1 | one    | f
     2 | two    | f
     3 | three  | t
     4 | four   | t
(4 rows)

testdb=> \crosstabview first second
 first | one | two | three | four
-------+-----+-----+-------+------
     1 | f   |     |       |
     2 |     | f   |       |
     3 |     |     | t     |
     4 |     |     |       | t
(4 rows)

This second example shows a multiplication table with rows sorted in reverse numerical order and columns with an independent, ascending numerical order.

testdb=> SELECT t1.first as "A", t2.first+100 AS "B", t1.first*(t2.first+100) as "AxB",
testdb(> row_number() over(order by t2.first) AS ord
testdb(> FROM my_table t1 CROSS JOIN my_table t2 ORDER BY 1 DESC
testdb(> \crosstabview "A" "B" "AxB" ord
 A | 101 | 102 | 103 | 104
---+-----+-----+-----+-----
 4 | 404 | 408 | 412 | 416
 3 | 303 | 306 | 309 | 312
 2 | 202 | 204 | 206 | 208
 1 | 101 | 102 | 103 | 104
(4 rows)

Prev | Up | Next
pg_verifybackup | Home | reindexdb


reindexdb
Prev Up PostgreSQL Client Applications Home Next

reindexdb

reindexdb — reindex a PostgreSQL database

Synopsis

reindexdb [connection-option...] [option...] [ -S | --schema schema ] ... [ -t | --table table ] ... [ -i | --index index ] ... [dbname]

reindexdb [connection-option...] [option...] -a | --all

reindexdb [connection-option...] [option...] -s | --system [dbname]

Description

reindexdb is a utility for rebuilding indexes in a PostgreSQL database.

reindexdb is a wrapper around the SQL command REINDEX. There is no effective difference between reindexing databases via this utility and via other methods for accessing the server.

Options

reindexdb accepts the following command-line arguments:

-a
--all

Reindex all databases.

--concurrently

Use the CONCURRENTLY option. See REINDEX, where all the caveats of this option are explained in detail.

[-d] dbname
[--dbname=]dbname

Specifies the name of the database to be reindexed, when -a/--all is not used. If this is not specified, the database name is read from the environment variable PGDATABASE. If that is not set, the user name specified for the connection is used. The dbname can be a connection string. If so, connection string parameters will override any conflicting command line options.

-e
--echo

Echo the commands that reindexdb generates and sends to the server.

-i index
--index=index

Recreate index only. Multiple indexes can be recreated by writing multiple -i switches.

-j njobs
--jobs=njobs

Execute the reindex commands in parallel by running njobs commands simultaneously. This option may reduce the processing time but it also increases the load on the database server.

reindexdb will open njobs connections to the database, so make sure your max_connections setting is high enough to accommodate all connections.

Note that this option is incompatible with the --index and --system options.

-q
--quiet

Do not display progress messages.

-s
--system

Reindex database's system catalogs only.

-S schema
--schema=schema

Reindex schema only. Multiple schemas can be reindexed by writing multiple -S switches.

-t table
--table=table

Reindex table only. Multiple tables can be reindexed by writing multiple -t switches.

--tablespace=tablespace

Specifies the tablespace where indexes are rebuilt. (This name is processed as a double-quoted identifier.)

-v
--verbose

Print detailed information during processing.

-V
--version

Print the reindexdb version and exit.

-?
--help

Show help about reindexdb command line arguments, and exit.

reindexdb also accepts the following command-line arguments for connection parameters:

-h host
--host=host

Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for the Unix domain socket.

-p port
--port=port

Specifies the TCP port or local Unix domain socket file extension on which the server is listening for connections.

-U username
--username=username

User name to connect as.

-w
--no-password

Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.

-W
--password

Force reindexdb to prompt for a password before connecting to a database.

This option is never essential, since reindexdb will automatically prompt for a password if the server demands password authentication. However, reindexdb will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt.

--maintenance-db=dbname

Specifies the name of the database to connect to to discover which databases should be reindexed, when -a/--all is used. If not specified, the postgres database will be used, or if that does not exist, template1 will be used. This can be a connection string. If so, connection string parameters will override any conflicting command line options. Also, connection string parameters other than the database name itself will be re-used when connecting to other databases.

Environment

PGDATABASE
PGHOST
PGPORT
PGUSER

Default connection parameters

PG_COLOR

Specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

This utility, like most other PostgreSQL utilities, also uses the environment variables supported by libpq (see Section 34.15).

Diagnostics

In case of difficulty, see REINDEX and psql for discussions of potential problems and error messages. The database server must be running at the targeted host. Also, any default connection settings and environment variables used by the libpq front-end library will apply.

Notes

reindexdb might need to connect several times to the PostgreSQL server, asking for a password each time. It is convenient to have a ~/.pgpass file in such cases. See Section 34.16 for more information.

Examples

To reindex the database test:

$ reindexdb test

To reindex the table foo and the index bar in a database named abcd:

$ reindexdb --table=foo --index=bar abcd

See Also

REINDEX
Prev | Up | Next
psql | Home | vacuumdb


vacuumdb
Prev Up PostgreSQL Client Applications Home Next

vacuumdb

vacuumdb — garbage-collect and analyze a PostgreSQL database

Synopsis

vacuumdb [connection-option...] [option...] [ -t | --table table [( column [,...] )] ] ... [dbname]

vacuumdb [connection-option...] [option...] -a | --all

Description

vacuumdb is a utility for cleaning a PostgreSQL database. vacuumdb will also generate internal statistics used by the PostgreSQL query optimizer.

vacuumdb is a wrapper around the SQL command VACUUM. There is no effective difference between vacuuming and analyzing databases via this utility and via other methods for accessing the server.

Options

vacuumdb accepts the following command-line arguments:

-a
--all

Vacuum all databases.

[-d] dbname
[--dbname=]dbname

Specifies the name of the database to be cleaned or analyzed, when -a/--all is not used. If this is not specified, the database name is read from the environment variable PGDATABASE. If that is not set, the user name specified for the connection is used. The dbname can be a connection string. If so, connection string parameters will override any conflicting command line options.

--disable-page-skipping

Disable skipping pages based on the contents of the visibility map.

Note

This option is only available for servers running PostgreSQL 9.6 and later.

-e
--echo

Echo the commands that vacuumdb generates and sends to the server.

-f
--full

Perform “full” vacuuming.

-F
--freeze

Aggressively “freeze” tuples.

--force-index-cleanup

Always remove index entries pointing to dead tuples.

Note

This option is only available for servers running PostgreSQL 12 and later.

-j njobs
--jobs=njobs

Execute the vacuum or analyze commands in parallel by running njobs commands simultaneously. This option may reduce the processing time but it also increases the load on the database server.

vacuumdb will open njobs connections to the database, so make sure your max_connections setting is high enough to accommodate all connections.

Note that using this mode together with the -f (FULL) option might cause deadlock failures if certain system catalogs are processed in parallel.

--min-mxid-age mxid_age

Only execute the vacuum or analyze commands on tables with a multixact ID age of at least mxid_age. This setting is useful for prioritizing tables to process to prevent multixact ID wraparound (see Section 25.1.5.1).

For the purposes of this option, the multixact ID age of a relation is the greatest of the ages of the main relation and its associated TOAST table, if one exists. Since the commands issued by vacuumdb will also process the TOAST table for the relation if necessary, it does not need to be considered separately.

Note

This option is only available for servers running PostgreSQL 9.6 and later.

--min-xid-age xid_age

Only execute the vacuum or analyze commands on tables with a transaction ID age of at least xid_age. This setting is useful for prioritizing tables to process to prevent transaction ID wraparound (see Section 25.1.5).

For the purposes of this option, the transaction ID age of a relation is the greatest of the ages of the main relation and its associated TOAST table, if one exists. Since the commands issued by vacuumdb will also process the TOAST table for the relation if necessary, it does not need to be considered separately.

Note

This option is only available for servers running PostgreSQL 9.6 and later.

--no-index-cleanup

Do not remove index entries pointing to dead tuples.

Note

This option is only available for servers running PostgreSQL 12 and later.

--no-process-toast

Skip the TOAST table associated with the table to vacuum, if any.

Note

This option is only available for servers running PostgreSQL 14 and later.

--no-truncate

Do not truncate empty pages at the end of the table.

Note

This option is only available for servers running PostgreSQL 12 and later.

-P parallel_workers
--parallel=parallel_workers

Specify the number of parallel workers for parallel vacuum. This allows the vacuum to leverage multiple CPUs to process indexes. See VACUUM.

Note

This option is only available for servers running PostgreSQL 13 and later.

-q
--quiet

Do not display progress messages.

--skip-locked

Skip relations that cannot be immediately locked for processing.

Note

This option is only available for servers running PostgreSQL 12 and later.

-t table [ (column [,...]) ]
--table=table [ (column [,...]) ]

Clean or analyze table only. Column names can be specified only in conjunction with the --analyze or --analyze-only options. Multiple tables can be vacuumed by writing multiple -t switches.

Tip

If you specify columns, you probably have to escape the parentheses from the shell. (See examples below.)

-v
--verbose

Print detailed information during processing.

-V
--version

Print the vacuumdb version and exit.

-z
--analyze

Also calculate statistics for use by the optimizer.

-Z
--analyze-only

Only calculate statistics for use by the optimizer (no vacuum).

--analyze-in-stages

Only calculate statistics for use by the optimizer (no vacuum), like --analyze-only. Run several (currently three) stages of analyze with different configuration settings, to produce usable statistics faster.

This option is useful to analyze a database that was newly populated from a restored dump or by pg_upgrade. This option will try to create some statistics as fast as possible, to make the database usable, and then produce full statistics in the subsequent stages.

-?
--help

Show help about vacuumdb command line arguments, and exit.

vacuumdb also accepts the following command-line arguments for connection parameters:

-h host
--host=host

Specifies the host name of the machine on which the server is running. If the value begins with a slash, it is used as the directory for the Unix domain socket.

-p port
--port=port

Specifies the TCP port or local Unix domain socket file extension on which the server is listening for connections.

-U username
--username=username

User name to connect as.

-w
--no-password

Never issue a password prompt. If the server requires password authentication and a password is not available by other means such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password.

-W
--password

Force vacuumdb to prompt for a password before connecting to a database.

This option is never essential, since vacuumdb will automatically prompt for a password if the server demands password authentication. However, vacuumdb will waste a connection attempt finding out that the server wants a password. In some cases it is worth typing -W to avoid the extra connection attempt.

--maintenance-db=dbname

Specifies the name of the database to connect to to discover which databases should be vacuumed, when -a/--all is used. If not specified, the postgres database will be used, or if that does not exist, template1 will be used. This can be a connection string. If so, connection string parameters will override any conflicting command line options. Also, connection string parameters other than the database name itself will be re-used when connecting to other databases.

Environment

PGDATABASE
PGHOST
PGPORT
PGUSER

Default connection parameters

PG_COLOR

Specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

This utility, like most other PostgreSQL utilities, also uses the environment variables supported by libpq (see Section 34.15).

Diagnostics

In case of difficulty, see VACUUM and psql for discussions of potential problems and error messages. The database server must be running at the targeted host. Also, any default connection settings and environment variables used by the libpq front-end library will apply.

Notes

vacuumdb might need to connect several times to the PostgreSQL server, asking for a password each time. It is convenient to have a ~/.pgpass file in such cases. See Section 34.16 for more information.

Examples

To clean the database test:

$ vacuumdb test

To clean and analyze for the optimizer a database named bigdb:

$ vacuumdb --analyze bigdb

To clean a single table foo in a database named xyzzy, and analyze a single column bar of the table for the optimizer:

$ vacuumdb --analyze --verbose --table='foo(bar)' xyzzy

See Also

VACUUM
Prev | Up | Next
reindexdb | Home | PostgreSQL Server Applications


PostgreSQL Server Applications
Prev Up Part VI. Reference Home Next

PostgreSQL Server Applications

This part contains reference information for PostgreSQL server applications and support utilities. These commands can only be run usefully on the host where the database server resides. Other utility programs are listed in PostgreSQL Client Applications.

Table of Contents

initdb — create a new PostgreSQL database cluster
pg_archivecleanup — clean up PostgreSQL WAL archive files
pg_checksums — enable, disable or check data checksums in a PostgreSQL database cluster
pg_controldata — display control information of a PostgreSQL database cluster
pg_ctl — initialize, start, stop, or control a PostgreSQL server
pg_resetwal — reset the write-ahead log and other control information of a PostgreSQL database cluster
pg_rewind — synchronize a PostgreSQL data directory with another data directory that was forked from it
pg_test_fsync — determine fastest wal_sync_method for PostgreSQL
pg_test_timing — measure timing overhead
pg_upgrade — upgrade a PostgreSQL server instance
pg_waldump — display a human-readable rendering of the write-ahead log of a PostgreSQL database cluster
postgres — PostgreSQL database server
postmaster — PostgreSQL database server
Prev | Up | Next
vacuumdb | Home | initdb


initdb
Prev Up PostgreSQL Server Applications Home Next

initdb

initdb — create a new PostgreSQL database cluster

Synopsis

initdb [option...] [ --pgdata | -D ] directory

Description

initdb creates a new PostgreSQL database cluster. A database cluster is a collection of databases that are managed by a single server instance.

Creating a database cluster consists of creating the directories in which the database data will live, generating the shared catalog tables (tables that belong to the whole cluster rather than to any particular database), and creating the template1 and postgres databases. When you later create a new database, everything in the template1 database is copied. (Therefore, anything installed in template1 is automatically copied into each database created later.) The postgres database is a default database meant for use by users, utilities and third party applications.

Although initdb will attempt to create the specified data directory, it might not have permission if the parent directory of the desired data directory is root-owned. To initialize in such a setup, create an empty data directory as root, then use chown to assign ownership of that directory to the database user account, then su to become the database user to run initdb.

initdb must be run as the user that will own the server process, because the server needs to have access to the files and directories that initdb creates. Since the server cannot be run as root, you must not run initdb as root either. (It will in fact refuse to do so.)

For security reasons the new cluster created by initdb will only be accessible by the cluster owner by default. The --allow-group-access option allows any user in the same group as the cluster owner to read files in the cluster. This is useful for performing backups as a non-privileged user.

initdb initializes the database cluster's default locale and character set encoding. The character set encoding, collation order (LC_COLLATE) and character set classes (LC_CTYPE, e.g., upper, lower, digit) can be set separately for a database when it is created. initdb determines those settings for the template1 database, which will serve as the default for all other databases.

To alter the default collation order or character set classes, use the --lc-collate and --lc-ctype options. Collation orders other than C or POSIX also have a performance penalty. For these reasons it is important to choose the right locale when running initdb.

The remaining locale categories can be changed later when the server is started. You can also use --locale to set the default for all locale categories, including collation order and character set classes. All server locale values (lc_*) can be displayed via SHOW ALL. More details can be found in Section 24.1.

To alter the default encoding, use the --encoding. More details can be found in Section 24.3.

Options

-A authmethod
--auth=authmethod

This option specifies the default authentication method for local users used in pg_hba.conf (host and local lines). initdb will prepopulate pg_hba.conf entries using the specified authentication method for non-replication as well as replication connections.

Do not use trust unless you trust all local users on your system. trust is the default for ease of installation.

--auth-host=authmethod

This option specifies the authentication method for local users via TCP/IP connections used in pg_hba.conf (host lines).

--auth-local=authmethod

This option specifies the authentication method for local users via Unix-domain socket connections used in pg_hba.conf (local lines).

-D directory
--pgdata=directory

This option specifies the directory where the database cluster should be stored. This is the only information required by initdb, but you can avoid writing it by setting the PGDATA environment variable, which can be convenient since the database server (postgres) can find the database directory later by the same variable.

-E encoding
--encoding=encoding

Selects the encoding of the template database. This will also be the default encoding of any database you create later, unless you override it there. The default is derived from the locale, or SQL_ASCII if that does not work. The character sets supported by the PostgreSQL server are described in Section 24.3.1.

-g
--allow-group-access

Allows users in the same group as the cluster owner to read all cluster files created by initdb. This option is ignored on Windows as it does not support POSIX-style group permissions.

-k
--data-checksums

Use checksums on data pages to help detect corruption by the I/O system that would otherwise be silent. Enabling checksums may incur a noticeable performance penalty. If set, checksums are calculated for all objects, in all databases. All checksum failures will be reported in the pg_stat_database view. See Section 30.2 for details.

--locale=locale

Sets the default locale for the database cluster. If this option is not specified, the locale is inherited from the environment that initdb runs in. Locale support is described in Section 24.1.

--lc-collate=locale
--lc-ctype=locale
--lc-messages=locale
--lc-monetary=locale
--lc-numeric=locale
--lc-time=locale

Like --locale, but only sets the locale in the specified category.

--no-locale

Equivalent to --locale=C.

-N
--no-sync

By default, initdb will wait for all files to be written safely to disk. This option causes initdb to return without waiting, which is faster, but means that a subsequent operating system crash can leave the data directory corrupt. Generally, this option is useful for testing, but should not be used when creating a production installation.

--no-instructions

By default, initdb will write instructions for how to start the cluster at the end of its output. This option causes those instructions to be left out. This is primarily intended for use by tools that wrap initdb in platform-specific behavior, where those instructions are likely to be incorrect.

--pwfile=filename

Makes initdb read the database superuser's password from a file. The first line of the file is taken as the password.

-S
--sync-only

Safely write all database files to disk and exit. This does not perform any of the normal initdb operations.

-T config
--text-search-config=config

Sets the default text search configuration. See default_text_search_config for further information.

-U username
--username=username

Selects the user name of the database superuser. This defaults to the name of the effective user running initdb. It is really not important what the superuser's name is, but one might choose to keep the customary name postgres, even if the operating system user's name is different.

-W
--pwprompt

Makes initdb prompt for a password to give the database superuser. If you don't plan on using password authentication, this is not important. Otherwise you won't be able to use password authentication until you have a password set up.

-X directory
--waldir=directory

This option specifies the directory where the write-ahead log should be stored.

--wal-segsize=size

Set the WAL segment size, in megabytes. This is the size of each individual file in the WAL log. The default size is 16 megabytes. The value must be a power of 2 between 1 and 1024 (megabytes). This option can only be set during initialization, and cannot be changed later.

It may be useful to adjust this size to control the granularity of WAL log shipping or archiving. Also, in databases with a high volume of WAL, the sheer number of WAL files per directory can become a performance and management problem. Increasing the WAL file size will reduce the number of WAL files.

Other, less commonly used, options are also available:

-d
--debug

Print debugging output from the bootstrap backend and a few other messages of lesser interest for the general public. The bootstrap backend is the program initdb uses to create the catalog tables. This option generates a tremendous amount of extremely boring output.

--discard-caches

Run the bootstrap backend with the debug_discard_caches=1 option. This takes a very long time and is only of use for deep debugging.

-L directory

Specifies where initdb should find its input files to initialize the database cluster. This is normally not necessary. You will be told if you need to specify their location explicitly.

-n
--no-clean

By default, when initdb determines that an error prevented it from completely creating the database cluster, it removes any files it might have created before discovering that it cannot finish the job. This option inhibits tidying-up and is thus useful for debugging.

Other options:

-V
--version

Print the initdb version and exit.

-?
--help

Show help about initdb command line arguments, and exit.

Environment

PGDATA

Specifies the directory where the database cluster is to be stored; can be overridden using the -D option.

PG_COLOR

Specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

TZ

Specifies the default time zone of the created database cluster. The value should be a full time zone name (see Section 8.5.3).

This utility, like most other PostgreSQL utilities, also uses the environment variables supported by libpq (see Section 34.15).

Notes

initdb can also be invoked via pg_ctl initdb.

See Also

pg_ctl, postgres
Prev | Up | Next
PostgreSQL Server Applications | Home | pg_archivecleanup


pg_archivecleanup
Prev Up PostgreSQL Server Applications Home Next

pg_archivecleanup

pg_archivecleanup — clean up PostgreSQL WAL archive files

Synopsis

pg_archivecleanup [option...] archivelocation oldestkeptwalfile

Description

pg_archivecleanup is designed to be used as an archive_cleanup_command to clean up WAL file archives when running as a standby server (see Section 27.2). pg_archivecleanup can also be used as a standalone program to clean WAL file archives.

To configure a standby server to use pg_archivecleanup, put this into its postgresql.conf configuration file:

archive_cleanup_command = 'pg_archivecleanup archivelocation %r'

where archivelocation is the directory from which WAL segment files should be removed.

When used within archive_cleanup_command, all WAL files logically preceding the value of the %r argument will be removed from archivelocation. This minimizes the number of files that need to be retained, while preserving crash-restart capability. Use of this parameter is appropriate if the archivelocation is a transient staging area for this particular standby server, but not when the archivelocation is intended as a long-term WAL archive area, or when multiple standby servers are recovering from the same archive location.

When used as a standalone program all WAL files logically preceding the oldestkeptwalfile will be removed from archivelocation. In this mode, if you specify a .partial or .backup file name, then only the file prefix will be used as the oldestkeptwalfile. This treatment of .backup file name allows you to remove all WAL files archived prior to a specific base backup without error. For example, the following example will remove all files older than WAL file name 000000010000003700000010:

pg_archivecleanup -d archive 000000010000003700000010.00000020.backup

pg_archivecleanup:  keep WAL file "archive/000000010000003700000010" and later
pg_archivecleanup:  removing file "archive/00000001000000370000000F"
pg_archivecleanup:  removing file "archive/00000001000000370000000E"

pg_archivecleanup assumes that archivelocation is a directory readable and writable by the server-owning user.

Options

pg_archivecleanup accepts the following command-line arguments:

-d

Print lots of debug logging output on stderr.

-n

Print the names of the files that would have been removed on stdout (performs a dry run).

-V
--version

Print the pg_archivecleanup version and exit.

-x extension

Provide an extension that will be stripped from all file names before deciding if they should be deleted. This is typically useful for cleaning up archives that have been compressed during storage, and therefore have had an extension added by the compression program. For example: -x .gz.

-?
--help

Show help about pg_archivecleanup command line arguments, and exit.

Environment

The environment variable PG_COLOR specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

Notes

pg_archivecleanup is designed to work with PostgreSQL 8.0 and later when used as a standalone utility, or with PostgreSQL 9.0 and later when used as an archive cleanup command.

pg_archivecleanup is written in C and has an easy-to-modify source code, with specifically designated sections to modify for your own needs

Examples

On Linux or Unix systems, you might use:

archive_cleanup_command = 'pg_archivecleanup -d /mnt/standby/archive %r 2>>cleanup.log'

where the archive directory is physically located on the standby server, so that the archive_command is accessing it across NFS, but the files are local to the standby. This will:

  • produce debugging output in cleanup.log

  • remove no-longer-needed files from the archive directory

Prev | Up | Next
initdb | Home | pg_checksums


pg_checksums
Prev Up PostgreSQL Server Applications Home Next

pg_checksums

pg_checksums — enable, disable or check data checksums in a PostgreSQL database cluster

Synopsis

pg_checksums [option...] [[ -D | --pgdata ]datadir]

Description

pg_checksums checks, enables or disables data checksums in a PostgreSQL cluster. The server must be shut down cleanly before running pg_checksums. When verifying checksums, the exit status is zero if there are no checksum errors, and nonzero if at least one checksum failure is detected. When enabling or disabling checksums, the exit status is nonzero if the operation failed.

When verifying checksums, every file in the cluster is scanned. When enabling checksums, every file in the cluster is rewritten in-place. Disabling checksums only updates the file pg_control.

Options

The following command-line options are available:

-D directory
--pgdata=directory

Specifies the directory where the database cluster is stored.

-c
--check

Checks checksums. This is the default mode if nothing else is specified.

-d
--disable

Disables checksums.

-e
--enable

Enables checksums.

-f filenode
--filenode=filenode

Only validate checksums in the relation with filenode filenode.

-N
--no-sync

By default, pg_checksums will wait for all files to be written safely to disk. This option causes pg_checksums to return without waiting, which is faster, but means that a subsequent operating system crash can leave the updated data directory corrupt. Generally, this option is useful for testing but should not be used on a production installation. This option has no effect when using --check.

-P
--progress

Enable progress reporting. Turning this on will deliver a progress report while checking or enabling checksums.

-v
--verbose

Enable verbose output. Lists all checked files.

-V
--version

Print the pg_checksums version and exit.

-?
--help

Show help about pg_checksums command line arguments, and exit.

Environment

PGDATA

Specifies the directory where the database cluster is stored; can be overridden using the -D option.

PG_COLOR

Specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

Notes

Enabling checksums in a large cluster can potentially take a long time. During this operation, the cluster or other programs that write to the data directory must not be started or else data loss may occur.

When using a replication setup with tools which perform direct copies of relation file blocks (for example pg_rewind), enabling or disabling checksums can lead to page corruptions in the shape of incorrect checksums if the operation is not done consistently across all nodes. When enabling or disabling checksums in a replication setup, it is thus recommended to stop all the clusters before switching them all consistently. Destroying all standbys, performing the operation on the primary and finally recreating the standbys from scratch is also safe.

If pg_checksums is aborted or killed while enabling or disabling checksums, the cluster's data checksum configuration remains unchanged, and pg_checksums can be re-run to perform the same operation.

Prev | Up | Next
pg_archivecleanup | Home | pg_controldata


pg_controldata
Prev Up PostgreSQL Server Applications Home Next

pg_controldata

pg_controldata — display control information of a PostgreSQL database cluster

Synopsis

pg_controldata [option] [[ -D | --pgdata ]datadir]

Description

pg_controldata prints information initialized during initdb, such as the catalog version. It also shows information about write-ahead logging and checkpoint processing. This information is cluster-wide, and not specific to any one database.

This utility can only be run by the user who initialized the cluster because it requires read access to the data directory. You can specify the data directory on the command line, or use the environment variable PGDATA. This utility supports the options -V and --version, which print the pg_controldata version and exit. It also supports options -? and --help, which output the supported arguments.

Environment

PGDATA

Default data directory location

PG_COLOR

Specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

Prev | Up | Next
pg_checksums | Home | pg_ctl


pg_ctl
Prev Up PostgreSQL Server Applications Home Next

pg_ctl

pg_ctl — initialize, start, stop, or control a PostgreSQL server

Synopsis

pg_ctl init[db] [-D datadir] [-s] [-o initdb-options]

pg_ctl start [-D datadir] [-l filename] [-W] [-t seconds] [-s] [-o options] [-p path] [-c]

pg_ctl stop [-D datadir] [-m s[mart] | f[ast] | i[mmediate] ] [-W] [-t seconds] [-s]

pg_ctl restart [-D datadir] [-m s[mart] | f[ast] | i[mmediate] ] [-W] [-t seconds] [-s] [-o options] [-c]

pg_ctl reload [-D datadir] [-s]

pg_ctl status [-D datadir]

pg_ctl promote [-D datadir] [-W] [-t seconds] [-s]

pg_ctl logrotate [-D datadir] [-s]

pg_ctl kill signal_name process_id

On Microsoft Windows, also:

pg_ctl register [-D datadir] [-N servicename] [-U username] [-P password] [-S a[uto] | d[emand] ] [-e source] [-W] [-t seconds] [-s] [-o options]

pg_ctl unregister [-N servicename]

Description

pg_ctl is a utility for initializing a PostgreSQL database cluster, starting, stopping, or restarting the PostgreSQL database server (postgres), or displaying the status of a running server. Although the server can be started manually, pg_ctl encapsulates tasks such as redirecting log output and properly detaching from the terminal and process group. It also provides convenient options for controlled shutdown.

The init or initdb mode creates a new PostgreSQL database cluster, that is, a collection of databases that will be managed by a single server instance. This mode invokes the initdb command. See initdb for details.

start mode launches a new server. The server is started in the background, and its standard input is attached to /dev/null (or nul on Windows). On Unix-like systems, by default, the server's standard output and standard error are sent to pg_ctl's standard output (not standard error). The standard output of pg_ctl should then be redirected to a file or piped to another process such as a log rotating program like rotatelogs; otherwise postgres will write its output to the controlling terminal (from the background) and will not leave the shell's process group. On Windows, by default the server's standard output and standard error are sent to the terminal. These default behaviors can be changed by using -l to append the server's output to a log file. Use of either -l or output redirection is recommended.

stop mode shuts down the server that is running in the specified data directory. Three different shutdown methods can be selected with the -m option. “Smart” mode disallows new connections, then waits for all existing clients to disconnect and any online backup to finish. If the server is in hot standby, recovery and streaming replication will be terminated once all clients have disconnected. “Fast” mode (the default) does not wait for clients to disconnect and will terminate an online backup in progress. All active transactions are rolled back and clients are forcibly disconnected, then the server is shut down. “Immediate” mode will abort all server processes immediately, without a clean shutdown. This choice will lead to a crash-recovery cycle during the next server start.

restart mode effectively executes a stop followed by a start. This allows changing the postgres command-line options, or changing configuration-file options that cannot be changed without restarting the server. If relative paths were used on the command line during server start, restart might fail unless pg_ctl is executed in the same current directory as it was during server start.

reload mode simply sends the postgres server process a SIGHUP signal, causing it to reread its configuration files (postgresql.conf, pg_hba.conf, etc.). This allows changing configuration-file options that do not require a full server restart to take effect.

status mode checks whether a server is running in the specified data directory. If it is, the server's PID and the command line options that were used to invoke it are displayed. If the server is not running, pg_ctl returns an exit status of 3. If an accessible data directory is not specified, pg_ctl returns an exit status of 4.

promote mode commands the standby server that is running in the specified data directory to end standby mode and begin read-write operations.

logrotate mode rotates the server log file. For details on how to use this mode with external log rotation tools, see Section 25.3.

kill mode sends a signal to a specified process. This is primarily valuable on Microsoft Windows which does not have a built-in kill command. Use --help to see a list of supported signal names.

register mode registers the PostgreSQL server as a system service on Microsoft Windows. The -S option allows selection of service start type, either “auto” (start service automatically on system startup) or “demand” (start service on demand).

unregister mode unregisters a system service on Microsoft Windows. This undoes the effects of the register command.

Options

-c
--core-files

Attempt to allow server crashes to produce core files, on platforms where this is possible, by lifting any soft resource limit placed on core files. This is useful in debugging or diagnosing problems by allowing a stack trace to be obtained from a failed server process.

-D datadir
--pgdata=datadir

Specifies the file system location of the database configuration files. If this option is omitted, the environment variable PGDATA is used.

-l filename
--log=filename

Append the server log output to filename. If the file does not exist, it is created. The umask is set to 077, so access to the log file is disallowed to other users by default.

-m mode
--mode=mode

Specifies the shutdown mode. mode can be smart, fast, or immediate, or the first letter of one of these three. If this option is omitted, fast is the default.

-o options
--options=options

Specifies options to be passed directly to the postgres command. -o can be specified multiple times, with all the given options being passed through.

The options should usually be surrounded by single or double quotes to ensure that they are passed through as a group.

-o initdb-options
--options=initdb-options

Specifies options to be passed directly to the initdb command. -o can be specified multiple times, with all the given options being passed through.

The initdb-options should usually be surrounded by single or double quotes to ensure that they are passed through as a group.

-p path

Specifies the location of the postgres executable. By default the postgres executable is taken from the same directory as pg_ctl, or failing that, the hard-wired installation directory. It is not necessary to use this option unless you are doing something unusual and get errors that the postgres executable was not found.

In init mode, this option analogously specifies the location of the initdb executable.

-s
--silent

Print only errors, no informational messages.

-t seconds
--timeout=seconds

Specifies the maximum number of seconds to wait when waiting for an operation to complete (see option -w). Defaults to the value of the PGCTLTIMEOUT environment variable or, if not set, to 60 seconds.

-V
--version

Print the pg_ctl version and exit.

-w
--wait

Wait for the operation to complete. This is supported for the modes start, stop, restart, promote, and register, and is the default for those modes.

When waiting, pg_ctl repeatedly checks the server's PID file, sleeping for a short amount of time between checks. Startup is considered complete when the PID file indicates that the server is ready to accept connections. Shutdown is considered complete when the server removes the PID file. pg_ctl returns an exit code based on the success of the startup or shutdown.

If the operation does not complete within the timeout (see option -t), then pg_ctl exits with a nonzero exit status. But note that the operation might continue in the background and eventually succeed.

-W
--no-wait

Do not wait for the operation to complete. This is the opposite of the option -w.

If waiting is disabled, the requested action is triggered, but there is no feedback about its success. In that case, the server log file or an external monitoring system would have to be used to check the progress and success of the operation.

In prior releases of PostgreSQL, this was the default except for the stop mode.

-?
--help

Show help about pg_ctl command line arguments, and exit.

If an option is specified that is valid, but not relevant to the selected operating mode, pg_ctl ignores it.

Options for Windows

-e source

Name of the event source for pg_ctl to use for logging to the event log when running as a Windows service. The default is PostgreSQL. Note that this only controls messages sent from pg_ctl itself; once started, the server will use the event source specified by its event_source parameter. Should the server fail very early in startup, before that parameter has been set, it might also log using the default event source name PostgreSQL.

-N servicename

Name of the system service to register. This name will be used as both the service name and the display name. The default is PostgreSQL.

-P password

Password for the user to run the service as.

-S start-type

Start type of the system service. start-type can be auto, or demand, or the first letter of one of these two. If this option is omitted, auto is the default.

-U username

User name for the user to run the service as. For domain users, use the format DOMAIN\username.

Environment

PGCTLTIMEOUT

Default limit on the number of seconds to wait when waiting for startup or shutdown to complete. If not set, the default is 60 seconds.

PGDATA

Default data directory location.

Most pg_ctl modes require knowing the data directory location; therefore, the -D option is required unless PGDATA is set.

pg_ctl, like most other PostgreSQL utilities, also uses the environment variables supported by libpq (see Section 34.15).

For additional variables that affect the server, see postgres.

Files

postmaster.pid

pg_ctl examines this file in the data directory to determine whether the server is currently running.

postmaster.opts

If this file exists in the data directory, pg_ctl (in restart mode) will pass the contents of the file as options to postgres, unless overridden by the -o option. The contents of this file are also displayed in status mode.

Examples

Starting the Server

To start the server, waiting until the server is accepting connections:

$ pg_ctl start

To start the server using port 5433, and running without fsync, use:

$ pg_ctl -o "-F -p 5433" start

Stopping the Server

To stop the server, use:

$ pg_ctl stop

The -m option allows control over how the server shuts down:

$ pg_ctl stop -m smart

Restarting the Server

Restarting the server is almost equivalent to stopping the server and starting it again, except that by default, pg_ctl saves and reuses the command line options that were passed to the previously-running instance. To restart the server using the same options as before, use:

$ pg_ctl restart

But if -o is specified, that replaces any previous options. To restart using port 5433, disabling fsync upon restart:

$ pg_ctl -o "-F -p 5433" restart

Showing the Server Status

Here is sample status output from pg_ctl:

$ pg_ctl status

pg_ctl: server is running (PID: 13718)
/usr/local/pgsql/bin/postgres "-D" "/usr/local/pgsql/data" "-p" "5433" "-B" "128"

The second line is the command that would be invoked in restart mode.

See Also

initdb, postgres
Prev | Up | Next
pg_controldata | Home | pg_resetwal


pg_resetwal
Prev Up PostgreSQL Server Applications Home Next

pg_resetwal

pg_resetwal — reset the write-ahead log and other control information of a PostgreSQL database cluster

Synopsis

pg_resetwal [ -f | --force ] [ -n | --dry-run ] [option...] [ -D | --pgdata ]datadir

Description

pg_resetwal clears the write-ahead log (WAL) and optionally resets some other control information stored in the pg_control file. This function is sometimes needed if these files have become corrupted. It should be used only as a last resort, when the server will not start due to such corruption.

After running this command, it should be possible to start the server, but bear in mind that the database might contain inconsistent data due to partially-committed transactions. You should immediately dump your data, run initdb, and reload. After reload, check for inconsistencies and repair as needed.

This utility can only be run by the user who installed the server, because it requires read/write access to the data directory. For safety reasons, you must specify the data directory on the command line. pg_resetwal does not use the environment variable PGDATA.

If pg_resetwal complains that it cannot determine valid data for pg_control, you can force it to proceed anyway by specifying the -f (force) option. In this case plausible values will be substituted for the missing data. Most of the fields can be expected to match, but manual assistance might be needed for the next OID, next transaction ID and epoch, next multitransaction ID and offset, and WAL starting location fields. These fields can be set using the options discussed below. If you are not able to determine correct values for all these fields, -f can still be used, but the recovered database must be treated with even more suspicion than usual: an immediate dump and reload is imperative. Do not execute any data-modifying operations in the database before you dump, as any such action is likely to make the corruption worse.

Options

-f
--force

Force pg_resetwal to proceed even if it cannot determine valid data for pg_control, as explained above.

-n
--dry-run

The -n/--dry-run option instructs pg_resetwal to print the values reconstructed from pg_control and values about to be changed, and then exit without modifying anything. This is mainly a debugging tool, but can be useful as a sanity check before allowing pg_resetwal to proceed for real.

-V
--version

Display version information, then exit.

-?
--help

Show help, then exit.

The following options are only needed when pg_resetwal is unable to determine appropriate values by reading pg_control. Safe values can be determined as described below. For values that take numeric arguments, hexadecimal values can be specified by using the prefix 0x.

-c xid,xid
--commit-timestamp-ids=xid,xid

Manually set the oldest and newest transaction IDs for which the commit time can be retrieved.

A safe value for the oldest transaction ID for which the commit time can be retrieved (first part) can be determined by looking for the numerically smallest file name in the directory pg_commit_ts under the data directory. Conversely, a safe value for the newest transaction ID for which the commit time can be retrieved (second part) can be determined by looking for the numerically greatest file name in the same directory. The file names are in hexadecimal.

-e xid_epoch
--epoch=xid_epoch

Manually set the next transaction ID's epoch.

The transaction ID epoch is not actually stored anywhere in the database except in the field that is set by pg_resetwal, so any value will work so far as the database itself is concerned. You might need to adjust this value to ensure that replication systems such as Slony-I and Skytools work correctly — if so, an appropriate value should be obtainable from the state of the downstream replicated database.

-l walfile
--next-wal-file=walfile

Manually set the WAL starting location by specifying the name of the next WAL segment file.

The name of next WAL segment file should be larger than any WAL segment file name currently existing in the directory pg_wal under the data directory. These names are also in hexadecimal and have three parts. The first part is the “timeline ID” and should usually be kept the same. For example, if 00000001000000320000004A is the largest entry in pg_wal, use -l 00000001000000320000004B or higher.

Note that when using nondefault WAL segment sizes, the numbers in the WAL file names are different from the LSNs that are reported by system functions and system views. This option takes a WAL file name, not an LSN.

Note

pg_resetwal itself looks at the files in pg_wal and chooses a default -l setting beyond the last existing file name. Therefore, manual adjustment of -l should only be needed if you are aware of WAL segment files that are not currently present in pg_wal, such as entries in an offline archive; or if the contents of pg_wal have been lost entirely.

-m mxid,mxid
--multixact-ids=mxid,mxid

Manually set the next and oldest multitransaction ID.

A safe value for the next multitransaction ID (first part) can be determined by looking for the numerically largest file name in the directory pg_multixact/offsets under the data directory, adding one, and then multiplying by 65536 (0x10000). Conversely, a safe value for the oldest multitransaction ID (second part of -m) can be determined by looking for the numerically smallest file name in the same directory and multiplying by 65536. The file names are in hexadecimal, so the easiest way to do this is to specify the option value in hexadecimal and append four zeroes.

-o oid
--next-oid=oid

Manually set the next OID.

There is no comparably easy way to determine a next OID that's beyond the largest one in the database, but fortunately it is not critical to get the next-OID setting right.

-O mxoff
--multixact-offset=mxoff

Manually set the next multitransaction offset.

A safe value can be determined by looking for the numerically largest file name in the directory pg_multixact/members under the data directory, adding one, and then multiplying by 52352 (0xCC80). The file names are in hexadecimal. There is no simple recipe such as the ones for other options of appending zeroes.

--wal-segsize=wal_segment_size

Set the new WAL segment size, in megabytes. The value must be set to a power of 2 between 1 and 1024 (megabytes). See the same option of initdb for more information.

Note

While pg_resetwal will set the WAL starting address beyond the latest existing WAL segment file, some segment size changes can cause previous WAL file names to be reused. It is recommended to use -l together with this option to manually set the WAL starting address if WAL file name overlap will cause problems with your archiving strategy.

-u xid
--oldest-transaction-id=xid

Manually set the oldest unfrozen transaction ID.

A safe value can be determined by looking for the numerically smallest file name in the directory pg_xact under the data directory and then multiplying by 1048576 (0x100000). Note that the file names are in hexadecimal. It is usually easiest to specify the option value in hexadecimal too. For example, if 0007 is the smallest entry in pg_xact, -u 0x700000 will work (five trailing zeroes provide the proper multiplier).

-x xid
--next-transaction-id=xid

Manually set the next transaction ID.

A safe value can be determined by looking for the numerically largest file name in the directory pg_xact under the data directory, adding one, and then multiplying by 1048576 (0x100000). Note that the file names are in hexadecimal. It is usually easiest to specify the option value in hexadecimal too. For example, if 0011 is the largest entry in pg_xact, -x 0x1200000 will work (five trailing zeroes provide the proper multiplier).

Environment

PG_COLOR

Specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

Notes

This command must not be used when the server is running. pg_resetwal will refuse to start up if it finds a server lock file in the data directory. If the server crashed then a lock file might have been left behind; in that case you can remove the lock file to allow pg_resetwal to run. But before you do so, make doubly certain that there is no server process still alive.

pg_resetwal works only with servers of the same major version.

See Also

pg_controldata
Prev | Up | Next
pg_ctl | Home | pg_rewind


pg_rewind
Prev Up PostgreSQL Server Applications Home Next

pg_rewind

pg_rewind — synchronize a PostgreSQL data directory with another data directory that was forked from it

Synopsis

pg_rewind [option...] { -D | --target-pgdata } directory { --source-pgdata=directory | --source-server=connstr }

Description

pg_rewind is a tool for synchronizing a PostgreSQL cluster with another copy of the same cluster, after the clusters' timelines have diverged. A typical scenario is to bring an old primary server back online after failover as a standby that follows the new primary.

After a successful rewind, the state of the target data directory is analogous to a base backup of the source data directory. Unlike taking a new base backup or using a tool like rsync, pg_rewind does not require comparing or copying unchanged relation blocks in the cluster. Only changed blocks from existing relation files are copied; all other files, including new relation files, configuration files, and WAL segments, are copied in full. As such the rewind operation is significantly faster than other approaches when the database is large and only a small fraction of blocks differ between the clusters.

pg_rewind examines the timeline histories of the source and target clusters to determine the point where they diverged, and expects to find WAL in the target cluster's pg_wal directory reaching all the way back to the point of divergence. The point of divergence can be found either on the target timeline, the source timeline, or their common ancestor. In the typical failover scenario where the target cluster was shut down soon after the divergence, this is not a problem, but if the target cluster ran for a long time after the divergence, its old WAL files might no longer be present. In this case, you can manually copy them from the WAL archive to the pg_wal directory, or run pg_rewind with the -c option to automatically retrieve them from the WAL archive. The use of pg_rewind is not limited to failover, e.g., a standby server can be promoted, run some write transactions, and then rewound to become a standby again.

After running pg_rewind, WAL replay needs to complete for the data directory to be in a consistent state. When the target server is started again it will enter archive recovery and replay all WAL generated in the source server from the last checkpoint before the point of divergence. If some of the WAL was no longer available in the source server when pg_rewind was run, and therefore could not be copied by the pg_rewind session, it must be made available when the target server is started. This can be done by creating a recovery.signal file in the target data directory and by configuring a suitable restore_command in postgresql.conf.

pg_rewind requires that the target server either has the wal_log_hints option enabled in postgresql.conf or data checksums enabled when the cluster was initialized with initdb. Neither of these are currently on by default. full_page_writes must also be set to on, but is enabled by default.

Warning

If pg_rewind fails while processing, then the data folder of the target is likely not in a state that can be recovered. In such a case, taking a new fresh backup is recommended.

As pg_rewind copies configuration files entirely from the source, it may be required to correct the configuration used for recovery before restarting the target server, especially if the target is reintroduced as a standby of the source. If you restart the server after the rewind operation has finished but without configuring recovery, the target may again diverge from the primary.

pg_rewind will fail immediately if it finds files it cannot write directly to. This can happen for example when the source and the target server use the same file mapping for read-only SSL keys and certificates. If such files are present on the target server it is recommended to remove them before running pg_rewind. After doing the rewind, some of those files may have been copied from the source, in which case it may be necessary to remove the data copied and restore back the set of links used before the rewind.

Options

pg_rewind accepts the following command-line arguments:

-D directory
--target-pgdata=directory

This option specifies the target data directory that is synchronized with the source. The target server must be shut down cleanly before running pg_rewind

--source-pgdata=directory

Specifies the file system path to the data directory of the source server to synchronize the target with. This option requires the source server to be cleanly shut down.

--source-server=connstr

Specifies a libpq connection string to connect to the source PostgreSQL server to synchronize the target with. The connection must be a normal (non-replication) connection with a role having sufficient permissions to execute the functions used by pg_rewind on the source server (see Notes section for details) or a superuser role. This option requires the source server to be running and accepting connections.

-R
--write-recovery-conf

Create standby.signal and append connection settings to postgresql.auto.conf in the output directory. --source-server is mandatory with this option.

-n
--dry-run

Do everything except actually modifying the target directory.

-N
--no-sync

By default, pg_rewind will wait for all files to be written safely to disk. This option causes pg_rewind to return without waiting, which is faster, but means that a subsequent operating system crash can leave the data directory corrupt. Generally, this option is useful for testing but should not be used on a production installation.

-P
--progress

Enables progress reporting. Turning this on will deliver an approximate progress report while copying data from the source cluster.

-c
--restore-target-wal

Use restore_command defined in the target cluster configuration to retrieve WAL files from the WAL archive if these files are no longer available in the pg_wal directory.

--debug

Print verbose debugging output that is mostly useful for developers debugging pg_rewind.

--no-ensure-shutdown

pg_rewind requires that the target server is cleanly shut down before rewinding. By default, if the target server is not shut down cleanly, pg_rewind starts the target server in single-user mode to complete crash recovery first, and stops it. By passing this option, pg_rewind skips this and errors out immediately if the server is not cleanly shut down. Users are expected to handle the situation themselves in that case.

-V
--version

Display version information, then exit.

-?
--help

Show help, then exit.

Environment

When --source-server option is used, pg_rewind also uses the environment variables supported by libpq (see Section 34.15).

The environment variable PG_COLOR specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

Notes

When executing pg_rewind using an online cluster as source, a role having sufficient permissions to execute the functions used by pg_rewind on the source cluster can be used instead of a superuser. Here is how to create such a role, named rewind_user here:

CREATE USER rewind_user LOGIN;
GRANT EXECUTE ON function pg_catalog.pg_ls_dir(text, boolean, boolean) TO rewind_user;
GRANT EXECUTE ON function pg_catalog.pg_stat_file(text, boolean) TO rewind_user;
GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text) TO rewind_user;
GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text, bigint, bigint, boolean) TO rewind_user;

When executing pg_rewind using an online cluster as source which has been recently promoted, it is necessary to execute a CHECKPOINT after promotion such that its control file reflects up-to-date timeline information, which is used by pg_rewind to check if the target cluster can be rewound using the designated source cluster.

How It Works

The basic idea is to copy all file system-level changes from the source cluster to the target cluster:

  1. Scan the WAL log of the target cluster, starting from the last checkpoint before the point where the source cluster's timeline history forked off from the target cluster. For each WAL record, record each data block that was touched. This yields a list of all the data blocks that were changed in the target cluster, after the source cluster forked off. If some of the WAL files are no longer available, try re-running pg_rewind with the -c option to search for the missing files in the WAL archive.

  2. Copy all those changed blocks from the source cluster to the target cluster, either using direct file system access (--source-pgdata) or SQL (--source-server). Relation files are now in a state equivalent to the moment of the last completed checkpoint prior to the point at which the WAL timelines of the source and target diverged plus the current state on the source of any blocks changed on the target after that divergence.

  3. Copy all other files, including new relation files, WAL segments, pg_xact, and configuration files from the source cluster to the target cluster. Similarly to base backups, the contents of the directories pg_dynshmem/, pg_notify/, pg_replslot/, pg_serial/, pg_snapshots/, pg_stat_tmp/, and pg_subtrans/ are omitted from the data copied from the source cluster. The files backup_label, tablespace_map, pg_internal.init, postmaster.opts, and postmaster.pid, as well as any file or directory beginning with pgsql_tmp, are omitted.

  4. Create a backup_label file to begin WAL replay at the checkpoint created at failover and configure the pg_control file with a minimum consistency LSN defined as the result of pg_current_wal_insert_lsn() when rewinding from a live source or the last checkpoint LSN when rewinding from a stopped source.

  5. When starting the target, PostgreSQL replays all the required WAL, resulting in a data directory in a consistent state.

Prev | Up | Next
pg_resetwal | Home | pg_test_fsync


pg_test_fsync
Prev Up PostgreSQL Server Applications Home Next

pg_test_fsync

pg_test_fsync — determine fastest wal_sync_method for PostgreSQL

Synopsis

pg_test_fsync [option...]

Description

pg_test_fsync is intended to give you a reasonable idea of what the fastest wal_sync_method is on your specific system, as well as supplying diagnostic information in the event of an identified I/O problem. However, differences shown by pg_test_fsync might not make any significant difference in real database throughput, especially since many database servers are not speed-limited by their write-ahead logs. pg_test_fsync reports average file sync operation time in microseconds for each wal_sync_method, which can also be used to inform efforts to optimize the value of commit_delay.

Options

pg_test_fsync accepts the following command-line options:

-f
--filename

Specifies the file name to write test data in. This file should be in the same file system that the pg_wal directory is or will be placed in. (pg_wal contains the WAL files.) The default is pg_test_fsync.out in the current directory.

-s
--secs-per-test

Specifies the number of seconds for each test. The more time per test, the greater the test's accuracy, but the longer it takes to run. The default is 5 seconds, which allows the program to complete in under 2 minutes.

-V
--version

Print the pg_test_fsync version and exit.

-?
--help

Show help about pg_test_fsync command line arguments, and exit.

Environment

The environment variable PG_COLOR specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

See Also

postgres
Prev | Up | Next
pg_rewind | Home | pg_test_timing


pg_test_timing
Prev Up PostgreSQL Server Applications Home Next

pg_test_timing

pg_test_timing — measure timing overhead

Synopsis

pg_test_timing [option...]

Description

pg_test_timing is a tool to measure the timing overhead on your system and confirm that the system time never moves backwards. Systems that are slow to collect timing data can give less accurate EXPLAIN ANALYZE results.

Options

pg_test_timing accepts the following command-line options:

-d duration
--duration=duration

Specifies the test duration, in seconds. Longer durations give slightly better accuracy, and are more likely to discover problems with the system clock moving backwards. The default test duration is 3 seconds.

-V
--version

Print the pg_test_timing version and exit.

-?
--help

Show help about pg_test_timing command line arguments, and exit.

Usage

Interpreting Results

Good results will show most (>90%) individual timing calls take less than one microsecond. Average per loop overhead will be even lower, below 100 nanoseconds. This example from an Intel i7-860 system using a TSC clock source shows excellent performance:

Testing timing overhead for 3 seconds.
Per loop time including overhead: 35.96 ns
Histogram of timing durations:
  < us   % of total      count
     1     96.40465   80435604
     2      3.59518    2999652
     4      0.00015        126
     8      0.00002         13
    16      0.00000          2

Note that different units are used for the per loop time than the histogram. The loop can have resolution within a few nanoseconds (ns), while the individual timing calls can only resolve down to one microsecond (us).

Measuring Executor Timing Overhead

When the query executor is running a statement using EXPLAIN ANALYZE, individual operations are timed as well as showing a summary. The overhead of your system can be checked by counting rows with the psql program:

CREATE TABLE t AS SELECT * FROM generate_series(1,100000);
\timing
SELECT COUNT(*) FROM t;
EXPLAIN ANALYZE SELECT COUNT(*) FROM t;

The i7-860 system measured runs the count query in 9.8 ms while the EXPLAIN ANALYZE version takes 16.6 ms, each processing just over 100,000 rows. That 6.8 ms difference means the timing overhead per row is 68 ns, about twice what pg_test_timing estimated it would be. Even that relatively small amount of overhead is making the fully timed count statement take almost 70% longer. On more substantial queries, the timing overhead would be less problematic.

Changing Time Sources

On some newer Linux systems, it's possible to change the clock source used to collect timing data at any time. A second example shows the slowdown possible from switching to the slower acpi_pm time source, on the same system used for the fast results above:

# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
tsc hpet acpi_pm
# echo acpi_pm > /sys/devices/system/clocksource/clocksource0/current_clocksource
# pg_test_timing
Per loop time including overhead: 722.92 ns
Histogram of timing durations:
  < us   % of total      count
     1     27.84870    1155682
     2     72.05956    2990371
     4      0.07810       3241
     8      0.01357        563
    16      0.00007          3

In this configuration, the sample EXPLAIN ANALYZE above takes 115.9 ms. That's 1061 ns of timing overhead, again a small multiple of what's measured directly by this utility. That much timing overhead means the actual query itself is only taking a tiny fraction of the accounted for time, most of it is being consumed in overhead instead. In this configuration, any EXPLAIN ANALYZE totals involving many timed operations would be inflated significantly by timing overhead.

FreeBSD also allows changing the time source on the fly, and it logs information about the timer selected during boot:

# dmesg | grep "Timecounter"
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
Timecounter "i8254" frequency 1193182 Hz quality 0
Timecounters tick every 10.000 msec
Timecounter "TSC" frequency 2531787134 Hz quality 800
# sysctl kern.timecounter.hardware=TSC
kern.timecounter.hardware: ACPI-fast -> TSC

Other systems may only allow setting the time source on boot. On older Linux systems the "clock" kernel setting is the only way to make this sort of change. And even on some more recent ones, the only option you'll see for a clock source is "jiffies". Jiffies are the older Linux software clock implementation, which can have good resolution when it's backed by fast enough timing hardware, as in this example:

$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource
jiffies
$ dmesg | grep time.c
time.c: Using 3.579545 MHz WALL PM GTOD PIT/TSC timer.
time.c: Detected 2400.153 MHz processor.
$ pg_test_timing
Testing timing overhead for 3 seconds.
Per timing duration including loop overhead: 97.75 ns
Histogram of timing durations:
  < us   % of total      count
     1     90.23734   27694571
     2      9.75277    2993204
     4      0.00981       3010
     8      0.00007         22
    16      0.00000          1
    32      0.00000          1

Clock Hardware and Timing Accuracy

Collecting accurate timing information is normally done on computers using hardware clocks with various levels of accuracy. With some hardware the operating systems can pass the system clock time almost directly to programs. A system clock can also be derived from a chip that simply provides timing interrupts, periodic ticks at some known time interval. In either case, operating system kernels provide a clock source that hides these details. But the accuracy of that clock source and how quickly it can return results varies based on the underlying hardware.

Inaccurate time keeping can result in system instability. Test any change to the clock source very carefully. Operating system defaults are sometimes made to favor reliability over best accuracy. And if you are using a virtual machine, look into the recommended time sources compatible with it. Virtual hardware faces additional difficulties when emulating timers, and there are often per operating system settings suggested by vendors.

The Time Stamp Counter (TSC) clock source is the most accurate one available on current generation CPUs. It's the preferred way to track the system time when it's supported by the operating system and the TSC clock is reliable. There are several ways that TSC can fail to provide an accurate timing source, making it unreliable. Older systems can have a TSC clock that varies based on the CPU temperature, making it unusable for timing. Trying to use TSC on some older multicore CPUs can give a reported time that's inconsistent among multiple cores. This can result in the time going backwards, a problem this program checks for. And even the newest systems can fail to provide accurate TSC timing with very aggressive power saving configurations.

Newer operating systems may check for the known TSC problems and switch to a slower, more stable clock source when they are seen. If your system supports TSC time but doesn't default to that, it may be disabled for a good reason. And some operating systems may not detect all the possible problems correctly, or will allow using TSC even in situations where it's known to be inaccurate.

The High Precision Event Timer (HPET) is the preferred timer on systems where it's available and TSC is not accurate. The timer chip itself is programmable to allow up to 100 nanosecond resolution, but you may not see that much accuracy in your system clock.

Advanced Configuration and Power Interface (ACPI) provides a Power Management (PM) Timer, which Linux refers to as the acpi_pm. The clock derived from acpi_pm will at best provide 300 nanosecond resolution.

Timers used on older PC hardware include the 8254 Programmable Interval Timer (PIT), the real-time clock (RTC), the Advanced Programmable Interrupt Controller (APIC) timer, and the Cyclone timer. These timers aim for millisecond resolution.

See Also

EXPLAIN
Prev | Up | Next
pg_test_fsync | Home | pg_upgrade


pg_upgrade
Prev Up PostgreSQL Server Applications Home Next

pg_upgrade

pg_upgrade — upgrade a PostgreSQL server instance

Synopsis

pg_upgrade -b oldbindir [-B newbindir] -d oldconfigdir -D newconfigdir [option...]

Description

pg_upgrade (formerly called pg_migrator) allows data stored in PostgreSQL data files to be upgraded to a later PostgreSQL major version without the data dump/reload typically required for major version upgrades, e.g., from 9.5.8 to 9.6.4 or from 10.7 to 11.2. It is not required for minor version upgrades, e.g., from 9.6.2 to 9.6.3 or from 10.1 to 10.2.

Major PostgreSQL releases regularly add new features that often change the layout of the system tables, but the internal data storage format rarely changes. pg_upgrade uses this fact to perform rapid upgrades by creating new system tables and simply reusing the old user data files. If a future major release ever changes the data storage format in a way that makes the old data format unreadable, pg_upgrade will not be usable for such upgrades. (The community will attempt to avoid such situations.)

pg_upgrade does its best to make sure the old and new clusters are binary-compatible, e.g., by checking for compatible compile-time settings, including 32/64-bit binaries. It is important that any external modules are also binary compatible, though this cannot be checked by pg_upgrade.

pg_upgrade supports upgrades from 8.4.X and later to the current major release of PostgreSQL, including snapshot and beta releases.

Options

pg_upgrade accepts the following command-line arguments:

-b bindir
--old-bindir=bindir

the old PostgreSQL executable directory; environment variable PGBINOLD

-B bindir
--new-bindir=bindir

the new PostgreSQL executable directory; default is the directory where pg_upgrade resides; environment variable PGBINNEW

-c
--check

check clusters only, don't change any data

-d configdir
--old-datadir=configdir

the old database cluster configuration directory; environment variable PGDATAOLD

-D configdir
--new-datadir=configdir

the new database cluster configuration directory; environment variable PGDATANEW

-j njobs
--jobs=njobs

number of simultaneous processes or threads to use

-k
--link

use hard links instead of copying files to the new cluster

-o options
--old-options options

options to be passed directly to the old postgres command; multiple option invocations are appended

-O options
--new-options options

options to be passed directly to the new postgres command; multiple option invocations are appended

-p port
--old-port=port

the old cluster port number; environment variable PGPORTOLD

-P port
--new-port=port

the new cluster port number; environment variable PGPORTNEW

-r
--retain

retain SQL and log files even after successful completion

-s dir
--socketdir=dir

directory to use for postmaster sockets during upgrade; default is current working directory; environment variable PGSOCKETDIR

-U username
--username=username

cluster's install user name; environment variable PGUSER

-v
--verbose

enable verbose internal logging

-V
--version

display version information, then exit

--clone

Use efficient file cloning (also known as “reflinks” on some systems) instead of copying files to the new cluster. This can result in near-instantaneous copying of the data files, giving the speed advantages of -k/--link while leaving the old cluster untouched.

File cloning is only supported on some operating systems and file systems. If it is selected but not supported, the pg_upgrade run will error. At present, it is supported on Linux (kernel 4.5 or later) with Btrfs and XFS (on file systems created with reflink support), and on macOS with APFS.

-?
--help

show help, then exit

Usage

These are the steps to perform an upgrade with pg_upgrade:

  1. Optionally move the old cluster

    If you are using a version-specific installation directory, e.g., /opt/PostgreSQL/14, you do not need to move the old cluster. The graphical installers all use version-specific installation directories.

    If your installation directory is not version-specific, e.g., /usr/local/pgsql, it is necessary to move the current PostgreSQL install directory so it does not interfere with the new PostgreSQL installation. Once the current PostgreSQL server is shut down, it is safe to rename the PostgreSQL installation directory; assuming the old directory is /usr/local/pgsql, you can do:

    mv /usr/local/pgsql /usr/local/pgsql.old

    to rename the directory.

  2. For source installs, build the new version

    Build the new PostgreSQL source with configure flags that are compatible with the old cluster. pg_upgrade will check pg_controldata to make sure all settings are compatible before starting the upgrade.

  3. Install the new PostgreSQL binaries

    Install the new server's binaries and support files. pg_upgrade is included in a default installation.

    For source installs, if you wish to install the new server in a custom location, use the prefix variable:

    make prefix=/usr/local/pgsql.new install

  4. Initialize the new PostgreSQL cluster

    Initialize the new cluster using initdb. Again, use compatible initdb flags that match the old cluster. Many prebuilt installers do this step automatically. There is no need to start the new cluster.

  5. Install extension shared object files

    Many extensions and custom modules, whether from contrib or another source, use shared object files (or DLLs), e.g., pgcrypto.so. If the old cluster used these, shared object files matching the new server binary must be installed in the new cluster, usually via operating system commands. Do not load the schema definitions, e.g., CREATE EXTENSION pgcrypto, because these will be duplicated from the old cluster. If extension updates are available, pg_upgrade will report this and create a script that can be run later to update them.

  6. Copy custom full-text search files

    Copy any custom full text search files (dictionary, synonym, thesaurus, stop words) from the old to the new cluster.

  7. Adjust authentication

    pg_upgrade will connect to the old and new servers several times, so you might want to set authentication to peer in pg_hba.conf or use a ~/.pgpass file (see Section 34.16).

  8. Stop both servers

    Make sure both database servers are stopped using, on Unix, e.g.:

    pg_ctl -D /opt/PostgreSQL/9.6 stop
pg_ctl -D /opt/PostgreSQL/14 stop

    or on Windows, using the proper service names:

    NET STOP postgresql-9.6
NET STOP postgresql-14

    Streaming replication and log-shipping standby servers can remain running until a later step.

  9. Prepare for standby server upgrades

    If you are upgrading standby servers using methods outlined in section Step 11, verify that the old standby servers are caught up by running pg_controldata against the old primary and standby clusters. Verify that the “Latest checkpoint location” values match in all clusters. (There will be a mismatch if old standby servers were shut down before the old primary or if the old standby servers are still running.) Also, make sure wal_level is not set to minimal in the postgresql.conf file on the new primary cluster.

  10. Run pg_upgrade

    Always run the pg_upgrade binary of the new server, not the old one. pg_upgrade requires the specification of the old and new cluster's data and executable (bin) directories. You can also specify user and port values, and whether you want the data files linked or cloned instead of the default copy behavior.

    If you use link mode, the upgrade will be much faster (no file copying) and use less disk space, but you will not be able to access your old cluster once you start the new cluster after the upgrade. Link mode also requires that the old and new cluster data directories be in the same file system. (Tablespaces and pg_wal can be on different file systems.) Clone mode provides the same speed and disk space advantages but does not cause the old cluster to be unusable once the new cluster is started. Clone mode also requires that the old and new data directories be in the same file system. This mode is only available on certain operating systems and file systems.

    The --jobs option allows multiple CPU cores to be used for copying/linking of files and to dump and reload database schemas in parallel; a good place to start is the maximum of the number of CPU cores and tablespaces. This option can dramatically reduce the time to upgrade a multi-database server running on a multiprocessor machine.

    For Windows users, you must be logged into an administrative account, and then start a shell as the postgres user and set the proper path:

    RUNAS /USER:postgres "CMD.EXE"
SET PATH=%PATH%;C:\Program Files\PostgreSQL\14\bin;

    and then run pg_upgrade with quoted directories, e.g.:

    pg_upgrade.exe
        --old-datadir "C:/Program Files/PostgreSQL/9.6/data"
        --new-datadir "C:/Program Files/PostgreSQL/14/data"
        --old-bindir "C:/Program Files/PostgreSQL/9.6/bin"
        --new-bindir "C:/Program Files/PostgreSQL/14/bin"

    Once started, pg_upgrade will verify the two clusters are compatible and then do the upgrade. You can use pg_upgrade --check to perform only the checks, even if the old server is still running. pg_upgrade --check will also outline any manual adjustments you will need to make after the upgrade. If you are going to be using link or clone mode, you should use the option --link or --clone with --check to enable mode-specific checks. pg_upgrade requires write permission in the current directory.

    Obviously, no one should be accessing the clusters during the upgrade. pg_upgrade defaults to running servers on port 50432 to avoid unintended client connections. You can use the same port number for both clusters when doing an upgrade because the old and new clusters will not be running at the same time. However, when checking an old running server, the old and new port numbers must be different.

    If an error occurs while restoring the database schema, pg_upgrade will exit and you will have to revert to the old cluster as outlined in Step 17 below. To try pg_upgrade again, you will need to modify the old cluster so the pg_upgrade schema restore succeeds. If the problem is a contrib module, you might need to uninstall the contrib module from the old cluster and install it in the new cluster after the upgrade, assuming the module is not being used to store user data.

  11. Upgrade streaming replication and log-shipping standby servers

    If you used link mode and have Streaming Replication (see Section 27.2.5) or Log-Shipping (see Section 27.2) standby servers, you can follow these steps to quickly upgrade them. You will not be running pg_upgrade on the standby servers, but rather rsync on the primary. Do not start any servers yet.

    If you did not use link mode, do not have or do not want to use rsync, or want an easier solution, skip the instructions in this section and simply recreate the standby servers once pg_upgrade completes and the new primary is running.

    1. Install the new PostgreSQL binaries on standby servers

      Make sure the new binaries and support files are installed on all standby servers.

    2. Make sure the new standby data directories do not exist

      Make sure the new standby data directories do not exist or are empty. If initdb was run, delete the standby servers' new data directories.

    3. Install extension shared object files

      Install the same extension shared object files on the new standbys that you installed in the new primary cluster.

    4. Stop standby servers

      If the standby servers are still running, stop them now using the above instructions.

    5. Save configuration files

      Save any configuration files from the old standbys' configuration directories you need to keep, e.g., postgresql.conf (and any files included by it), postgresql.auto.conf, pg_hba.conf, because these will be overwritten or removed in the next step.

    6. Run rsync

      When using link mode, standby servers can be quickly upgraded using rsync. To accomplish this, from a directory on the primary server that is above the old and new database cluster directories, run this on the primary for each standby server:

      rsync --archive --delete --hard-links --size-only --no-inc-recursive old_cluster new_cluster remote_dir

      where old_cluster and new_cluster are relative to the current directory on the primary, and remote_dir is above the old and new cluster directories on the standby. The directory structure under the specified directories on the primary and standbys must match. Consult the rsync manual page for details on specifying the remote directory, e.g.,

      rsync --archive --delete --hard-links --size-only --no-inc-recursive /opt/PostgreSQL/9.5 \
      /opt/PostgreSQL/9.6 standby.example.com:/opt/PostgreSQL

      You can verify what the command will do using rsync's --dry-run option. While rsync must be run on the primary for at least one standby, it is possible to run rsync on an upgraded standby to upgrade other standbys, as long as the upgraded standby has not been started.

      What this does is to record the links created by pg_upgrade's link mode that connect files in the old and new clusters on the primary server. It then finds matching files in the standby's old cluster and creates links for them in the standby's new cluster. Files that were not linked on the primary are copied from the primary to the standby. (They are usually small.) This provides rapid standby upgrades. Unfortunately, rsync needlessly copies files associated with temporary and unlogged tables because these files don't normally exist on standby servers.

      If you have tablespaces, you will need to run a similar rsync command for each tablespace directory, e.g.:

      rsync --archive --delete --hard-links --size-only --no-inc-recursive /vol1/pg_tblsp/PG_9.5_201510051 \
      /vol1/pg_tblsp/PG_9.6_201608131 standby.example.com:/vol1/pg_tblsp

      If you have relocated pg_wal outside the data directories, rsync must be run on those directories too.

    7. Configure streaming replication and log-shipping standby servers

      Configure the servers for log shipping. (You do not need to run pg_start_backup() and pg_stop_backup() or take a file system backup as the standbys are still synchronized with the primary.)

  12. Restore pg_hba.conf

    If you modified pg_hba.conf, restore its original settings. It might also be necessary to adjust other configuration files in the new cluster to match the old cluster, e.g., postgresql.conf (and any files included by it), postgresql.auto.conf.

  13. Start the new server

    The new server can now be safely started, and then any rsync'ed standby servers.

  14. Post-upgrade processing

    If any post-upgrade processing is required, pg_upgrade will issue warnings as it completes. It will also generate script files that must be run by the administrator. The script files will connect to each database that needs post-upgrade processing. Each script should be run using:

    psql --username=postgres --file=script.sql postgres

    The scripts can be run in any order and can be deleted once they have been run.

    Caution

    In general it is unsafe to access tables referenced in rebuild scripts until the rebuild scripts have run to completion; doing so could yield incorrect results or poor performance. Tables not referenced in rebuild scripts can be accessed immediately.

  15. Statistics

    Because optimizer statistics are not transferred by pg_upgrade, you will be instructed to run a command to regenerate that information at the end of the upgrade. You might need to set connection parameters to match your new cluster.

  16. Delete old cluster

    Once you are satisfied with the upgrade, you can delete the old cluster's data directories by running the script mentioned when pg_upgrade completes. (Automatic deletion is not possible if you have user-defined tablespaces inside the old data directory.) You can also delete the old installation directories (e.g., bin, share).

  17. Reverting to old cluster

    If, after running pg_upgrade, you wish to revert to the old cluster, there are several options:

    • If the --check option was used, the old cluster was unmodified; it can be restarted.

    • If the --link option was not used, the old cluster was unmodified; it can be restarted.

    • If the --link option was used, the data files might be shared between the old and new cluster:

      • If pg_upgrade aborted before linking started, the old cluster was unmodified; it can be restarted.

      • If you did not start the new cluster, the old cluster was unmodified except that, when linking started, a .old suffix was appended to $PGDATA/global/pg_control. To reuse the old cluster, remove the .old suffix from $PGDATA/global/pg_control; you can then restart the old cluster.

      • If you did start the new cluster, it has written to shared files and it is unsafe to use the old cluster. The old cluster will need to be restored from backup in this case.

Notes

pg_upgrade creates various working files, such as schema dumps, in the current working directory. For security, be sure that that directory is not readable or writable by any other users.

pg_upgrade launches short-lived postmasters in the old and new data directories. Temporary Unix socket files for communication with these postmasters are, by default, made in the current working directory. In some situations the path name for the current directory might be too long to be a valid socket name. In that case you can use the -s option to put the socket files in some directory with a shorter path name. For security, be sure that that directory is not readable or writable by any other users. (This is not supported on Windows.)

All failure, rebuild, and reindex cases will be reported by pg_upgrade if they affect your installation; post-upgrade scripts to rebuild tables and indexes will be generated automatically. If you are trying to automate the upgrade of many clusters, you should find that clusters with identical database schemas require the same post-upgrade steps for all cluster upgrades; this is because the post-upgrade steps are based on the database schemas, and not user data.

For deployment testing, create a schema-only copy of the old cluster, insert dummy data, and upgrade that.

pg_upgrade does not support upgrading of databases containing table columns using these reg* OID-referencing system data types:

regcollation
regconfig
regdictionary
regnamespace
regoper
regoperator
regproc
regprocedure

(regclass, regrole, and regtype can be upgraded.)

If you are upgrading a pre-PostgreSQL 9.2 cluster that uses a configuration-file-only directory, you must pass the real data directory location to pg_upgrade, and pass the configuration directory location to the server, e.g., -d /real-data-directory -o '-D /configuration-directory'.

If using a pre-9.1 old server that is using a non-default Unix-domain socket directory or a default that differs from the default of the new cluster, set PGHOST to point to the old server's socket location. (This is not relevant on Windows.)

If you want to use link mode and you do not want your old cluster to be modified when the new cluster is started, consider using the clone mode. If that is not available, make a copy of the old cluster and upgrade that in link mode. To make a valid copy of the old cluster, use rsync to create a dirty copy of the old cluster while the server is running, then shut down the old server and run rsync --checksum again to update the copy with any changes to make it consistent. (--checksum is necessary because rsync only has file modification-time granularity of one second.) You might want to exclude some files, e.g., postmaster.pid, as documented in Section 26.3.3. If your file system supports file system snapshots or copy-on-write file copies, you can use that to make a backup of the old cluster and tablespaces, though the snapshot and copies must be created simultaneously or while the database server is down.

See Also

initdb, pg_ctl, pg_dump, postgres
Prev | Up | Next
pg_test_timing | Home | pg_waldump


pg_waldump
Prev Up PostgreSQL Server Applications Home Next

pg_waldump

pg_waldump — display a human-readable rendering of the write-ahead log of a PostgreSQL database cluster

Synopsis

pg_waldump [option...] [startseg [endseg]]

Description

pg_waldump displays the write-ahead log (WAL) and is mainly useful for debugging or educational purposes.

This utility can only be run by the user who installed the server, because it requires read-only access to the data directory.

Options

The following command-line options control the location and format of the output:

startseg

Start reading at the specified log segment file. This implicitly determines the path in which files will be searched for, and the timeline to use.

endseg

Stop after reading the specified log segment file.

-b
--bkp-details

Output detailed information about backup blocks.

-e end
--end=end

Stop reading at the specified WAL location, instead of reading to the end of the log stream.

-f
--follow

After reaching the end of valid WAL, keep polling once per second for new WAL to appear.

-n limit
--limit=limit

Display the specified number of records, then stop.

-p path
--path=path

Specifies a directory to search for log segment files or a directory with a pg_wal subdirectory that contains such files. The default is to search in the current directory, the pg_wal subdirectory of the current directory, and the pg_wal subdirectory of PGDATA.

-q
--quiet

Do not print any output, except for errors. This option can be useful when you want to know whether a range of WAL records can be successfully parsed but don't care about the record contents.

-r rmgr
--rmgr=rmgr

Only display records generated by the specified resource manager. If list is passed as name, print a list of valid resource manager names, and exit.

-s start
--start=start

WAL location at which to start reading. The default is to start reading the first valid log record found in the earliest file found.

-t timeline
--timeline=timeline

Timeline from which to read log records. The default is to use the value in startseg, if that is specified; otherwise, the default is 1.

-V
--version

Print the pg_waldump version and exit.

-x xid
--xid=xid

Only display records marked with the given transaction ID.

-z
--stats[=record]

Display summary statistics (number and size of records and full-page images) instead of individual records. Optionally generate statistics per-record instead of per-rmgr.

-?
--help

Show help about pg_waldump command line arguments, and exit.

Environment

PGDATA

Data directory; see also the -p option.

PG_COLOR

Specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

Notes

Can give wrong results when the server is running.

Only the specified timeline is displayed (or the default, if none is specified). Records in other timelines are ignored.

pg_waldump cannot read WAL files with suffix .partial. If those files need to be read, .partial suffix needs to be removed from the file name.

See Also

Section 30.6
Prev | Up | Next
pg_upgrade | Home | postgres


postgres
Prev Up PostgreSQL Server Applications Home Next

postgres

postgres — PostgreSQL database server

Synopsis

postgres [option...]

Description

postgres is the PostgreSQL database server. In order for a client application to access a database it connects (over a network or locally) to a running postgres instance. The postgres instance then starts a separate server process to handle the connection.

One postgres instance always manages the data of exactly one database cluster. A database cluster is a collection of databases that is stored at a common file system location (the “data area”). More than one postgres instance can run on a system at one time, so long as they use different data areas and different communication ports (see below). When postgres starts it needs to know the location of the data area. The location must be specified by the -D option or the PGDATA environment variable; there is no default. Typically, -D or PGDATA points directly to the data area directory created by initdb. Other possible file layouts are discussed in Section 20.2.

By default postgres starts in the foreground and prints log messages to the standard error stream. In practical applications postgres should be started as a background process, perhaps at boot time.

The postgres command can also be called in single-user mode. The primary use for this mode is during bootstrapping by initdb. Sometimes it is used for debugging or disaster recovery; note that running a single-user server is not truly suitable for debugging the server, since no realistic interprocess communication and locking will happen. When invoked in single-user mode from the shell, the user can enter queries and the results will be printed to the screen, but in a form that is more useful for developers than end users. In the single-user mode, the session user will be set to the user with ID 1, and implicit superuser powers are granted to this user. This user does not actually have to exist, so the single-user mode can be used to manually recover from certain kinds of accidental damage to the system catalogs.

Options

postgres accepts the following command-line arguments. For a detailed discussion of the options consult Chapter 20. You can save typing most of these options by setting up a configuration file. Some (safe) options can also be set from the connecting client in an application-dependent way to apply only for that session. For example, if the environment variable PGOPTIONS is set, then libpq-based clients will pass that string to the server, which will interpret it as postgres command-line options.

General Purpose

-B nbuffers

Sets the number of shared buffers for use by the server processes. The default value of this parameter is chosen automatically by initdb. Specifying this option is equivalent to setting the shared_buffers configuration parameter.

-c name=value

Sets a named run-time parameter. The configuration parameters supported by PostgreSQL are described in Chapter 20. Most of the other command line options are in fact short forms of such a parameter assignment. -c can appear multiple times to set multiple parameters.

-C name

Prints the value of the named run-time parameter, and exits. (See the -c option above for details.) This can be used on a running server, and returns values from postgresql.conf, modified by any parameters supplied in this invocation. It does not reflect parameters supplied when the cluster was started.

This option is meant for other programs that interact with a server instance, such as pg_ctl, to query configuration parameter values. User-facing applications should instead use SHOW or the pg_settings view.

-d debug-level

Sets the debug level. The higher this value is set, the more debugging output is written to the server log. Values are from 1 to 5. It is also possible to pass -d 0 for a specific session, which will prevent the server log level of the parent postgres process from being propagated to this session.

-D datadir

Specifies the file system location of the database configuration files. See Section 20.2 for details.

-e

Sets the default date style to “European”, that is DMY ordering of input date fields. This also causes the day to be printed before the month in certain date output formats. See Section 8.5 for more information.

-F

Disables fsync calls for improved performance, at the risk of data corruption in the event of a system crash. Specifying this option is equivalent to disabling the fsync configuration parameter. Read the detailed documentation before using this!

-h hostname

Specifies the IP host name or address on which postgres is to listen for TCP/IP connections from client applications. The value can also be a comma-separated list of addresses, or * to specify listening on all available interfaces. An empty value specifies not listening on any IP addresses, in which case only Unix-domain sockets can be used to connect to the server. Defaults to listening only on localhost. Specifying this option is equivalent to setting the listen_addresses configuration parameter.

-i

Allows remote clients to connect via TCP/IP (Internet domain) connections. Without this option, only local connections are accepted. This option is equivalent to setting listen_addresses to * in postgresql.conf or via -h.

This option is deprecated since it does not allow access to the full functionality of listen_addresses. It's usually better to set listen_addresses directly.

-k directory

Specifies the directory of the Unix-domain socket on which postgres is to listen for connections from client applications. The value can also be a comma-separated list of directories. An empty value specifies not listening on any Unix-domain sockets, in which case only TCP/IP sockets can be used to connect to the server. The default value is normally /tmp, but that can be changed at build time. Specifying this option is equivalent to setting the unix_socket_directories configuration parameter.

-l

Enables secure connections using SSL. PostgreSQL must have been compiled with support for SSL for this option to be available. For more information on using SSL, refer to Section 19.9.

-N max-connections

Sets the maximum number of client connections that this server will accept. The default value of this parameter is chosen automatically by initdb. Specifying this option is equivalent to setting the max_connections configuration parameter.

-p port

Specifies the TCP/IP port or local Unix domain socket file extension on which postgres is to listen for connections from client applications. Defaults to the value of the PGPORT environment variable, or if PGPORT is not set, then defaults to the value established during compilation (normally 5432). If you specify a port other than the default port, then all client applications must specify the same port using either command-line options or PGPORT.

-s

Print time information and other statistics at the end of each command. This is useful for benchmarking or for use in tuning the number of buffers.

-S work-mem

Specifies the base amount of memory to be used by sorts and hash tables before resorting to temporary disk files. See the description of the work_mem configuration parameter in Section 20.4.1.

-V
--version

Print the postgres version and exit.

--name=value

Sets a named run-time parameter; a shorter form of -c.

--describe-config

This option dumps out the server's internal configuration variables, descriptions, and defaults in tab-delimited COPY format. It is designed primarily for use by administration tools.

-?
--help

Show help about postgres command line arguments, and exit.

Semi-Internal Options

The options described here are used mainly for debugging purposes, and in some cases to assist with recovery of severely damaged databases. There should be no reason to use them in a production database setup. They are listed here only for use by PostgreSQL system developers. Furthermore, these options might change or be removed in a future release without notice.

-f { s | i | o | b | t | n | m | h }

Forbids the use of particular scan and join methods: s and i disable sequential and index scans respectively, o, b and t disable index-only scans, bitmap index scans, and TID scans respectively, while n, m, and h disable nested-loop, merge and hash joins respectively.

Neither sequential scans nor nested-loop joins can be disabled completely; the -fs and -fn options simply discourage the optimizer from using those plan types if it has any other alternative.

-n

This option is for debugging problems that cause a server process to die abnormally. The ordinary strategy in this situation is to notify all other server processes that they must terminate and then reinitialize the shared memory and semaphores. This is because an errant server process could have corrupted some shared state before terminating. This option specifies that postgres will not reinitialize shared data structures. A knowledgeable system programmer can then use a debugger to examine shared memory and semaphore state.

-O

Allows the structure of system tables to be modified. This is used by initdb.

-P

Ignore system indexes when reading system tables, but still update the indexes when modifying the tables. This is useful when recovering from damaged system indexes.

-t pa[rser] | pl[anner] | e[xecutor]

Print timing statistics for each query relating to each of the major system modules. This option cannot be used together with the -s option.

-T

This option is for debugging problems that cause a server process to die abnormally. The ordinary strategy in this situation is to notify all other server processes that they must terminate and then reinitialize the shared memory and semaphores. This is because an errant server process could have corrupted some shared state before terminating. This option specifies that postgres will stop all other server processes by sending the signal SIGSTOP, but will not cause them to terminate. This permits system programmers to collect core dumps from all server processes by hand.

-v protocol

Specifies the version number of the frontend/backend protocol to be used for a particular session. This option is for internal use only.

-W seconds

A delay of this many seconds occurs when a new server process is started, after it conducts the authentication procedure. This is intended to give an opportunity to attach to the server process with a debugger.

Options for Single-User Mode

The following options only apply to the single-user mode (see Single-User Mode below).

--single

Selects the single-user mode. This must be the first argument on the command line.

database

Specifies the name of the database to be accessed. This must be the last argument on the command line. If it is omitted it defaults to the user name.

-E

Echo all commands to standard output before executing them.

-j

Use semicolon followed by two newlines, rather than just newline, as the command entry terminator.

-r filename

Send all server log output to filename. This option is only honored when supplied as a command-line option.

Environment

PGCLIENTENCODING

Default character encoding used by clients. (The clients can override this individually.) This value can also be set in the configuration file.

PGDATA

Default data directory location

PGDATESTYLE

Default value of the DateStyle run-time parameter. (The use of this environment variable is deprecated.)

PGPORT

Default port number (preferably set in the configuration file)

Diagnostics

A failure message mentioning semget or shmget probably indicates you need to configure your kernel to provide adequate shared memory and semaphores. For more discussion see Section 19.4. You might be able to postpone reconfiguring your kernel by decreasing shared_buffers to reduce the shared memory consumption of PostgreSQL, and/or by reducing max_connections to reduce the semaphore consumption.

A failure message suggesting that another server is already running should be checked carefully, for example by using the command

$ ps ax | grep postgres

or

$ ps -ef | grep postgres

depending on your system. If you are certain that no conflicting server is running, you can remove the lock file mentioned in the message and try again.

A failure message indicating inability to bind to a port might indicate that that port is already in use by some non-PostgreSQL process. You might also get this error if you terminate postgres and immediately restart it using the same port; in this case, you must simply wait a few seconds until the operating system closes the port before trying again. Finally, you might get this error if you specify a port number that your operating system considers to be reserved. For example, many versions of Unix consider port numbers under 1024 to be “trusted” and only permit the Unix superuser to access them.

Notes

The utility command pg_ctl can be used to start and shut down the postgres server safely and comfortably.

If at all possible, do not use SIGKILL to kill the main postgres server. Doing so will prevent postgres from freeing the system resources (e.g., shared memory and semaphores) that it holds before terminating. This might cause problems for starting a fresh postgres run.

To terminate the postgres server normally, the signals SIGTERM, SIGINT, or SIGQUIT can be used. The first will wait for all clients to terminate before quitting, the second will forcefully disconnect all clients, and the third will quit immediately without proper shutdown, resulting in a recovery run during restart.

The SIGHUP signal will reload the server configuration files. It is also possible to send SIGHUP to an individual server process, but that is usually not sensible.

To cancel a running query, send the SIGINT signal to the process running that command. To terminate a backend process cleanly, send SIGTERM to that process. See also pg_cancel_backend and pg_terminate_backend in Section 9.27.2 for the SQL-callable equivalents of these two actions.

The postgres server uses SIGQUIT to tell subordinate server processes to terminate without normal cleanup. This signal should not be used by users. It is also unwise to send SIGKILL to a server process — the main postgres process will interpret this as a crash and will force all the sibling processes to quit as part of its standard crash-recovery procedure.

Bugs

The -- options will not work on FreeBSD or OpenBSD. Use -c instead. This is a bug in the affected operating systems; a future release of PostgreSQL will provide a workaround if this is not fixed.

Single-User Mode

To start a single-user mode server, use a command like

postgres --single -D /usr/local/pgsql/data other-options my_database

Provide the correct path to the database directory with -D, or make sure that the environment variable PGDATA is set. Also specify the name of the particular database you want to work in.

Normally, the single-user mode server treats newline as the command entry terminator; there is no intelligence about semicolons, as there is in psql. To continue a command across multiple lines, you must type backslash just before each newline except the last one. The backslash and adjacent newline are both dropped from the input command. Note that this will happen even when within a string literal or comment.

But if you use the -j command line switch, a single newline does not terminate command entry; instead, the sequence semicolon-newline-newline does. That is, type a semicolon immediately followed by a completely empty line. Backslash-newline is not treated specially in this mode. Again, there is no intelligence about such a sequence appearing within a string literal or comment.

In either input mode, if you type a semicolon that is not just before or part of a command entry terminator, it is considered a command separator. When you do type a command entry terminator, the multiple statements you've entered will be executed as a single transaction.

To quit the session, type EOF (Control+D, usually). If you've entered any text since the last command entry terminator, then EOF will be taken as a command entry terminator, and another EOF will be needed to exit.

Note that the single-user mode server does not provide sophisticated line-editing features (no command history, for example). Single-user mode also does not do any background processing, such as automatic checkpoints or replication.

Examples

To start postgres in the background using default values, type:

$ nohup postgres >logfile 2>&1 </dev/null &

To start postgres with a specific port, e.g., 1234:

$ postgres -p 1234

To connect to this server using psql, specify this port with the -p option:

$ psql -p 1234

or set the environment variable PGPORT:

$ export PGPORT=1234
$ psql

Named run-time parameters can be set in either of these styles:

$ postgres -c work_mem=1234
$ postgres --work-mem=1234

Either form overrides whatever setting might exist for work_mem in postgresql.conf. Notice that underscores in parameter names can be written as either underscore or dash on the command line. Except for short-term experiments, it's probably better practice to edit the setting in postgresql.conf than to rely on a command-line switch to set a parameter.

See Also

initdb, pg_ctl

Prev | Up | Next
pg_waldump | Home | postmaster


postmaster
Prev Up PostgreSQL Server Applications Home Next

postmaster

postmaster — PostgreSQL database server

Synopsis

postmaster [option...]

Description

postmaster is a deprecated alias of postgres.

See Also

postgres

Prev | Up | Next
postgres | Home | Part VII. Internals


Part VII. Internals
Prev Up PostgreSQL 14.4 Documentation Home Next

Part VII. Internals

This part contains assorted information that might be of use to PostgreSQL developers.

Table of Contents

51. Overview of PostgreSQL Internals
51.1. The Path of a Query
51.2. How Connections Are Established
51.3. The Parser Stage
51.4. The PostgreSQL Rule System
51.5. Planner/Optimizer
51.6. Executor
52. System Catalogs
52.1. Overview
52.2. pg_aggregate
52.3. pg_am
52.4. pg_amop
52.5. pg_amproc
52.6. pg_attrdef
52.7. pg_attribute
52.8. pg_authid
52.9. pg_auth_members
52.10. pg_cast
52.11. pg_class
52.12. pg_collation
52.13. pg_constraint
52.14. pg_conversion
52.15. pg_database
52.16. pg_db_role_setting
52.17. pg_default_acl
52.18. pg_depend
52.19. pg_description
52.20. pg_enum
52.21. pg_event_trigger
52.22. pg_extension
52.23. pg_foreign_data_wrapper
52.24. pg_foreign_server
52.25. pg_foreign_table
52.26. pg_index
52.27. pg_inherits
52.28. pg_init_privs
52.29. pg_language
52.30. pg_largeobject
52.31. pg_largeobject_metadata
52.32. pg_namespace
52.33. pg_opclass
52.34. pg_operator
52.35. pg_opfamily
52.36. pg_partitioned_table
52.37. pg_policy
52.38. pg_proc
52.39. pg_publication
52.40. pg_publication_rel
52.41. pg_range
52.42. pg_replication_origin
52.43. pg_rewrite
52.44. pg_seclabel
52.45. pg_sequence
52.46. pg_shdepend
52.47. pg_shdescription
52.48. pg_shseclabel
52.49. pg_statistic
52.50. pg_statistic_ext
52.51. pg_statistic_ext_data
52.52. pg_subscription
52.53. pg_subscription_rel
52.54. pg_tablespace
52.55. pg_transform
52.56. pg_trigger
52.57. pg_ts_config
52.58. pg_ts_config_map
52.59. pg_ts_dict
52.60. pg_ts_parser
52.61. pg_ts_template
52.62. pg_type
52.63. pg_user_mapping
52.64. System Views
52.65. pg_available_extensions
52.66. pg_available_extension_versions
52.67. pg_backend_memory_contexts
52.68. pg_config
52.69. pg_cursors
52.70. pg_file_settings
52.71. pg_group
52.72. pg_hba_file_rules
52.73. pg_indexes
52.74. pg_locks
52.75. pg_matviews
52.76. pg_policies
52.77. pg_prepared_statements
52.78. pg_prepared_xacts
52.79. pg_publication_tables
52.80. pg_replication_origin_status
52.81. pg_replication_slots
52.82. pg_roles
52.83. pg_rules
52.84. pg_seclabels
52.85. pg_sequences
52.86. pg_settings
52.87. pg_shadow
52.88. pg_shmem_allocations
52.89. pg_stats
52.90. pg_stats_ext
52.91. pg_stats_ext_exprs
52.92. pg_tables
52.93. pg_timezone_abbrevs
52.94. pg_timezone_names
52.95. pg_user
52.96. pg_user_mappings
52.97. pg_views
53. Frontend/Backend Protocol
53.1. Overview
53.2. Message Flow
53.3. SASL Authentication
53.4. Streaming Replication Protocol
53.5. Logical Streaming Replication Protocol
53.6. Message Data Types
53.7. Message Formats
53.8. Error and Notice Message Fields
53.9. Logical Replication Message Formats
53.10. Summary of Changes since Protocol 2.0
54. PostgreSQL Coding Conventions
54.1. Formatting
54.2. Reporting Errors Within the Server
54.3. Error Message Style Guide
54.4. Miscellaneous Coding Conventions
55. Native Language Support
55.1. For the Translator
55.2. For the Programmer
56. Writing a Procedural Language Handler
57. Writing a Foreign Data Wrapper
57.1. Foreign Data Wrapper Functions
57.2. Foreign Data Wrapper Callback Routines
57.3. Foreign Data Wrapper Helper Functions
57.4. Foreign Data Wrapper Query Planning
57.5. Row Locking in Foreign Data Wrappers
58. Writing a Table Sampling Method
58.1. Sampling Method Support Functions
59. Writing a Custom Scan Provider
59.1. Creating Custom Scan Paths
59.2. Creating Custom Scan Plans
59.3. Executing Custom Scans
60. Genetic Query Optimizer
60.1. Query Handling as a Complex Optimization Problem
60.2. Genetic Algorithms
60.3. Genetic Query Optimization (GEQO) in PostgreSQL
60.4. Further Reading
61. Table Access Method Interface Definition
62. Index Access Method Interface Definition
62.1. Basic API Structure for Indexes
62.2. Index Access Method Functions
62.3. Index Scanning
62.4. Index Locking Considerations
62.5. Index Uniqueness Checks
62.6. Index Cost Estimation Functions
63. Generic WAL Records
64. B-Tree Indexes
64.1. Introduction
64.2. Behavior of B-Tree Operator Classes
64.3. B-Tree Support Functions
64.4. Implementation
65. GiST Indexes
65.1. Introduction
65.2. Built-in Operator Classes
65.3. Extensibility
65.4. Implementation
65.5. Examples
66. SP-GiST Indexes
66.1. Introduction
66.2. Built-in Operator Classes
66.3. Extensibility
66.4. Implementation
66.5. Examples
67. GIN Indexes
67.1. Introduction
67.2. Built-in Operator Classes
67.3. Extensibility
67.4. Implementation
67.5. GIN Tips and Tricks
67.6. Limitations
67.7. Examples
68. BRIN Indexes
68.1. Introduction
68.2. Built-in Operator Classes
68.3. Extensibility
69. Hash Indexes
69.1. Overview
69.2. Implementation
70. Database Physical Storage
70.1. Database File Layout
70.2. TOAST
70.3. Free Space Map
70.4. Visibility Map
70.5. The Initialization Fork
70.6. Database Page Layout
71. System Catalog Declarations and Initial Contents
71.1. System Catalog Declaration Rules
71.2. System Catalog Initial Data
71.3. BKI File Format
71.4. BKI Commands
71.5. Structure of the Bootstrap BKI File
71.6. BKI Example
72. How the Planner Uses Statistics
72.1. Row Estimation Examples
72.2. Multivariate Statistics Examples
72.3. Planner Statistics and Security
73. Backup Manifest Format
73.1. Backup Manifest Top-level Object
73.2. Backup Manifest File Object
73.3. Backup Manifest WAL Range Object
Prev | Up | Next
postmaster | Home | Chapter 51. Overview of PostgreSQL Internals


Chapter 51. Overview of PostgreSQL Internals
Prev Up Part VII. Internals Home Next

Chapter 51. Overview of PostgreSQL Internals

Table of Contents

51.1. The Path of a Query
51.2. How Connections Are Established
51.3. The Parser Stage
51.3.1. Parser
51.3.2. Transformation Process
51.4. The PostgreSQL Rule System
51.5. Planner/Optimizer
51.5.1. Generating Possible Plans
51.6. Executor

Author

This chapter originated as part of [sim98] Stefan Simkovics' Master's Thesis prepared at Vienna University of Technology under the direction of O.Univ.Prof.Dr. Georg Gottlob and Univ.Ass. Mag. Katrin Seyr.

This chapter gives an overview of the internal structure of the backend of PostgreSQL. After having read the following sections you should have an idea of how a query is processed. This chapter is intended to help the reader understand the general sequence of operations that occur within the backend from the point at which a query is received, to the point at which the results are returned to the client.

Prev | Up | Next
Part VII. Internals | Home | 51.1. The Path of a Query


51.1. The Path of a Query
Prev Up Chapter 51. Overview of PostgreSQL Internals Home Next

51.1. The Path of a Query

Here we give a short overview of the stages a query has to pass to obtain a result.

  1. A connection from an application program to the PostgreSQL server has to be established. The application program transmits a query to the server and waits to receive the results sent back by the server.

  2. The parser stage checks the query transmitted by the application program for correct syntax and creates a query tree.

  3. The rewrite system takes the query tree created by the parser stage and looks for any rules (stored in the system catalogs) to apply to the query tree. It performs the transformations given in the rule bodies.

    One application of the rewrite system is in the realization of views. Whenever a query against a view (i.e., a virtual table) is made, the rewrite system rewrites the user's query to a query that accesses the base tables given in the view definition instead.

  4. The planner/optimizer takes the (rewritten) query tree and creates a query plan that will be the input to the executor.

    It does so by first creating all possible paths leading to the same result. For example if there is an index on a relation to be scanned, there are two paths for the scan. One possibility is a simple sequential scan and the other possibility is to use the index. Next the cost for the execution of each path is estimated and the cheapest path is chosen. The cheapest path is expanded into a complete plan that the executor can use.

  5. The executor recursively steps through the plan tree and retrieves rows in the way represented by the plan. The executor makes use of the storage system while scanning relations, performs sorts and joins, evaluates qualifications and finally hands back the rows derived.

In the following sections we will cover each of the above listed items in more detail to give a better understanding of PostgreSQL's internal control and data structures.

Prev | Up | Next
Chapter 51. Overview of PostgreSQL Internals | Home | 51.2. How Connections Are Established


51.2. How Connections Are Established
Prev Up Chapter 51. Overview of PostgreSQL Internals Home Next

51.2. How Connections Are Established

PostgreSQL implements a “process per user” client/server model. In this model, every client process connects to exactly one backend process. As we do not know ahead of time how many connections will be made, we have to use a “supervisor process” that spawns a new backend process every time a connection is requested. This supervisor process is called postmaster and listens at a specified TCP/IP port for incoming connections. Whenever it detects a request for a connection, it spawns a new backend process. Those backend processes communicate with each other and with other processes of the instance using semaphores and shared memory to ensure data integrity throughout concurrent data access.

The client process can be any program that understands the PostgreSQL protocol described in Chapter 53. Many clients are based on the C-language library libpq, but several independent implementations of the protocol exist, such as the Java JDBC driver.

Once a connection is established, the client process can send a query to the backend process it's connected to. The query is transmitted using plain text, i.e., there is no parsing done in the client. The backend process parses the query, creates an execution plan, executes the plan, and returns the retrieved rows to the client by transmitting them over the established connection.

Prev | Up | Next
51.1. The Path of a Query | Home | 51.3. The Parser Stage


51.3. The Parser Stage
Prev Up Chapter 51. Overview of PostgreSQL Internals Home Next

51.3. The Parser Stage

51.3.1. Parser
51.3.2. Transformation Process

The parser stage consists of two parts:

  • The parser defined in gram.y and scan.l is built using the Unix tools bison and flex.

  • The transformation process does modifications and augmentations to the data structures returned by the parser.

51.3.1. Parser

The parser has to check the query string (which arrives as plain text) for valid syntax. If the syntax is correct a parse tree is built up and handed back; otherwise an error is returned. The parser and lexer are implemented using the well-known Unix tools bison and flex.

The lexer is defined in the file scan.l and is responsible for recognizing identifiers, the SQL key words etc. For every key word or identifier that is found, a token is generated and handed to the parser.

The parser is defined in the file gram.y and consists of a set of grammar rules and actions that are executed whenever a rule is fired. The code of the actions (which is actually C code) is used to build up the parse tree.

The file scan.l is transformed to the C source file scan.c using the program flex and gram.y is transformed to gram.c using bison. After these transformations have taken place a normal C compiler can be used to create the parser. Never make any changes to the generated C files as they will be overwritten the next time flex or bison is called.

Note

The mentioned transformations and compilations are normally done automatically using the makefiles shipped with the PostgreSQL source distribution.

A detailed description of bison or the grammar rules given in gram.y would be beyond the scope of this manual. There are many books and documents dealing with flex and bison. You should be familiar with bison before you start to study the grammar given in gram.y otherwise you won't understand what happens there.

51.3.2. Transformation Process

The parser stage creates a parse tree using only fixed rules about the syntactic structure of SQL. It does not make any lookups in the system catalogs, so there is no possibility to understand the detailed semantics of the requested operations. After the parser completes, the transformation process takes the tree handed back by the parser as input and does the semantic interpretation needed to understand which tables, functions, and operators are referenced by the query. The data structure that is built to represent this information is called the query tree.

The reason for separating raw parsing from semantic analysis is that system catalog lookups can only be done within a transaction, and we do not wish to start a transaction immediately upon receiving a query string. The raw parsing stage is sufficient to identify the transaction control commands (BEGIN, ROLLBACK, etc), and these can then be correctly executed without any further analysis. Once we know that we are dealing with an actual query (such as SELECT or UPDATE), it is okay to start a transaction if we're not already in one. Only then can the transformation process be invoked.

The query tree created by the transformation process is structurally similar to the raw parse tree in most places, but it has many differences in detail. For example, a FuncCall node in the parse tree represents something that looks syntactically like a function call. This might be transformed to either a FuncExpr or Aggref node depending on whether the referenced name turns out to be an ordinary function or an aggregate function. Also, information about the actual data types of columns and expression results is added to the query tree.

Prev | Up | Next
51.2. How Connections Are Established | Home | 51.4. The PostgreSQL Rule System


51.5. Planner/Optimizer
Prev Up Chapter 51. Overview of PostgreSQL Internals Home Next

51.5. Planner/Optimizer

51.5.1. Generating Possible Plans

The task of the planner/optimizer is to create an optimal execution plan. A given SQL query (and hence, a query tree) can be actually executed in a wide variety of different ways, each of which will produce the same set of results. If it is computationally feasible, the query optimizer will examine each of these possible execution plans, ultimately selecting the execution plan that is expected to run the fastest.

Note

In some situations, examining each possible way in which a query can be executed would take an excessive amount of time and memory. In particular, this occurs when executing queries involving large numbers of join operations. In order to determine a reasonable (not necessarily optimal) query plan in a reasonable amount of time, PostgreSQL uses a Genetic Query Optimizer (see Chapter 60) when the number of joins exceeds a threshold (see geqo_threshold).

The planner's search procedure actually works with data structures called paths, which are simply cut-down representations of plans containing only as much information as the planner needs to make its decisions. After the cheapest path is determined, a full-fledged plan tree is built to pass to the executor. This represents the desired execution plan in sufficient detail for the executor to run it. In the rest of this section we'll ignore the distinction between paths and plans.

51.5.1. Generating Possible Plans

The planner/optimizer starts by generating plans for scanning each individual relation (table) used in the query. The possible plans are determined by the available indexes on each relation. There is always the possibility of performing a sequential scan on a relation, so a sequential scan plan is always created. Assume an index is defined on a relation (for example a B-tree index) and a query contains the restriction relation.attribute OPR constant. If relation.attribute happens to match the key of the B-tree index and OPR is one of the operators listed in the index's operator class, another plan is created using the B-tree index to scan the relation. If there are further indexes present and the restrictions in the query happen to match a key of an index, further plans will be considered. Index scan plans are also generated for indexes that have a sort ordering that can match the query's ORDER BY clause (if any), or a sort ordering that might be useful for merge joining (see below).

If the query requires joining two or more relations, plans for joining relations are considered after all feasible plans have been found for scanning single relations. The three available join strategies are:

  • nested loop join: The right relation is scanned once for every row found in the left relation. This strategy is easy to implement but can be very time consuming. (However, if the right relation can be scanned with an index scan, this can be a good strategy. It is possible to use values from the current row of the left relation as keys for the index scan of the right.)

  • merge join: Each relation is sorted on the join attributes before the join starts. Then the two relations are scanned in parallel, and matching rows are combined to form join rows. This kind of join is attractive because each relation has to be scanned only once. The required sorting might be achieved either by an explicit sort step, or by scanning the relation in the proper order using an index on the join key.

  • hash join: the right relation is first scanned and loaded into a hash table, using its join attributes as hash keys. Next the left relation is scanned and the appropriate values of every row found are used as hash keys to locate the matching rows in the table.

When the query involves more than two relations, the final result must be built up by a tree of join steps, each with two inputs. The planner examines different possible join sequences to find the cheapest one.

If the query uses fewer than geqo_threshold relations, a near-exhaustive search is conducted to find the best join sequence. The planner preferentially considers joins between any two relations for which there exists a corresponding join clause in the WHERE qualification (i.e., for which a restriction like where rel1.attr1=rel2.attr2 exists). Join pairs with no join clause are considered only when there is no other choice, that is, a particular relation has no available join clauses to any other relation. All possible plans are generated for every join pair considered by the planner, and the one that is (estimated to be) the cheapest is chosen.

When geqo_threshold is exceeded, the join sequences considered are determined by heuristics, as described in Chapter 60. Otherwise the process is the same.

The finished plan tree consists of sequential or index scans of the base relations, plus nested-loop, merge, or hash join nodes as needed, plus any auxiliary steps needed, such as sort nodes or aggregate-function calculation nodes. Most of these plan node types have the additional ability to do selection (discarding rows that do not meet a specified Boolean condition) and projection (computation of a derived column set based on given column values, that is, evaluation of scalar expressions where needed). One of the responsibilities of the planner is to attach selection conditions from the WHERE clause and computation of required output expressions to the most appropriate nodes of the plan tree.

Prev | Up | Next
51.4. The PostgreSQL Rule System | Home | 51.6. Executor


51.6. Executor
Prev Up Chapter 51. Overview of PostgreSQL Internals Home Next

51.6. Executor

The executor takes the plan created by the planner/optimizer and recursively processes it to extract the required set of rows. This is essentially a demand-pull pipeline mechanism. Each time a plan node is called, it must deliver one more row, or report that it is done delivering rows.

To provide a concrete example, assume that the top node is a MergeJoin node. Before any merge can be done two rows have to be fetched (one from each subplan). So the executor recursively calls itself to process the subplans (it starts with the subplan attached to lefttree). The new top node (the top node of the left subplan) is, let's say, a Sort node and again recursion is needed to obtain an input row. The child node of the Sort might be a SeqScan node, representing actual reading of a table. Execution of this node causes the executor to fetch a row from the table and return it up to the calling node. The Sort node will repeatedly call its child to obtain all the rows to be sorted. When the input is exhausted (as indicated by the child node returning a NULL instead of a row), the Sort code performs the sort, and finally is able to return its first output row, namely the first one in sorted order. It keeps the remaining rows stored so that it can deliver them in sorted order in response to later demands.

The MergeJoin node similarly demands the first row from its right subplan. Then it compares the two rows to see if they can be joined; if so, it returns a join row to its caller. On the next call, or immediately if it cannot join the current pair of inputs, it advances to the next row of one table or the other (depending on how the comparison came out), and again checks for a match. Eventually, one subplan or the other is exhausted, and the MergeJoin node returns NULL to indicate that no more join rows can be formed.

Complex queries can involve many levels of plan nodes, but the general approach is the same: each node computes and returns its next output row each time it is called. Each node is also responsible for applying any selection or projection expressions that were assigned to it by the planner.

The executor mechanism is used to evaluate all four basic SQL query types: SELECT, INSERT, UPDATE, and DELETE. For SELECT, the top-level executor code only needs to send each row returned by the query plan tree off to the client. INSERT ... SELECT, UPDATE, and DELETE are effectively SELECTs under a special top-level plan node called ModifyTable.

INSERT ... SELECT feeds the rows up to ModifyTable for insertion. For UPDATE, the planner arranges that each computed row includes all the updated column values, plus the TID (tuple ID, or row ID) of the original target row; this data is fed up to the ModifyTable node, which uses the information to create a new updated row and mark the old row deleted. For DELETE, the only column that is actually returned by the plan is the TID, and the ModifyTable node simply uses the TID to visit each target row and mark it deleted.

A simple INSERT ... VALUES command creates a trivial plan tree consisting of a single Result node, which computes just one result row, feeding that up to ModifyTable to perform the insertion.

Prev | Up | Next
51.5. Planner/Optimizer | Home | Chapter 52. System Catalogs


Chapter 52. System Catalogs
Prev Up Part VII. Internals Home Next

Chapter 52. System Catalogs

Table of Contents

52.1. Overview
52.2. pg_aggregate
52.3. pg_am
52.4. pg_amop
52.5. pg_amproc
52.6. pg_attrdef
52.7. pg_attribute
52.8. pg_authid
52.9. pg_auth_members
52.10. pg_cast
52.11. pg_class
52.12. pg_collation
52.13. pg_constraint
52.14. pg_conversion
52.15. pg_database
52.16. pg_db_role_setting
52.17. pg_default_acl
52.18. pg_depend
52.19. pg_description
52.20. pg_enum
52.21. pg_event_trigger
52.22. pg_extension
52.23. pg_foreign_data_wrapper
52.24. pg_foreign_server
52.25. pg_foreign_table
52.26. pg_index
52.27. pg_inherits
52.28. pg_init_privs
52.29. pg_language
52.30. pg_largeobject
52.31. pg_largeobject_metadata
52.32. pg_namespace
52.33. pg_opclass
52.34. pg_operator
52.35. pg_opfamily
52.36. pg_partitioned_table
52.37. pg_policy
52.38. pg_proc
52.39. pg_publication
52.40. pg_publication_rel
52.41. pg_range
52.42. pg_replication_origin
52.43. pg_rewrite
52.44. pg_seclabel
52.45. pg_sequence
52.46. pg_shdepend
52.47. pg_shdescription
52.48. pg_shseclabel
52.49. pg_statistic
52.50. pg_statistic_ext
52.51. pg_statistic_ext_data
52.52. pg_subscription
52.53. pg_subscription_rel
52.54. pg_tablespace
52.55. pg_transform
52.56. pg_trigger
52.57. pg_ts_config
52.58. pg_ts_config_map
52.59. pg_ts_dict
52.60. pg_ts_parser
52.61. pg_ts_template
52.62. pg_type
52.63. pg_user_mapping
52.64. System Views
52.65. pg_available_extensions
52.66. pg_available_extension_versions
52.67. pg_backend_memory_contexts
52.68. pg_config
52.69. pg_cursors
52.70. pg_file_settings
52.71. pg_group
52.72. pg_hba_file_rules
52.73. pg_indexes
52.74. pg_locks
52.75. pg_matviews
52.76. pg_policies
52.77. pg_prepared_statements
52.78. pg_prepared_xacts
52.79. pg_publication_tables
52.80. pg_replication_origin_status
52.81. pg_replication_slots
52.82. pg_roles
52.83. pg_rules
52.84. pg_seclabels
52.85. pg_sequences
52.86. pg_settings
52.87. pg_shadow
52.88. pg_shmem_allocations
52.89. pg_stats
52.90. pg_stats_ext
52.91. pg_stats_ext_exprs
52.92. pg_tables
52.93. pg_timezone_abbrevs
52.94. pg_timezone_names
52.95. pg_user
52.96. pg_user_mappings
52.97. pg_views

The system catalogs are the place where a relational database management system stores schema metadata, such as information about tables and columns, and internal bookkeeping information. PostgreSQL's system catalogs are regular tables. You can drop and recreate the tables, add columns, insert and update values, and severely mess up your system that way. Normally, one should not change the system catalogs by hand, there are normally SQL commands to do that. (For example, CREATE DATABASE inserts a row into the pg_database catalog — and actually creates the database on disk.) There are some exceptions for particularly esoteric operations, but many of those have been made available as SQL commands over time, and so the need for direct manipulation of the system catalogs is ever decreasing.

Prev | Up | Next
51.6. Executor | Home | 52.1. Overview


52.1. Overview
Prev Up Chapter 52. System Catalogs Home Next

52.1. Overview

Table 52.1 lists the system catalogs. More detailed documentation of each catalog follows below.

Most system catalogs are copied from the template database during database creation and are thereafter database-specific. A few catalogs are physically shared across all databases in a cluster; these are noted in the descriptions of the individual catalogs.

Table 52.1. System Catalogs

Catalog Name | Purpose
pg_aggregate | aggregate functions
pg_am | relation access methods
pg_amop | access method operators
pg_amproc | access method support functions
pg_attrdef | column default values
pg_attribute | table columns (“attributes”)
pg_authid | authorization identifiers (roles)
pg_auth_members | authorization identifier membership relationships
pg_cast | casts (data type conversions)
pg_class | tables, indexes, sequences, views (“relations”)
pg_collation | collations (locale information)
pg_constraint | check constraints, unique constraints, primary key constraints, foreign key constraints
pg_conversion | encoding conversion information
pg_database | databases within this database cluster
pg_db_role_setting | per-role and per-database settings
pg_default_acl | default privileges for object types
pg_depend | dependencies between database objects
pg_description | descriptions or comments on database objects
pg_enum | enum label and value definitions
pg_event_trigger | event triggers
pg_extension | installed extensions
pg_foreign_data_wrapper | foreign-data wrapper definitions
pg_foreign_server | foreign server definitions
pg_foreign_table | additional foreign table information
pg_index | additional index information
pg_inherits | table inheritance hierarchy
pg_init_privs | object initial privileges
pg_language | languages for writing functions
pg_largeobject | data pages for large objects
pg_largeobject_metadata | metadata for large objects
pg_namespace | schemas
pg_opclass | access method operator classes
pg_operator | operators
pg_opfamily | access method operator families
pg_partitioned_table | information about partition key of tables
pg_policy | row-security policies
pg_proc | functions and procedures
pg_publication | publications for logical replication
pg_publication_rel | relation to publication mapping
pg_range | information about range types
pg_replication_origin | registered replication origins
pg_rewrite | query rewrite rules
pg_seclabel | security labels on database objects
pg_sequence | information about sequences
pg_shdepend | dependencies on shared objects
pg_shdescription | comments on shared objects
pg_shseclabel | security labels on shared database objects
pg_statistic | planner statistics
pg_statistic_ext | extended planner statistics (definition)
pg_statistic_ext_data | extended planner statistics (built statistics)
pg_subscription | logical replication subscriptions
pg_subscription_rel | relation state for subscriptions
pg_tablespace | tablespaces within this database cluster
pg_transform | transforms (data type to procedural language conversions)
pg_trigger | triggers
pg_ts_config | text search configurations
pg_ts_config_map | text search configurations' token mappings
pg_ts_dict | text search dictionaries
pg_ts_parser | text search parsers
pg_ts_template | text search templates
pg_type | data types
pg_user_mapping | mappings of users to foreign servers

Prev | Up | Next
Chapter 52. System Catalogs | Home | 52.2. pg_aggregate


52.2. pg_aggregate
Prev Up Chapter 52. System Catalogs Home Next

52.2. pg_aggregate

The catalog pg_aggregate stores information about aggregate functions. An aggregate function is a function that operates on a set of values (typically one column from each row that matches a query condition) and returns a single value computed from all these values. Typical aggregate functions are sum, count, and max. Each entry in pg_aggregate is an extension of an entry in pg_proc. The pg_proc entry carries the aggregate's name, input and output data types, and other information that is similar to ordinary functions.

Table 52.2. pg_aggregate Columns

Column Type

Description

aggfnoid regproc (references pg_proc.oid)

pg_proc OID of the aggregate function

aggkind char

Aggregate kind: n for “normal” aggregates, o for “ordered-set” aggregates, or h for “hypothetical-set” aggregates

aggnumdirectargs int2

Number of direct (non-aggregated) arguments of an ordered-set or hypothetical-set aggregate, counting a variadic array as one argument. If equal to pronargs, the aggregate must be variadic and the variadic array describes the aggregated arguments as well as the final direct arguments. Always zero for normal aggregates.

aggtransfn regproc (references pg_proc.oid)

Transition function

aggfinalfn regproc (references pg_proc.oid)

Final function (zero if none)

aggcombinefn regproc (references pg_proc.oid)

Combine function (zero if none)

aggserialfn regproc (references pg_proc.oid)

Serialization function (zero if none)

aggdeserialfn regproc (references pg_proc.oid)

Deserialization function (zero if none)

aggmtransfn regproc (references pg_proc.oid)

Forward transition function for moving-aggregate mode (zero if none)

aggminvtransfn regproc (references pg_proc.oid)

Inverse transition function for moving-aggregate mode (zero if none)

aggmfinalfn regproc (references pg_proc.oid)

Final function for moving-aggregate mode (zero if none)

aggfinalextra bool

True to pass extra dummy arguments to aggfinalfn

aggmfinalextra bool

True to pass extra dummy arguments to aggmfinalfn

aggfinalmodify char

Whether aggfinalfn modifies the transition state value: r if it is read-only, s if the aggtransfn cannot be applied after the aggfinalfn, or w if it writes on the value

aggmfinalmodify char

Like aggfinalmodify, but for the aggmfinalfn

aggsortop oid (references pg_operator.oid)

Associated sort operator (zero if none)

aggtranstype oid (references pg_type.oid)

Data type of the aggregate function's internal transition (state) data

aggtransspace int4

Approximate average size (in bytes) of the transition state data, or zero to use a default estimate

aggmtranstype oid (references pg_type.oid)

Data type of the aggregate function's internal transition (state) data for moving-aggregate mode (zero if none)

aggmtransspace int4

Approximate average size (in bytes) of the transition state data for moving-aggregate mode, or zero to use a default estimate

agginitval text

The initial value of the transition state. This is a text field containing the initial value in its external string representation. If this field is null, the transition state value starts out null.

aggminitval text

The initial value of the transition state for moving-aggregate mode. This is a text field containing the initial value in its external string representation. If this field is null, the transition state value starts out null.

New aggregate functions are registered with the CREATE AGGREGATE command. See Section 38.12 for more information about writing aggregate functions and the meaning of the transition functions, etc.

Prev | Up | Next
52.1. Overview | Home | 52.3. pg_am


52.3. pg_am
Prev Up Chapter 52. System Catalogs Home Next

52.3. pg_am

The catalog pg_am stores information about relation access methods. There is one row for each access method supported by the system. Currently, only tables and indexes have access methods. The requirements for table and index access methods are discussed in detail in Chapter 61 and Chapter 62 respectively.

Table 52.3. pg_am Columns

Column Type

Description

oid oid

Row identifier

amname name

Name of the access method

amhandler regproc (references pg_proc.oid)

OID of a handler function that is responsible for supplying information about the access method

amtype char

t = table (including materialized views), i = index.

Note

Before PostgreSQL 9.6, pg_am contained many additional columns representing properties of index access methods. That data is now only directly visible at the C code level. However, pg_index_column_has_property() and related functions have been added to allow SQL queries to inspect index access method properties; see Table 9.70.

Prev | Up | Next
52.2. pg_aggregate | Home | 52.4. pg_amop


52.4. pg_amop
Prev Up Chapter 52. System Catalogs Home Next

52.4. pg_amop

The catalog pg_amop stores information about operators associated with access method operator families. There is one row for each operator that is a member of an operator family. A family member can be either a search operator or an ordering operator. An operator can appear in more than one family, but cannot appear in more than one search position nor more than one ordering position within a family. (It is allowed, though unlikely, for an operator to be used for both search and ordering purposes.)

Table 52.4. pg_amop Columns

Column Type

Description

oid oid

Row identifier

amopfamily oid (references pg_opfamily.oid)

The operator family this entry is for

amoplefttype oid (references pg_type.oid)

Left-hand input data type of operator

amoprighttype oid (references pg_type.oid)

Right-hand input data type of operator

amopstrategy int2

Operator strategy number

amoppurpose char

Operator purpose, either s for search or o for ordering

amopopr oid (references pg_operator.oid)

OID of the operator

amopmethod oid (references pg_am.oid)

Index access method operator family is for

amopsortfamily oid (references pg_opfamily.oid)

The B-tree operator family this entry sorts according to, if an ordering operator; zero if a search operator

A “search” operator entry indicates that an index of this operator family can be searched to find all rows satisfying WHERE indexed_column operator constant. Obviously, such an operator must return boolean, and its left-hand input type must match the index's column data type.

An “ordering” operator entry indicates that an index of this operator family can be scanned to return rows in the order represented by ORDER BY indexed_column operator constant. Such an operator could return any sortable data type, though again its left-hand input type must match the index's column data type. The exact semantics of the ORDER BY are specified by the amopsortfamily column, which must reference a B-tree operator family for the operator's result type.

Note

At present, it's assumed that the sort order for an ordering operator is the default for the referenced operator family, i.e., ASC NULLS LAST. This might someday be relaxed by adding additional columns to specify sort options explicitly.

An entry's amopmethod must match the opfmethod of its containing operator family (including amopmethod here is an intentional denormalization of the catalog structure for performance reasons). Also, amoplefttype and amoprighttype must match the oprleft and oprright fields of the referenced pg_operator entry.

Prev | Up | Next
52.3. pg_am | Home | 52.5. pg_amproc


52.5. pg_amproc
Prev Up Chapter 52. System Catalogs Home Next

52.5. pg_amproc

The catalog pg_amproc stores information about support functions associated with access method operator families. There is one row for each support function belonging to an operator family.

Table 52.5. pg_amproc Columns

Column Type

Description

oid oid

Row identifier

amprocfamily oid (references pg_opfamily.oid)

The operator family this entry is for

amproclefttype oid (references pg_type.oid)

Left-hand input data type of associated operator

amprocrighttype oid (references pg_type.oid)

Right-hand input data type of associated operator

amprocnum int2

Support function number

amproc regproc (references pg_proc.oid)

OID of the function

The usual interpretation of the amproclefttype and amprocrighttype fields is that they identify the left and right input types of the operator(s) that a particular support function supports. For some access methods these match the input data type(s) of the support function itself, for others not. There is a notion of “default” support functions for an index, which are those with amproclefttype and amprocrighttype both equal to the index operator class's opcintype.

Prev | Up | Next
52.4. pg_amop | Home | 52.6. pg_attrdef


52.6. pg_attrdef
Prev Up Chapter 52. System Catalogs Home Next

52.6. pg_attrdef

The catalog pg_attrdef stores column default values. The main information about columns is stored in pg_attribute. Only columns for which a default value has been explicitly set will have an entry here.

Table 52.6. pg_attrdef Columns

Column Type

Description

oid oid

Row identifier

adrelid oid (references pg_class.oid)

The table this column belongs to

adnum int2 (references pg_attribute.attnum)

The number of the column

adbin pg_node_tree

The column default value, in nodeToString() representation. Use pg_get_expr(adbin, adrelid) to convert it to an SQL expression.

Prev | Up | Next
52.5. pg_amproc | Home | 52.7. pg_attribute


52.7. pg_attribute
Prev Up Chapter 52. System Catalogs Home Next

52.7. pg_attribute

The catalog pg_attribute stores information about table columns. There will be exactly one pg_attribute row for every column in every table in the database. (There will also be attribute entries for indexes, and indeed all objects that have pg_class entries.)

The term attribute is equivalent to column and is used for historical reasons.

Table 52.7. pg_attribute Columns

Column Type

Description

attrelid oid (references pg_class.oid)

The table this column belongs to

attname name

The column name

atttypid oid (references pg_type.oid)

The data type of this column (zero for a dropped column)

attstattarget int4

attstattarget controls the level of detail of statistics accumulated for this column by ANALYZE. A zero value indicates that no statistics should be collected. A negative value says to use the system default statistics target. The exact meaning of positive values is data type-dependent. For scalar data types, attstattarget is both the target number of “most common values” to collect, and the target number of histogram bins to create.

attlen int2

A copy of pg_type.typlen of this column's type

attnum int2

The number of the column. Ordinary columns are numbered from 1 up. System columns, such as ctid, have (arbitrary) negative numbers.

attndims int4

Number of dimensions, if the column is an array type; otherwise 0. (Presently, the number of dimensions of an array is not enforced, so any nonzero value effectively means “it's an array”.)

attcacheoff int4

Always -1 in storage, but when loaded into a row descriptor in memory this might be updated to cache the offset of the attribute within the row

atttypmod int4

atttypmod records type-specific data supplied at table creation time (for example, the maximum length of a varchar column). It is passed to type-specific input functions and length coercion functions. The value will generally be -1 for types that do not need atttypmod.

attbyval bool

A copy of pg_type.typbyval of this column's type

attalign char

A copy of pg_type.typalign of this column's type

attstorage char

Normally a copy of pg_type.typstorage of this column's type. For TOAST-able data types, this can be altered after column creation to control storage policy.

attcompression char

The current compression method of the column. Typically this is '\0' to specify use of the current default setting (see default_toast_compression). Otherwise, 'p' selects pglz compression, while 'l' selects LZ4 compression. However, this field is ignored whenever attstorage does not allow compression.

attnotnull bool

This represents a not-null constraint.

atthasdef bool

This column has a default expression or generation expression, in which case there will be a corresponding entry in the pg_attrdef catalog that actually defines the expression. (Check attgenerated to determine whether this is a default or a generation expression.)

atthasmissing bool

This column has a value which is used where the column is entirely missing from the row, as happens when a column is added with a non-volatile DEFAULT value after the row is created. The actual value used is stored in the attmissingval column.

attidentity char

If a zero byte (''), then not an identity column. Otherwise, a = generated always, d = generated by default.

attgenerated char

If a zero byte (''), then not a generated column. Otherwise, s = stored. (Other values might be added in the future.)

attisdropped bool

This column has been dropped and is no longer valid. A dropped column is still physically present in the table, but is ignored by the parser and so cannot be accessed via SQL.

attislocal bool

This column is defined locally in the relation. Note that a column can be locally defined and inherited simultaneously.

attinhcount int4

The number of direct ancestors this column has. A column with a nonzero number of ancestors cannot be dropped nor renamed.

attcollation oid (references pg_collation.oid)

The defined collation of the column, or zero if the column is not of a collatable data type

attacl aclitem[]

Column-level access privileges, if any have been granted specifically on this column

attoptions text[]

Attribute-level options, as “keyword=value” strings

attfdwoptions text[]

Attribute-level foreign data wrapper options, as “keyword=value” strings

attmissingval anyarray

This column has a one element array containing the value used when the column is entirely missing from the row, as happens when the column is added with a non-volatile DEFAULT value after the row is created. The value is only used when atthasmissing is true. If there is no value the column is null.

In a dropped column's pg_attribute entry, atttypid is reset to zero, but attlen and the other fields copied from pg_type are still valid. This arrangement is needed to cope with the situation where the dropped column's data type was later dropped, and so there is no pg_type row anymore. attlen and the other fields can be used to interpret the contents of a row of the table.

Prev | Up | Next
52.6. pg_attrdef | Home | 52.8. pg_authid


52.8. pg_authid
Prev Up Chapter 52. System Catalogs Home Next

52.8. pg_authid

The catalog pg_authid contains information about database authorization identifiers (roles). A role subsumes the concepts of “users” and “groups”. A user is essentially just a role with the rolcanlogin flag set. Any role (with or without rolcanlogin) can have other roles as members; see pg_auth_members.

Since this catalog contains passwords, it must not be publicly readable. pg_roles is a publicly readable view on pg_authid that blanks out the password field.

Chapter 22 contains detailed information about user and privilege management.

Because user identities are cluster-wide, pg_authid is shared across all databases of a cluster: there is only one copy of pg_authid per cluster, not one per database.

Table 52.8. pg_authid Columns

Column Type

Description

oid oid

Row identifier

rolname name

Role name

rolsuper bool

Role has superuser privileges

rolinherit bool

Role automatically inherits privileges of roles it is a member of

rolcreaterole bool

Role can create more roles

rolcreatedb bool

Role can create databases

rolcanlogin bool

Role can log in. That is, this role can be given as the initial session authorization identifier.

rolreplication bool

Role is a replication role. A replication role can initiate replication connections and create and drop replication slots.

rolbypassrls bool

Role bypasses every row-level security policy, see Section 5.8 for more information.

rolconnlimit int4

For roles that can log in, this sets maximum number of concurrent connections this role can make. -1 means no limit.

rolpassword text

Password (possibly encrypted); null if none. The format depends on the form of encryption used.

rolvaliduntil timestamptz

Password expiry time (only used for password authentication); null if no expiration

For an MD5 encrypted password, rolpassword column will begin with the string md5 followed by a 32-character hexadecimal MD5 hash. The MD5 hash will be of the user's password concatenated to their user name. For example, if user joe has password xyzzy, PostgreSQL will store the md5 hash of xyzzyjoe.

If the password is encrypted with SCRAM-SHA-256, it has the format:

SCRAM-SHA-256$<iteration count>:<salt>$<StoredKey>:<ServerKey>

where salt, StoredKey and ServerKey are in Base64 encoded format. This format is the same as that specified by RFC 5803.

A password that does not follow either of those formats is assumed to be unencrypted.

Prev | Up | Next
52.7. pg_attribute | Home | 52.9. pg_auth_members


52.9. pg_auth_members
Prev Up Chapter 52. System Catalogs Home Next

52.9. pg_auth_members

The catalog pg_auth_members shows the membership relations between roles. Any non-circular set of relationships is allowed.

Because user identities are cluster-wide, pg_auth_members is shared across all databases of a cluster: there is only one copy of pg_auth_members per cluster, not one per database.

Table 52.9. pg_auth_members Columns

Column Type

Description

roleid oid (references pg_authid.oid)

ID of a role that has a member

member oid (references pg_authid.oid)

ID of a role that is a member of roleid

grantor oid (references pg_authid.oid)

ID of the role that granted this membership

admin_option bool

True if member can grant membership in roleid to others

Prev | Up | Next
52.8. pg_authid | Home | 52.10. pg_cast


52.10. pg_cast
Prev Up Chapter 52. System Catalogs Home Next

52.10. pg_cast

The catalog pg_cast stores data type conversion paths, both built-in and user-defined.

It should be noted that pg_cast does not represent every type conversion that the system knows how to perform; only those that cannot be deduced from some generic rule. For example, casting between a domain and its base type is not explicitly represented in pg_cast. Another important exception is that “automatic I/O conversion casts”, those performed using a data type's own I/O functions to convert to or from text or other string types, are not explicitly represented in pg_cast.

Table 52.10. pg_cast Columns

Column Type

Description

oid oid

Row identifier

castsource oid (references pg_type.oid)

OID of the source data type

casttarget oid (references pg_type.oid)

OID of the target data type

castfunc oid (references pg_proc.oid)

The OID of the function to use to perform this cast. Zero is stored if the cast method doesn't require a function.

castcontext char

Indicates what contexts the cast can be invoked in. e means only as an explicit cast (using CAST or :: syntax). a means implicitly in assignment to a target column, as well as explicitly. i means implicitly in expressions, as well as the other cases.

castmethod char

Indicates how the cast is performed. f means that the function specified in the castfunc field is used. i means that the input/output functions are used. b means that the types are binary-coercible, thus no conversion is required.

The cast functions listed in pg_cast must always take the cast source type as their first argument type, and return the cast destination type as their result type. A cast function can have up to three arguments. The second argument, if present, must be type integer; it receives the type modifier associated with the destination type, or -1 if there is none. The third argument, if present, must be type boolean; it receives true if the cast is an explicit cast, false otherwise.

It is legitimate to create a pg_cast entry in which the source and target types are the same, if the associated function takes more than one argument. Such entries represent “length coercion functions” that coerce values of the type to be legal for a particular type modifier value.

When a pg_cast entry has different source and target types and a function that takes more than one argument, it represents converting from one type to another and applying a length coercion in a single step. When no such entry is available, coercion to a type that uses a type modifier involves two steps, one to convert between data types and a second to apply the modifier.

Prev | Up | Next
52.9. pg_auth_members | Home | 52.11. pg_class


52.11. pg_class
Prev Up Chapter 52. System Catalogs Home Next

52.11. pg_class

The catalog pg_class catalogs tables and most everything else that has columns or is otherwise similar to a table. This includes indexes (but see also pg_index), sequences (but see also pg_sequence), views, materialized views, composite types, and TOAST tables; see relkind. Below, when we mean all of these kinds of objects we speak of “relations”. Not all columns are meaningful for all relation types.

Table 52.11. pg_class Columns

Column Type

Description

oid oid

Row identifier

relname name

Name of the table, index, view, etc.

relnamespace oid (references pg_namespace.oid)

The OID of the namespace that contains this relation

reltype oid (references pg_type.oid)

The OID of the data type that corresponds to this table's row type, if any; zero for indexes, sequences, and toast tables, which have no pg_type entry

reloftype oid (references pg_type.oid)

For typed tables, the OID of the underlying composite type; zero for all other relations

relowner oid (references pg_authid.oid)

Owner of the relation

relam oid (references pg_am.oid)

If this is a table or an index, the access method used (heap, B-tree, hash, etc.); otherwise zero (zero occurs for sequences, as well as relations without storage, such as views)

relfilenode oid

Name of the on-disk file of this relation; zero means this is a “mapped” relation whose disk file name is determined by low-level state

reltablespace oid (references pg_tablespace.oid)

The tablespace in which this relation is stored. If zero, the database's default tablespace is implied. (Not meaningful if the relation has no on-disk file.)

relpages int4

Size of the on-disk representation of this table in pages (of size BLCKSZ). This is only an estimate used by the planner. It is updated by VACUUM, ANALYZE, and a few DDL commands such as CREATE INDEX.

reltuples float4

Number of live rows in the table. This is only an estimate used by the planner. It is updated by VACUUM, ANALYZE, and a few DDL commands such as CREATE INDEX. If the table has never yet been vacuumed or analyzed, reltuples contains -1 indicating that the row count is unknown.

relallvisible int4

Number of pages that are marked all-visible in the table's visibility map. This is only an estimate used by the planner. It is updated by VACUUM, ANALYZE, and a few DDL commands such as CREATE INDEX.

reltoastrelid oid (references pg_class.oid)

OID of the TOAST table associated with this table, zero if none. The TOAST table stores large attributes “out of line” in a secondary table.

relhasindex bool

True if this is a table and it has (or recently had) any indexes

relisshared bool

True if this table is shared across all databases in the cluster. Only certain system catalogs (such as pg_database) are shared.

relpersistence char

p = permanent table, u = unlogged table, t = temporary table

relkind char

r = ordinary table, i = index, S = sequence, t = TOAST table, v = view, m = materialized view, c = composite type, f = foreign table, p = partitioned table, I = partitioned index

relnatts int2

Number of user columns in the relation (system columns not counted). There must be this many corresponding entries in pg_attribute. See also pg_attribute.attnum.

relchecks int2

Number of CHECK constraints on the table; see pg_constraint catalog

relhasrules bool

True if table has (or once had) rules; see pg_rewrite catalog

relhastriggers bool

True if table has (or once had) triggers; see pg_trigger catalog

relhassubclass bool

True if table or index has (or once had) any inheritance children

relrowsecurity bool

True if table has row-level security enabled; see pg_policy catalog

relforcerowsecurity bool

True if row-level security (when enabled) will also apply to table owner; see pg_policy catalog

relispopulated bool

True if relation is populated (this is true for all relations other than some materialized views)

relreplident char

Columns used to form “replica identity” for rows: d = default (primary key, if any), n = nothing, f = all columns, i = index with indisreplident set (same as nothing if the index used has been dropped)

relispartition bool

True if table or index is a partition

relrewrite oid (references pg_class.oid)

For new relations being written during a DDL operation that requires a table rewrite, this contains the OID of the original relation; otherwise zero. That state is only visible internally; this field should never contain anything other than zero for a user-visible relation.

relfrozenxid xid

All transaction IDs before this one have been replaced with a permanent (“frozen”) transaction ID in this table. This is used to track whether the table needs to be vacuumed in order to prevent transaction ID wraparound or to allow pg_xact to be shrunk. Zero (InvalidTransactionId) if the relation is not a table.

relminmxid xid

All multixact IDs before this one have been replaced by a transaction ID in this table. This is used to track whether the table needs to be vacuumed in order to prevent multixact ID wraparound or to allow pg_multixact to be shrunk. Zero (InvalidMultiXactId) if the relation is not a table.

relacl aclitem[]

Access privileges; see Section 5.7 for details

reloptions text[]

Access-method-specific options, as “keyword=value” strings

relpartbound pg_node_tree

If table is a partition (see relispartition), internal representation of the partition bound

Several of the Boolean flags in pg_class are maintained lazily: they are guaranteed to be true if that's the correct state, but may not be reset to false immediately when the condition is no longer true. For example, relhasindex is set by CREATE INDEX, but it is never cleared by DROP INDEX. Instead, VACUUM clears relhasindex if it finds the table has no indexes. This arrangement avoids race conditions and improves concurrency.

Prev | Up | Next
52.10. pg_cast | Home | 52.12. pg_collation


52.12. pg_collation
Prev Up Chapter 52. System Catalogs Home Next

52.12. pg_collation

The catalog pg_collation describes the available collations, which are essentially mappings from an SQL name to operating system locale categories. See Section 24.2 for more information.

Table 52.12. pg_collation Columns

Column Type

Description

oid oid

Row identifier

collname name

Collation name (unique per namespace and encoding)

collnamespace oid (references pg_namespace.oid)

The OID of the namespace that contains this collation

collowner oid (references pg_authid.oid)

Owner of the collation

collprovider char

Provider of the collation: d = database default, c = libc, i = icu

collisdeterministic bool

Is the collation deterministic?

collencoding int4

Encoding in which the collation is applicable, or -1 if it works for any encoding

collcollate name

LC_COLLATE for this collation object

collctype name

LC_CTYPE for this collation object

collversion text

Provider-specific version of the collation. This is recorded when the collation is created and then checked when it is used, to detect changes in the collation definition that could lead to data corruption.

Note that the unique key on this catalog is (collname, collencoding, collnamespace) not just (collname, collnamespace). PostgreSQL generally ignores all collations that do not have collencoding equal to either the current database's encoding or -1, and creation of new entries with the same name as an entry with collencoding = -1 is forbidden. Therefore it is sufficient to use a qualified SQL name (schema.name) to identify a collation, even though this is not unique according to the catalog definition. The reason for defining the catalog this way is that initdb fills it in at cluster initialization time with entries for all locales available on the system, so it must be able to hold entries for all encodings that might ever be used in the cluster.

In the template0 database, it could be useful to create collations whose encoding does not match the database encoding, since they could match the encodings of databases later cloned from template0. This would currently have to be done manually.

Prev | Up | Next
52.11. pg_class | Home | 52.13. pg_constraint


52.13. pg_constraint
Prev Up Chapter 52. System Catalogs Home Next

52.13. pg_constraint

The catalog pg_constraint stores check, primary key, unique, foreign key, and exclusion constraints on tables. (Column constraints are not treated specially. Every column constraint is equivalent to some table constraint.) Not-null constraints are represented in the pg_attribute catalog, not here.

User-defined constraint triggers (created with CREATE CONSTRAINT TRIGGER) also give rise to an entry in this table.

Check constraints on domains are stored here, too.

Table 52.13. pg_constraint Columns

Column Type

Description

oid oid

Row identifier

conname name

Constraint name (not necessarily unique!)

connamespace oid (references pg_namespace.oid)

The OID of the namespace that contains this constraint

contype char

c = check constraint, f = foreign key constraint, p = primary key constraint, u = unique constraint, t = constraint trigger, x = exclusion constraint

condeferrable bool

Is the constraint deferrable?

condeferred bool

Is the constraint deferred by default?

convalidated bool

Has the constraint been validated? Currently, can be false only for foreign keys and CHECK constraints

conrelid oid (references pg_class.oid)

The table this constraint is on; zero if not a table constraint

contypid oid (references pg_type.oid)

The domain this constraint is on; zero if not a domain constraint

conindid oid (references pg_class.oid)

The index supporting this constraint, if it's a unique, primary key, foreign key, or exclusion constraint; else zero

conparentid oid (references pg_constraint.oid)

The corresponding constraint of the parent partitioned table, if this is a constraint on a partition; else zero

confrelid oid (references pg_class.oid)

If a foreign key, the referenced table; else zero

confupdtype char

Foreign key update action code: a = no action, r = restrict, c = cascade, n = set null, d = set default

confdeltype char

Foreign key deletion action code: a = no action, r = restrict, c = cascade, n = set null, d = set default

confmatchtype char

Foreign key match type: f = full, p = partial, s = simple

conislocal bool

This constraint is defined locally for the relation. Note that a constraint can be locally defined and inherited simultaneously.

coninhcount int4

The number of direct inheritance ancestors this constraint has. A constraint with a nonzero number of ancestors cannot be dropped nor renamed.

connoinherit bool

This constraint is defined locally for the relation. It is a non-inheritable constraint.

conkey int2[] (references pg_attribute.attnum)

If a table constraint (including foreign keys, but not constraint triggers), list of the constrained columns

confkey int2[] (references pg_attribute.attnum)

If a foreign key, list of the referenced columns

conpfeqop oid[] (references pg_operator.oid)

If a foreign key, list of the equality operators for PK = FK comparisons

conppeqop oid[] (references pg_operator.oid)

If a foreign key, list of the equality operators for PK = PK comparisons

conffeqop oid[] (references pg_operator.oid)

If a foreign key, list of the equality operators for FK = FK comparisons

conexclop oid[] (references pg_operator.oid)

If an exclusion constraint, list of the per-column exclusion operators

conbin pg_node_tree

If a check constraint, an internal representation of the expression. (It's recommended to use pg_get_constraintdef() to extract the definition of a check constraint.)

In the case of an exclusion constraint, conkey is only useful for constraint elements that are simple column references. For other cases, a zero appears in conkey and the associated index must be consulted to discover the expression that is constrained. (conkey thus has the same contents as pg_index.indkey for the index.)

Note

pg_class.relchecks needs to agree with the number of check-constraint entries found in this table for each relation.

Prev | Up | Next
52.12. pg_collation | Home | 52.14. pg_conversion


52.14. pg_conversion
Prev Up Chapter 52. System Catalogs Home Next

52.14. pg_conversion

The catalog pg_conversion describes encoding conversion functions. See CREATE CONVERSION for more information.

Table 52.14. pg_conversion Columns

Column Type

Description

oid oid

Row identifier

conname name

Conversion name (unique within a namespace)

connamespace oid (references pg_namespace.oid)

The OID of the namespace that contains this conversion

conowner oid (references pg_authid.oid)

Owner of the conversion

conforencoding int4

Source encoding ID

contoencoding int4

Destination encoding ID

conproc regproc (references pg_proc.oid)

Conversion function

condefault bool

True if this is the default conversion

Prev | Up | Next
52.13. pg_constraint | Home | 52.15. pg_database


52.15. pg_database
Prev Up Chapter 52. System Catalogs Home Next

52.15. pg_database

The catalog pg_database stores information about the available databases. Databases are created with the CREATE DATABASE command. Consult Chapter 23 for details about the meaning of some of the parameters.

Unlike most system catalogs, pg_database is shared across all databases of a cluster: there is only one copy of pg_database per cluster, not one per database.

Table 52.15. pg_database Columns

Column Type

Description

oid oid

Row identifier

datname name

Database name

datdba oid (references pg_authid.oid)

Owner of the database, usually the user who created it

encoding int4

Character encoding for this database (pg_encoding_to_char() can translate this number to the encoding name)

datcollate name

LC_COLLATE for this database

datctype name

LC_CTYPE for this database

datistemplate bool

If true, then this database can be cloned by any user with CREATEDB privileges; if false, then only superusers or the owner of the database can clone it.

datallowconn bool

If false then no one can connect to this database. This is used to protect the template0 database from being altered.

datconnlimit int4

Sets maximum number of concurrent connections that can be made to this database. -1 means no limit.

datlastsysoid oid

Last system OID in the database; useful particularly to pg_dump

datfrozenxid xid

All transaction IDs before this one have been replaced with a permanent (“frozen”) transaction ID in this database. This is used to track whether the database needs to be vacuumed in order to prevent transaction ID wraparound or to allow pg_xact to be shrunk. It is the minimum of the per-table pg_class.relfrozenxid values.

datminmxid xid

All multixact IDs before this one have been replaced with a transaction ID in this database. This is used to track whether the database needs to be vacuumed in order to prevent multixact ID wraparound or to allow pg_multixact to be shrunk. It is the minimum of the per-table pg_class.relminmxid values.

dattablespace oid (references pg_tablespace.oid)

The default tablespace for the database. Within this database, all tables for which pg_class.reltablespace is zero will be stored in this tablespace; in particular, all the non-shared system catalogs will be there.

datacl aclitem[]

Access privileges; see Section 5.7 for details

Prev | Up | Next
52.14. pg_conversion | Home | 52.16. pg_db_role_setting


52.16. pg_db_role_setting
Prev Up Chapter 52. System Catalogs Home Next

52.16. pg_db_role_setting

The catalog pg_db_role_setting records the default values that have been set for run-time configuration variables, for each role and database combination.

Unlike most system catalogs, pg_db_role_setting is shared across all databases of a cluster: there is only one copy of pg_db_role_setting per cluster, not one per database.

Table 52.16. pg_db_role_setting Columns

Column Type

Description

setdatabase oid (references pg_database.oid)

The OID of the database the setting is applicable to, or zero if not database-specific

setrole oid (references pg_authid.oid)

The OID of the role the setting is applicable to, or zero if not role-specific

setconfig text[]

Defaults for run-time configuration variables

Prev | Up | Next
52.15. pg_database | Home | 52.17. pg_default_acl


52.17. pg_default_acl
Prev Up Chapter 52. System Catalogs Home Next

52.17. pg_default_acl

The catalog pg_default_acl stores initial privileges to be assigned to newly created objects.

Table 52.17. pg_default_acl Columns

Column Type

Description

oid oid

Row identifier

defaclrole oid (references pg_authid.oid)

The OID of the role associated with this entry

defaclnamespace oid (references pg_namespace.oid)

The OID of the namespace associated with this entry, or zero if none

defaclobjtype char

Type of object this entry is for: r = relation (table, view), S = sequence, f = function, T = type, n = schema

defaclacl aclitem[]

Access privileges that this type of object should have on creation

A pg_default_acl entry shows the initial privileges to be assigned to an object belonging to the indicated user. There are currently two types of entry: “global” entries with defaclnamespace = zero, and “per-schema” entries that reference a particular schema. If a global entry is present then it overrides the normal hard-wired default privileges for the object type. A per-schema entry, if present, represents privileges to be added to the global or hard-wired default privileges.

Note that when an ACL entry in another catalog is null, it is taken to represent the hard-wired default privileges for its object, not whatever might be in pg_default_acl at the moment. pg_default_acl is only consulted during object creation.

Prev | Up | Next
52.16. pg_db_role_setting | Home | 52.18. pg_depend


52.18. pg_depend
Prev Up Chapter 52. System Catalogs Home Next

52.18. pg_depend

The catalog pg_depend records the dependency relationships between database objects. This information allows DROP commands to find which other objects must be dropped by DROP CASCADE or prevent dropping in the DROP RESTRICT case.

See also pg_shdepend, which performs a similar function for dependencies involving objects that are shared across a database cluster.

Table 52.18. pg_depend Columns

Column Type

Description

classid oid (references pg_class.oid)

The OID of the system catalog the dependent object is in, or zero for a DEPENDENCY_PIN entry

objid oid (references any OID column)

The OID of the specific dependent object, or zero for a DEPENDENCY_PIN entry

objsubid int4

For a table column, this is the column number (the objid and classid refer to the table itself). For all other object types, this column is zero.

refclassid oid (references pg_class.oid)

The OID of the system catalog the referenced object is in

refobjid oid (references any OID column)

The OID of the specific referenced object

refobjsubid int4

For a table column, this is the column number (the refobjid and refclassid refer to the table itself). For all other object types, this column is zero.

deptype char

A code defining the specific semantics of this dependency relationship; see text

In all cases, a pg_depend entry indicates that the referenced object cannot be dropped without also dropping the dependent object. However, there are several subflavors identified by deptype:

DEPENDENCY_NORMAL (n)

A normal relationship between separately-created objects. The dependent object can be dropped without affecting the referenced object. The referenced object can only be dropped by specifying CASCADE, in which case the dependent object is dropped, too. Example: a table column has a normal dependency on its data type.

DEPENDENCY_AUTO (a)

The dependent object can be dropped separately from the referenced object, and should be automatically dropped (regardless of RESTRICT or CASCADE mode) if the referenced object is dropped. Example: a named constraint on a table is made auto-dependent on the table, so that it will go away if the table is dropped.

DEPENDENCY_INTERNAL (i)

The dependent object was created as part of creation of the referenced object, and is really just a part of its internal implementation. A direct DROP of the dependent object will be disallowed outright (we'll tell the user to issue a DROP against the referenced object, instead). A DROP of the referenced object will result in automatically dropping the dependent object whether CASCADE is specified or not. If the dependent object has to be dropped due to a dependency on some other object being removed, its drop is converted to a drop of the referenced object, so that NORMAL and AUTO dependencies of the dependent object behave much like they were dependencies of the referenced object. Example: a view's ON SELECT rule is made internally dependent on the view, preventing it from being dropped while the view remains. Dependencies of the rule (such as tables it refers to) act as if they were dependencies of the view.

DEPENDENCY_PARTITION_PRI (P)
DEPENDENCY_PARTITION_SEC (S)

The dependent object was created as part of creation of the referenced object, and is really just a part of its internal implementation; however, unlike INTERNAL, there is more than one such referenced object. The dependent object must not be dropped unless at least one of these referenced objects is dropped; if any one is, the dependent object should be dropped whether or not CASCADE is specified. Also unlike INTERNAL, a drop of some other object that the dependent object depends on does not result in automatic deletion of any partition-referenced object. Hence, if the drop does not cascade to at least one of these objects via some other path, it will be refused. (In most cases, the dependent object shares all its non-partition dependencies with at least one partition-referenced object, so that this restriction does not result in blocking any cascaded delete.) Primary and secondary partition dependencies behave identically except that the primary dependency is preferred for use in error messages; hence, a partition-dependent object should have one primary partition dependency and one or more secondary partition dependencies. Note that partition dependencies are made in addition to, not instead of, any dependencies the object would normally have. This simplifies ATTACH/DETACH PARTITION operations: the partition dependencies need only be added or removed. Example: a child partitioned index is made partition-dependent on both the partition table it is on and the parent partitioned index, so that it goes away if either of those is dropped, but not otherwise. The dependency on the parent index is primary, so that if the user tries to drop the child partitioned index, the error message will suggest dropping the parent index instead (not the table).

DEPENDENCY_EXTENSION (e)

The dependent object is a member of the extension that is the referenced object (see pg_extension). The dependent object can be dropped only via DROP EXTENSION on the referenced object. Functionally this dependency type acts the same as an INTERNAL dependency, but it's kept separate for clarity and to simplify pg_dump.

DEPENDENCY_AUTO_EXTENSION (x)

The dependent object is not a member of the extension that is the referenced object (and so it should not be ignored by pg_dump), but it cannot function without the extension and should be auto-dropped if the extension is. The dependent object may be dropped on its own as well. Functionally this dependency type acts the same as an AUTO dependency, but it's kept separate for clarity and to simplify pg_dump.

DEPENDENCY_PIN (p)

There is no dependent object; this type of entry is a signal that the system itself depends on the referenced object, and so that object must never be deleted. Entries of this type are created only by initdb. The columns for the dependent object contain zeroes.

Other dependency flavors might be needed in future.

Note that it's quite possible for two objects to be linked by more than one pg_depend entry. For example, a child partitioned index would have both a partition-type dependency on its associated partition table, and an auto dependency on each column of that table that it indexes. This sort of situation expresses the union of multiple dependency semantics. A dependent object can be dropped without CASCADE if any of its dependencies satisfies its condition for automatic dropping. Conversely, all the dependencies' restrictions about which objects must be dropped together must be satisfied.

Prev | Up | Next
52.17. pg_default_acl | Home | 52.19. pg_description


52.19. pg_description
Prev Up Chapter 52. System Catalogs Home Next

52.19. pg_description

The catalog pg_description stores optional descriptions (comments) for each database object. Descriptions can be manipulated with the COMMENT command and viewed with psql's \d commands. Descriptions of many built-in system objects are provided in the initial contents of pg_description.

See also pg_shdescription, which performs a similar function for descriptions involving objects that are shared across a database cluster.

Table 52.19. pg_description Columns

Column Type

Description

objoid oid (references any OID column)

The OID of the object this description pertains to

classoid oid (references pg_class.oid)

The OID of the system catalog this object appears in

objsubid int4

For a comment on a table column, this is the column number (the objoid and classoid refer to the table itself). For all other object types, this column is zero.

description text

Arbitrary text that serves as the description of this object

Prev | Up | Next
52.18. pg_depend | Home | 52.20. pg_enum


52.20. pg_enum
Prev Up Chapter 52. System Catalogs Home Next

52.20. pg_enum

The pg_enum catalog contains entries showing the values and labels for each enum type. The internal representation of a given enum value is actually the OID of its associated row in pg_enum.

Table 52.20. pg_enum Columns

Column Type

Description

oid oid

Row identifier

enumtypid oid (references pg_type.oid)

The OID of the pg_type entry owning this enum value

enumsortorder float4

The sort position of this enum value within its enum type

enumlabel name

The textual label for this enum value

The OIDs for pg_enum rows follow a special rule: even-numbered OIDs are guaranteed to be ordered in the same way as the sort ordering of their enum type. That is, if two even OIDs belong to the same enum type, the smaller OID must have the smaller enumsortorder value. Odd-numbered OID values need bear no relationship to the sort order. This rule allows the enum comparison routines to avoid catalog lookups in many common cases. The routines that create and alter enum types attempt to assign even OIDs to enum values whenever possible.

When an enum type is created, its members are assigned sort-order positions 1..n. But members added later might be given negative or fractional values of enumsortorder. The only requirement on these values is that they be correctly ordered and unique within each enum type.

Prev | Up | Next
52.19. pg_description | Home | 52.21. pg_event_trigger


52.21. pg_event_trigger
Prev Up Chapter 52. System Catalogs Home Next

52.21. pg_event_trigger

The catalog pg_event_trigger stores event triggers. See Chapter 40 for more information.

Table 52.21. pg_event_trigger Columns

Column Type

Description

oid oid

Row identifier

evtname name

Trigger name (must be unique)

evtevent name

Identifies the event for which this trigger fires

evtowner oid (references pg_authid.oid)

Owner of the event trigger

evtfoid oid (references pg_proc.oid)

The function to be called

evtenabled char

Controls in which session_replication_role modes the event trigger fires. O = trigger fires in “origin” and “local” modes, D = trigger is disabled, R = trigger fires in “replica” mode, A = trigger fires always.

evttags text[]

Command tags for which this trigger will fire. If NULL, the firing of this trigger is not restricted on the basis of the command tag.

Prev | Up | Next
52.20. pg_enum | Home | 52.22. pg_extension


52.22. pg_extension
Prev Up Chapter 52. System Catalogs Home Next

52.22. pg_extension

The catalog pg_extension stores information about the installed extensions. See Section 38.17 for details about extensions.

Table 52.22. pg_extension Columns

Column Type

Description

oid oid

Row identifier

extname name

Name of the extension

extowner oid (references pg_authid.oid)

Owner of the extension

extnamespace oid (references pg_namespace.oid)

Schema containing the extension's exported objects

extrelocatable bool

True if extension can be relocated to another schema

extversion text

Version name for the extension

extconfig oid[] (references pg_class.oid)

Array of regclass OIDs for the extension's configuration table(s), or NULL if none

extcondition text[]

Array of WHERE-clause filter conditions for the extension's configuration table(s), or NULL if none

Note that unlike most catalogs with a “namespace” column, extnamespace is not meant to imply that the extension belongs to that schema. Extension names are never schema-qualified. Rather, extnamespace indicates the schema that contains most or all of the extension's objects. If extrelocatable is true, then this schema must in fact contain all schema-qualifiable objects belonging to the extension.

Prev | Up | Next
52.21. pg_event_trigger | Home | 52.23. pg_foreign_data_wrapper


52.23. pg_foreign_data_wrapper
Prev Up Chapter 52. System Catalogs Home Next

52.23. pg_foreign_data_wrapper

The catalog pg_foreign_data_wrapper stores foreign-data wrapper definitions. A foreign-data wrapper is the mechanism by which external data, residing on foreign servers, is accessed.

Table 52.23. pg_foreign_data_wrapper Columns

Column Type

Description

oid oid

Row identifier

fdwname name

Name of the foreign-data wrapper

fdwowner oid (references pg_authid.oid)

Owner of the foreign-data wrapper

fdwhandler oid (references pg_proc.oid)

References a handler function that is responsible for supplying execution routines for the foreign-data wrapper. Zero if no handler is provided

fdwvalidator oid (references pg_proc.oid)

References a validator function that is responsible for checking the validity of the options given to the foreign-data wrapper, as well as options for foreign servers and user mappings using the foreign-data wrapper. Zero if no validator is provided

fdwacl aclitem[]

Access privileges; see Section 5.7 for details

fdwoptions text[]

Foreign-data wrapper specific options, as “keyword=value” strings

Prev | Up | Next
52.22. pg_extension | Home | 52.24. pg_foreign_server


52.24. pg_foreign_server
Prev Up Chapter 52. System Catalogs Home Next

52.24. pg_foreign_server

The catalog pg_foreign_server stores foreign server definitions. A foreign server describes a source of external data, such as a remote server. Foreign servers are accessed via foreign-data wrappers.

Table 52.24. pg_foreign_server Columns

Column Type

Description

oid oid

Row identifier

srvname name

Name of the foreign server

srvowner oid (references pg_authid.oid)

Owner of the foreign server

srvfdw oid (references pg_foreign_data_wrapper.oid)

OID of the foreign-data wrapper of this foreign server

srvtype text

Type of the server (optional)

srvversion text

Version of the server (optional)

srvacl aclitem[]

Access privileges; see Section 5.7 for details

srvoptions text[]

Foreign server specific options, as “keyword=value” strings

Prev | Up | Next
52.23. pg_foreign_data_wrapper | Home | 52.25. pg_foreign_table


52.25. pg_foreign_table
Prev Up Chapter 52. System Catalogs Home Next

52.25. pg_foreign_table

The catalog pg_foreign_table contains auxiliary information about foreign tables. A foreign table is primarily represented by a pg_class entry, just like a regular table. Its pg_foreign_table entry contains the information that is pertinent only to foreign tables and not any other kind of relation.

Table 52.25. pg_foreign_table Columns

Column Type

Description

ftrelid oid (references pg_class.oid)

The OID of the pg_class entry for this foreign table

ftserver oid (references pg_foreign_server.oid)

OID of the foreign server for this foreign table

ftoptions text[]

Foreign table options, as “keyword=value” strings

Prev | Up | Next
52.24. pg_foreign_server | Home | 52.26. pg_index


52.26. pg_index
Prev Up Chapter 52. System Catalogs Home Next

52.26. pg_index

The catalog pg_index contains part of the information about indexes. The rest is mostly in pg_class.

Table 52.26. pg_index Columns

Column Type

Description

indexrelid oid (references pg_class.oid)

The OID of the pg_class entry for this index

indrelid oid (references pg_class.oid)

The OID of the pg_class entry for the table this index is for

indnatts int2

The total number of columns in the index (duplicates pg_class.relnatts); this number includes both key and included attributes

indnkeyatts int2

The number of key columns in the index, not counting any included columns, which are merely stored and do not participate in the index semantics

indisunique bool

If true, this is a unique index

indisprimary bool

If true, this index represents the primary key of the table (indisunique should always be true when this is true)

indisexclusion bool

If true, this index supports an exclusion constraint

indimmediate bool

If true, the uniqueness check is enforced immediately on insertion (irrelevant if indisunique is not true)

indisclustered bool

If true, the table was last clustered on this index

indisvalid bool

If true, the index is currently valid for queries. False means the index is possibly incomplete: it must still be modified by INSERT/UPDATE operations, but it cannot safely be used for queries. If it is unique, the uniqueness property is not guaranteed true either.

indcheckxmin bool

If true, queries must not use the index until the xmin of this pg_index row is below their TransactionXmin event horizon, because the table may contain broken HOT chains with incompatible rows that they can see

indisready bool

If true, the index is currently ready for inserts. False means the index must be ignored by INSERT/UPDATE operations.

indislive bool

If false, the index is in process of being dropped, and should be ignored for all purposes (including HOT-safety decisions)

indisreplident bool

If true this index has been chosen as “replica identity” using ALTER TABLE ... REPLICA IDENTITY USING INDEX ...

indkey int2vector (references pg_attribute.attnum)

This is an array of indnatts values that indicate which table columns this index indexes. For example a value of 1 3 would mean that the first and the third table columns make up the index entries. Key columns come before non-key (included) columns. A zero in this array indicates that the corresponding index attribute is an expression over the table columns, rather than a simple column reference.

indcollation oidvector (references pg_collation.oid)

For each column in the index key (indnkeyatts values), this contains the OID of the collation to use for the index, or zero if the column is not of a collatable data type.

indclass oidvector (references pg_opclass.oid)

For each column in the index key (indnkeyatts values), this contains the OID of the operator class to use. See pg_opclass for details.

indoption int2vector

This is an array of indnkeyatts values that store per-column flag bits. The meaning of the bits is defined by the index's access method.

indexprs pg_node_tree

Expression trees (in nodeToString() representation) for index attributes that are not simple column references. This is a list with one element for each zero entry in indkey. Null if all index attributes are simple references.

indpred pg_node_tree

Expression tree (in nodeToString() representation) for partial index predicate. Null if not a partial index.

Prev | Up | Next
52.25. pg_foreign_table | Home | 52.27. pg_inherits


52.27. pg_inherits
Prev Up Chapter 52. System Catalogs Home Next

52.27. pg_inherits

The catalog pg_inherits records information about table and index inheritance hierarchies. There is one entry for each direct parent-child table or index relationship in the database. (Indirect inheritance can be determined by following chains of entries.)

Table 52.27. pg_inherits Columns

Column Type

Description

inhrelid oid (references pg_class.oid)

The OID of the child table or index

inhparent oid (references pg_class.oid)

The OID of the parent table or index

inhseqno int4

If there is more than one direct parent for a child table (multiple inheritance), this number tells the order in which the inherited columns are to be arranged. The count starts at 1.

Indexes cannot have multiple inheritance, since they can only inherit when using declarative partitioning.

inhdetachpending bool

true for a partition that is in the process of being detached; false otherwise.

Prev | Up | Next
52.26. pg_index | Home | 52.28. pg_init_privs


52.28. pg_init_privs
Prev Up Chapter 52. System Catalogs Home Next

52.28. pg_init_privs

The catalog pg_init_privs records information about the initial privileges of objects in the system. There is one entry for each object in the database which has a non-default (non-NULL) initial set of privileges.

Objects can have initial privileges either by having those privileges set when the system is initialized (by initdb) or when the object is created during a CREATE EXTENSION and the extension script sets initial privileges using the GRANT system. Note that the system will automatically handle recording of the privileges during the extension script and that extension authors need only use the GRANT and REVOKE statements in their script to have the privileges recorded. The privtype column indicates if the initial privilege was set by initdb or during a CREATE EXTENSION command.

Objects which have initial privileges set by initdb will have entries where privtype is 'i', while objects which have initial privileges set by CREATE EXTENSION will have entries where privtype is 'e'.

Table 52.28. pg_init_privs Columns

Column Type

Description

objoid oid (references any OID column)

The OID of the specific object

classoid oid (references pg_class.oid)

The OID of the system catalog the object is in

objsubid int4

For a table column, this is the column number (the objoid and classoid refer to the table itself). For all other object types, this column is zero.

privtype char

A code defining the type of initial privilege of this object; see text

initprivs aclitem[]

The initial access privileges; see Section 5.7 for details

Prev | Up | Next
52.27. pg_inherits | Home | 52.29. pg_language


52.29. pg_language
Prev Up Chapter 52. System Catalogs Home Next

52.29. pg_language

The catalog pg_language registers languages in which you can write functions or stored procedures. See CREATE LANGUAGE and Chapter 42 for more information about language handlers.

Table 52.29. pg_language Columns

Column Type

Description

oid oid

Row identifier

lanname name

Name of the language

lanowner oid (references pg_authid.oid)

Owner of the language

lanispl bool

This is false for internal languages (such as SQL) and true for user-defined languages. Currently, pg_dump still uses this to determine which languages need to be dumped, but this might be replaced by a different mechanism in the future.

lanpltrusted bool

True if this is a trusted language, which means that it is believed not to grant access to anything outside the normal SQL execution environment. Only superusers can create functions in untrusted languages.

lanplcallfoid oid (references pg_proc.oid)

For noninternal languages this references the language handler, which is a special function that is responsible for executing all functions that are written in the particular language. Zero for internal languages.

laninline oid (references pg_proc.oid)

This references a function that is responsible for executing “inline” anonymous code blocks (DO blocks). Zero if inline blocks are not supported.

lanvalidator oid (references pg_proc.oid)

This references a language validator function that is responsible for checking the syntax and validity of new functions when they are created. Zero if no validator is provided.

lanacl aclitem[]

Access privileges; see Section 5.7 for details

Prev | Up | Next
52.28. pg_init_privs | Home | 52.30. pg_largeobject


52.30. pg_largeobject
Prev Up Chapter 52. System Catalogs Home Next

52.30. pg_largeobject

The catalog pg_largeobject holds the data making up “large objects”. A large object is identified by an OID assigned when it is created. Each large object is broken into segments or “pages” small enough to be conveniently stored as rows in pg_largeobject. The amount of data per page is defined to be LOBLKSIZE (which is currently BLCKSZ/4, or typically 2 kB).

Prior to PostgreSQL 9.0, there was no permission structure associated with large objects. As a result, pg_largeobject was publicly readable and could be used to obtain the OIDs (and contents) of all large objects in the system. This is no longer the case; use pg_largeobject_metadata to obtain a list of large object OIDs.

Table 52.30. pg_largeobject Columns

Column Type

Description

loid oid (references pg_largeobject_metadata.oid)

Identifier of the large object that includes this page

pageno int4

Page number of this page within its large object (counting from zero)

data bytea

Actual data stored in the large object. This will never be more than LOBLKSIZE bytes and might be less.

Each row of pg_largeobject holds data for one page of a large object, beginning at byte offset (pageno * LOBLKSIZE) within the object. The implementation allows sparse storage: pages might be missing, and might be shorter than LOBLKSIZE bytes even if they are not the last page of the object. Missing regions within a large object read as zeroes.

Prev | Up | Next
52.29. pg_language | Home | 52.31. pg_largeobject_metadata


52.31. pg_largeobject_metadata
Prev Up Chapter 52. System Catalogs Home Next

52.31. pg_largeobject_metadata

The catalog pg_largeobject_metadata holds metadata associated with large objects. The actual large object data is stored in pg_largeobject.

Table 52.31. pg_largeobject_metadata Columns

Column Type

Description

oid oid

Row identifier

lomowner oid (references pg_authid.oid)

Owner of the large object

lomacl aclitem[]

Access privileges; see Section 5.7 for details

Prev | Up | Next
52.30. pg_largeobject | Home | 52.32. pg_namespace


52.32. pg_namespace
Prev Up Chapter 52. System Catalogs Home Next

52.32. pg_namespace

The catalog pg_namespace stores namespaces. A namespace is the structure underlying SQL schemas: each namespace can have a separate collection of relations, types, etc. without name conflicts.

Table 52.32. pg_namespace Columns

Column Type

Description

oid oid

Row identifier

nspname name

Name of the namespace

nspowner oid (references pg_authid.oid)

Owner of the namespace

nspacl aclitem[]

Access privileges; see Section 5.7 for details

Prev | Up | Next
52.31. pg_largeobject_metadata | Home | 52.33. pg_opclass


52.33. pg_opclass
Prev Up Chapter 52. System Catalogs Home Next

52.33. pg_opclass

The catalog pg_opclass defines index access method operator classes. Each operator class defines semantics for index columns of a particular data type and a particular index access method. An operator class essentially specifies that a particular operator family is applicable to a particular indexable column data type. The set of operators from the family that are actually usable with the indexed column are whichever ones accept the column's data type as their left-hand input.

Operator classes are described at length in Section 38.16.

Table 52.33. pg_opclass Columns

Column Type

Description

oid oid

Row identifier

opcmethod oid (references pg_am.oid)

Index access method operator class is for

opcname name

Name of this operator class

opcnamespace oid (references pg_namespace.oid)

Namespace of this operator class

opcowner oid (references pg_authid.oid)

Owner of the operator class

opcfamily oid (references pg_opfamily.oid)

Operator family containing the operator class

opcintype oid (references pg_type.oid)

Data type that the operator class indexes

opcdefault bool

True if this operator class is the default for opcintype

opckeytype oid (references pg_type.oid)

Type of data stored in index, or zero if same as opcintype

An operator class's opcmethod must match the opfmethod of its containing operator family. Also, there must be no more than one pg_opclass row having opcdefault true for any given combination of opcmethod and opcintype.

Prev | Up | Next
52.32. pg_namespace | Home | 52.34. pg_operator


52.34. pg_operator
Prev Up Chapter 52. System Catalogs Home Next

52.34. pg_operator

The catalog pg_operator stores information about operators. See CREATE OPERATOR and Section 38.14 for more information.

Table 52.34. pg_operator Columns

Column Type

Description

oid oid

Row identifier

oprname name

Name of the operator

oprnamespace oid (references pg_namespace.oid)

The OID of the namespace that contains this operator

oprowner oid (references pg_authid.oid)

Owner of the operator

oprkind char

b = infix operator (“both”), or l = prefix operator (“left”)

oprcanmerge bool

This operator supports merge joins

oprcanhash bool

This operator supports hash joins

oprleft oid (references pg_type.oid)

Type of the left operand (zero for a prefix operator)

oprright oid (references pg_type.oid)

Type of the right operand

oprresult oid (references pg_type.oid)

Type of the result (zero for a not-yet-defined “shell” operator)

oprcom oid (references pg_operator.oid)

Commutator of this operator (zero if none)

oprnegate oid (references pg_operator.oid)

Negator of this operator (zero if none)

oprcode regproc (references pg_proc.oid)

Function that implements this operator (zero for a not-yet-defined “shell” operator)

oprrest regproc (references pg_proc.oid)

Restriction selectivity estimation function for this operator (zero if none)

oprjoin regproc (references pg_proc.oid)

Join selectivity estimation function for this operator (zero if none)

Prev | Up | Next
52.33. pg_opclass | Home | 52.35. pg_opfamily


52.35. pg_opfamily
Prev Up Chapter 52. System Catalogs Home Next

52.35. pg_opfamily

The catalog pg_opfamily defines operator families. Each operator family is a collection of operators and associated support routines that implement the semantics specified for a particular index access method. Furthermore, the operators in a family are all “compatible”, in a way that is specified by the access method. The operator family concept allows cross-data-type operators to be used with indexes and to be reasoned about using knowledge of access method semantics.

Operator families are described at length in Section 38.16.

Table 52.35. pg_opfamily Columns

Column Type

Description

oid oid

Row identifier

opfmethod oid (references pg_am.oid)

Index access method operator family is for

opfname name

Name of this operator family

opfnamespace oid (references pg_namespace.oid)

Namespace of this operator family

opfowner oid (references pg_authid.oid)

Owner of the operator family

The majority of the information defining an operator family is not in its pg_opfamily row, but in the associated rows in pg_amop, pg_amproc, and pg_opclass.

Prev | Up | Next
52.34. pg_operator | Home | 52.36. pg_partitioned_table


52.36. pg_partitioned_table
Prev Up Chapter 52. System Catalogs Home Next

52.36. pg_partitioned_table

The catalog pg_partitioned_table stores information about how tables are partitioned.

Table 52.36. pg_partitioned_table Columns

Column Type

Description

partrelid oid (references pg_class.oid)

The OID of the pg_class entry for this partitioned table

partstrat char

Partitioning strategy; h = hash partitioned table, l = list partitioned table, r = range partitioned table

partnatts int2

The number of columns in the partition key

partdefid oid (references pg_class.oid)

The OID of the pg_class entry for the default partition of this partitioned table, or zero if this partitioned table does not have a default partition

partattrs int2vector (references pg_attribute.attnum)

This is an array of partnatts values that indicate which table columns are part of the partition key. For example, a value of 1 3 would mean that the first and the third table columns make up the partition key. A zero in this array indicates that the corresponding partition key column is an expression, rather than a simple column reference.

partclass oidvector (references pg_opclass.oid)

For each column in the partition key, this contains the OID of the operator class to use. See pg_opclass for details.

partcollation oidvector (references pg_collation.oid)

For each column in the partition key, this contains the OID of the collation to use for partitioning, or zero if the column is not of a collatable data type.

partexprs pg_node_tree

Expression trees (in nodeToString() representation) for partition key columns that are not simple column references. This is a list with one element for each zero entry in partattrs. Null if all partition key columns are simple references.

Prev | Up | Next
52.35. pg_opfamily | Home | 52.37. pg_policy


52.37. pg_policy
Prev Up Chapter 52. System Catalogs Home Next

52.37. pg_policy

The catalog pg_policy stores row-level security policies for tables. A policy includes the kind of command that it applies to (possibly all commands), the roles that it applies to, the expression to be added as a security-barrier qualification to queries that include the table, and the expression to be added as a WITH CHECK option for queries that attempt to add new records to the table.

Table 52.37. pg_policy Columns

Column Type

Description

oid oid

Row identifier

polname name

The name of the policy

polrelid oid (references pg_class.oid)

The table to which the policy applies

polcmd char

The command type to which the policy is applied: r for SELECT, a for INSERT, w for UPDATE, d for DELETE, or * for all

polpermissive bool

Is the policy permissive or restrictive?

polroles oid[] (references pg_authid.oid)

The roles to which the policy is applied; zero means PUBLIC (and normally appears alone in the array)

polqual pg_node_tree

The expression tree to be added to the security barrier qualifications for queries that use the table

polwithcheck pg_node_tree

The expression tree to be added to the WITH CHECK qualifications for queries that attempt to add rows to the table

Note

Policies stored in pg_policy are applied only when pg_class.relrowsecurity is set for their table.

Prev | Up | Next
52.36. pg_partitioned_table | Home | 52.38. pg_proc


52.38. pg_proc
Prev Up Chapter 52. System Catalogs Home Next

52.38. pg_proc

The catalog pg_proc stores information about functions, procedures, aggregate functions, and window functions (collectively also known as routines). See CREATE FUNCTION, CREATE PROCEDURE, and Section 38.3 for more information.

If prokind indicates that the entry is for an aggregate function, there should be a matching row in pg_aggregate.

Table 52.38. pg_proc Columns

Column Type

Description

oid oid

Row identifier

proname name

Name of the function

pronamespace oid (references pg_namespace.oid)

The OID of the namespace that contains this function

proowner oid (references pg_authid.oid)

Owner of the function

prolang oid (references pg_language.oid)

Implementation language or call interface of this function

procost float4

Estimated execution cost (in units of cpu_operator_cost); if proretset, this is cost per row returned

prorows float4

Estimated number of result rows (zero if not proretset)

provariadic oid (references pg_type.oid)

Data type of the variadic array parameter's elements, or zero if the function does not have a variadic parameter

prosupport regproc (references pg_proc.oid)

Planner support function for this function (see Section 38.11), or zero if none

prokind char

f for a normal function, p for a procedure, a for an aggregate function, or w for a window function

prosecdef bool

Function is a security definer (i.e., a “setuid” function)

proleakproof bool

The function has no side effects. No information about the arguments is conveyed except via the return value. Any function that might throw an error depending on the values of its arguments is not leak-proof.

proisstrict bool

Function returns null if any call argument is null. In that case the function won't actually be called at all. Functions that are not “strict” must be prepared to handle null inputs.

proretset bool

Function returns a set (i.e., multiple values of the specified data type)

provolatile char

provolatile tells whether the function's result depends only on its input arguments, or is affected by outside factors. It is i for “immutable” functions, which always deliver the same result for the same inputs. It is s for “stable” functions, whose results (for fixed inputs) do not change within a scan. It is v for “volatile” functions, whose results might change at any time. (Use v also for functions with side-effects, so that calls to them cannot get optimized away.)

proparallel char

proparallel tells whether the function can be safely run in parallel mode. It is s for functions which are safe to run in parallel mode without restriction. It is r for functions which can be run in parallel mode, but their execution is restricted to the parallel group leader; parallel worker processes cannot invoke these functions. It is u for functions which are unsafe in parallel mode; the presence of such a function forces a serial execution plan.

pronargs int2

Number of input arguments

pronargdefaults int2

Number of arguments that have defaults

prorettype oid (references pg_type.oid)

Data type of the return value

proargtypes oidvector (references pg_type.oid)

An array of the data types of the function arguments. This includes only input arguments (including INOUT and VARIADIC arguments), and thus represents the call signature of the function.

proallargtypes oid[] (references pg_type.oid)

An array of the data types of the function arguments. This includes all arguments (including OUT and INOUT arguments); however, if all the arguments are IN arguments, this field will be null. Note that subscripting is 1-based, whereas for historical reasons proargtypes is subscripted from 0.

proargmodes char[]

An array of the modes of the function arguments, encoded as i for IN arguments, o for OUT arguments, b for INOUT arguments, v for VARIADIC arguments, t for TABLE arguments. If all the arguments are IN arguments, this field will be null. Note that subscripts correspond to positions of proallargtypes not proargtypes.

proargnames text[]

An array of the names of the function arguments. Arguments without a name are set to empty strings in the array. If none of the arguments have a name, this field will be null. Note that subscripts correspond to positions of proallargtypes not proargtypes.

proargdefaults pg_node_tree

Expression trees (in nodeToString() representation) for default values. This is a list with pronargdefaults elements, corresponding to the last N input arguments (i.e., the last N proargtypes positions). If none of the arguments have defaults, this field will be null.

protrftypes oid[] (references pg_type.oid)

An array of the argument/result data type(s) for which to apply transforms (from the function's TRANSFORM clause). Null if none.

prosrc text

This tells the function handler how to invoke the function. It might be the actual source code of the function for interpreted languages, a link symbol, a file name, or just about anything else, depending on the implementation language/call convention.

probin text

Additional information about how to invoke the function. Again, the interpretation is language-specific.

prosqlbody pg_node_tree

Pre-parsed SQL function body. This is used for SQL-language functions when the body is given in SQL-standard notation rather than as a string literal. It's null in other cases.

proconfig text[]

Function's local settings for run-time configuration variables

proacl aclitem[]

Access privileges; see Section 5.7 for details

For compiled functions, both built-in and dynamically loaded, prosrc contains the function's C-language name (link symbol). For SQL-language functions, prosrc contains the function's source text if that is specified as a string literal; but if the function body is specified in SQL-standard style, prosrc is unused (typically it's an empty string) and prosqlbody contains the pre-parsed definition. For all other currently-known language types, prosrc contains the function's source text. probin is null except for dynamically-loaded C functions, for which it gives the name of the shared library file containing the function.

Prev | Up | Next
52.37. pg_policy | Home | 52.39. pg_publication


52.39. pg_publication
Prev Up Chapter 52. System Catalogs Home Next

52.39. pg_publication

The catalog pg_publication contains all publications created in the database. For more on publications see Section 31.1.

Table 52.39. pg_publication Columns

Column Type

Description

oid oid

Row identifier

pubname name

Name of the publication

pubowner oid (references pg_authid.oid)

Owner of the publication

puballtables bool

If true, this publication automatically includes all tables in the database, including any that will be created in the future.

pubinsert bool

If true, INSERT operations are replicated for tables in the publication.

pubupdate bool

If true, UPDATE operations are replicated for tables in the publication.

pubdelete bool

If true, DELETE operations are replicated for tables in the publication.

pubtruncate bool

If true, TRUNCATE operations are replicated for tables in the publication.

pubviaroot bool

If true, operations on a leaf partition are replicated using the identity and schema of its topmost partitioned ancestor mentioned in the publication instead of its own.

Prev | Up | Next
52.38. pg_proc | Home | 52.40. pg_publication_rel


52.40. pg_publication_rel
Prev Up Chapter 52. System Catalogs Home Next

52.40. pg_publication_rel

The catalog pg_publication_rel contains the mapping between relations and publications in the database. This is a many-to-many mapping. See also Section 52.79 for a more user-friendly view of this information.

Table 52.40. pg_publication_rel Columns

Column Type

Description

oid oid

Row identifier

prpubid oid (references pg_publication.oid)

Reference to publication

prrelid oid (references pg_class.oid)

Reference to relation

Prev | Up | Next
52.39. pg_publication | Home | 52.41. pg_range


52.41. pg_range
Prev Up Chapter 52. System Catalogs Home Next

52.41. pg_range

The catalog pg_range stores information about range types. This is in addition to the types' entries in pg_type.

Table 52.41. pg_range Columns

Column Type

Description

rngtypid oid (references pg_type.oid)

OID of the range type

rngsubtype oid (references pg_type.oid)

OID of the element type (subtype) of this range type

rngmultitypid oid (references pg_type.oid)

OID of the multirange type for this range type

rngcollation oid (references pg_collation.oid)

OID of the collation used for range comparisons, or zero if none

rngsubopc oid (references pg_opclass.oid)

OID of the subtype's operator class used for range comparisons

rngcanonical regproc (references pg_proc.oid)

OID of the function to convert a range value into canonical form, or zero if none

rngsubdiff regproc (references pg_proc.oid)

OID of the function to return the difference between two element values as double precision, or zero if none

rngsubopc (plus rngcollation, if the element type is collatable) determines the sort ordering used by the range type. rngcanonical is used when the element type is discrete. rngsubdiff is optional but should be supplied to improve performance of GiST indexes on the range type.

Prev | Up | Next
52.40. pg_publication_rel | Home | 52.42. pg_replication_origin


52.42. pg_replication_origin
Prev Up Chapter 52. System Catalogs Home Next

52.42. pg_replication_origin

The pg_replication_origin catalog contains all replication origins created. For more on replication origins see Chapter 50.

Unlike most system catalogs, pg_replication_origin is shared across all databases of a cluster: there is only one copy of pg_replication_origin per cluster, not one per database.

Table 52.42. pg_replication_origin Columns

Column Type

Description

roident oid

A unique, cluster-wide identifier for the replication origin. Should never leave the system.

roname text

The external, user defined, name of a replication origin.

Prev | Up | Next
52.41. pg_range | Home | 52.43. pg_rewrite


52.43. pg_rewrite
Prev Up Chapter 52. System Catalogs Home Next

52.43. pg_rewrite

The catalog pg_rewrite stores rewrite rules for tables and views.

Table 52.43. pg_rewrite Columns

Column Type

Description

oid oid

Row identifier

rulename name

Rule name

ev_class oid (references pg_class.oid)

The table this rule is for

ev_type char

Event type that the rule is for: 1 = SELECT, 2 = UPDATE, 3 = INSERT, 4 = DELETE

ev_enabled char

Controls in which session_replication_role modes the rule fires. O = rule fires in “origin” and “local” modes, D = rule is disabled, R = rule fires in “replica” mode, A = rule fires always.

is_instead bool

True if the rule is an INSTEAD rule

ev_qual pg_node_tree

Expression tree (in the form of a nodeToString() representation) for the rule's qualifying condition

ev_action pg_node_tree

Query tree (in the form of a nodeToString() representation) for the rule's action

Note

pg_class.relhasrules must be true if a table has any rules in this catalog.

Prev | Up | Next
52.42. pg_replication_origin | Home | 52.44. pg_seclabel


52.44. pg_seclabel
Prev Up Chapter 52. System Catalogs Home Next

52.44. pg_seclabel

The catalog pg_seclabel stores security labels on database objects. Security labels can be manipulated with the SECURITY LABEL command. For an easier way to view security labels, see Section 52.84.

See also pg_shseclabel, which performs a similar function for security labels of database objects that are shared across a database cluster.

Table 52.44. pg_seclabel Columns

Column Type

Description

objoid oid (references any OID column)

The OID of the object this security label pertains to

classoid oid (references pg_class.oid)

The OID of the system catalog this object appears in

objsubid int4

For a security label on a table column, this is the column number (the objoid and classoid refer to the table itself). For all other object types, this column is zero.

provider text

The label provider associated with this label.

label text

The security label applied to this object.

Prev | Up | Next
52.43. pg_rewrite | Home | 52.45. pg_sequence


52.45. pg_sequence
Prev Up Chapter 52. System Catalogs Home Next

52.45. pg_sequence

The catalog pg_sequence contains information about sequences. Some of the information about sequences, such as the name and the schema, is in pg_class

Table 52.45. pg_sequence Columns

Column Type

Description

seqrelid oid (references pg_class.oid)

The OID of the pg_class entry for this sequence

seqtypid oid (references pg_type.oid)

Data type of the sequence

seqstart int8

Start value of the sequence

seqincrement int8

Increment value of the sequence

seqmax int8

Maximum value of the sequence

seqmin int8

Minimum value of the sequence

seqcache int8

Cache size of the sequence

seqcycle bool

Whether the sequence cycles

Prev | Up | Next
52.44. pg_seclabel | Home | 52.46. pg_shdepend


52.46. pg_shdepend
Prev Up Chapter 52. System Catalogs Home Next

52.46. pg_shdepend

The catalog pg_shdepend records the dependency relationships between database objects and shared objects, such as roles. This information allows PostgreSQL to ensure that those objects are unreferenced before attempting to delete them.

See also pg_depend, which performs a similar function for dependencies involving objects within a single database.

Unlike most system catalogs, pg_shdepend is shared across all databases of a cluster: there is only one copy of pg_shdepend per cluster, not one per database.

Table 52.46. pg_shdepend Columns

Column Type

Description

dbid oid (references pg_database.oid)

The OID of the database the dependent object is in, or zero for a shared object or a SHARED_DEPENDENCY_PIN entry

classid oid (references pg_class.oid)

The OID of the system catalog the dependent object is in, or zero for a SHARED_DEPENDENCY_PIN entry

objid oid (references any OID column)

The OID of the specific dependent object, or zero for a SHARED_DEPENDENCY_PIN entry

objsubid int4

For a table column, this is the column number (the objid and classid refer to the table itself). For all other object types, this column is zero.

refclassid oid (references pg_class.oid)

The OID of the system catalog the referenced object is in (must be a shared catalog)

refobjid oid (references any OID column)

The OID of the specific referenced object

deptype char

A code defining the specific semantics of this dependency relationship; see text

In all cases, a pg_shdepend entry indicates that the referenced object cannot be dropped without also dropping the dependent object. However, there are several subflavors identified by deptype:

SHARED_DEPENDENCY_OWNER (o)

The referenced object (which must be a role) is the owner of the dependent object.

SHARED_DEPENDENCY_ACL (a)

The referenced object (which must be a role) is mentioned in the ACL (access control list, i.e., privileges list) of the dependent object. (A SHARED_DEPENDENCY_ACL entry is not made for the owner of the object, since the owner will have a SHARED_DEPENDENCY_OWNER entry anyway.)

SHARED_DEPENDENCY_POLICY (r)

The referenced object (which must be a role) is mentioned as the target of a dependent policy object.

SHARED_DEPENDENCY_PIN (p)

There is no dependent object; this type of entry is a signal that the system itself depends on the referenced object, and so that object must never be deleted. Entries of this type are created only by initdb. The columns for the dependent object contain zeroes.

SHARED_DEPENDENCY_TABLESPACE (t)

The referenced object (which must be a tablespace) is mentioned as the tablespace for a relation that doesn't have storage.

Other dependency flavors might be needed in future. Note in particular that the current definition only supports roles and tablespaces as referenced objects.

Prev | Up | Next
52.45. pg_sequence | Home | 52.47. pg_shdescription


52.47. pg_shdescription
Prev Up Chapter 52. System Catalogs Home Next

52.47. pg_shdescription

The catalog pg_shdescription stores optional descriptions (comments) for shared database objects. Descriptions can be manipulated with the COMMENT command and viewed with psql's \d commands.

See also pg_description, which performs a similar function for descriptions involving objects within a single database.

Unlike most system catalogs, pg_shdescription is shared across all databases of a cluster: there is only one copy of pg_shdescription per cluster, not one per database.

Table 52.47. pg_shdescription Columns

Column Type

Description

objoid oid (references any OID column)

The OID of the object this description pertains to

classoid oid (references pg_class.oid)

The OID of the system catalog this object appears in

description text

Arbitrary text that serves as the description of this object

Prev | Up | Next
52.46. pg_shdepend | Home | 52.48. pg_shseclabel


52.48. pg_shseclabel
Prev Up Chapter 52. System Catalogs Home Next

52.48. pg_shseclabel

The catalog pg_shseclabel stores security labels on shared database objects. Security labels can be manipulated with the SECURITY LABEL command. For an easier way to view security labels, see Section 52.84.

See also pg_seclabel, which performs a similar function for security labels involving objects within a single database.

Unlike most system catalogs, pg_shseclabel is shared across all databases of a cluster: there is only one copy of pg_shseclabel per cluster, not one per database.

Table 52.48. pg_shseclabel Columns

Column Type

Description

objoid oid (references any OID column)

The OID of the object this security label pertains to

classoid oid (references pg_class.oid)

The OID of the system catalog this object appears in

provider text

The label provider associated with this label.

label text

The security label applied to this object.

Prev | Up | Next
52.47. pg_shdescription | Home | 52.49. pg_statistic


52.49. pg_statistic
Prev Up Chapter 52. System Catalogs Home Next

52.49. pg_statistic

The catalog pg_statistic stores statistical data about the contents of the database. Entries are created by ANALYZE and subsequently used by the query planner. Note that all the statistical data is inherently approximate, even assuming that it is up-to-date.

Normally there is one entry, with stainherit = false, for each table column that has been analyzed. If the table has inheritance children, a second entry with stainherit = true is also created. This row represents the column's statistics over the inheritance tree, i.e., statistics for the data you'd see with SELECT column FROM table*, whereas the stainherit = false row represents the results of SELECT column FROM ONLY table.

pg_statistic also stores statistical data about the values of index expressions. These are described as if they were actual data columns; in particular, starelid references the index. No entry is made for an ordinary non-expression index column, however, since it would be redundant with the entry for the underlying table column. Currently, entries for index expressions always have stainherit = false.

Since different kinds of statistics might be appropriate for different kinds of data, pg_statistic is designed not to assume very much about what sort of statistics it stores. Only extremely general statistics (such as nullness) are given dedicated columns in pg_statistic. Everything else is stored in “slots”, which are groups of associated columns whose content is identified by a code number in one of the slot's columns. For more information see src/include/catalog/pg_statistic.h.

pg_statistic should not be readable by the public, since even statistical information about a table's contents might be considered sensitive. (Example: minimum and maximum values of a salary column might be quite interesting.) pg_stats is a publicly readable view on pg_statistic that only exposes information about those tables that are readable by the current user.

Table 52.49. pg_statistic Columns

Column Type

Description

starelid oid (references pg_class.oid)

The table or index that the described column belongs to

staattnum int2 (references pg_attribute.attnum)

The number of the described column

stainherit bool

If true, the stats include inheritance child columns, not just the values in the specified relation

stanullfrac float4

The fraction of the column's entries that are null

stawidth int4

The average stored width, in bytes, of nonnull entries

stadistinct float4

The number of distinct nonnull data values in the column. A value greater than zero is the actual number of distinct values. A value less than zero is the negative of a multiplier for the number of rows in the table; for example, a column in which about 80% of the values are nonnull and each nonnull value appears about twice on average could be represented by stadistinct = -0.4. A zero value means the number of distinct values is unknown.

stakindN int2

A code number indicating the kind of statistics stored in the Nth “slot” of the pg_statistic row.

staopN oid (references pg_operator.oid)

An operator used to derive the statistics stored in the Nth “slot”. For example, a histogram slot would show the < operator that defines the sort order of the data. Zero if the statistics kind does not require an operator.

stacollN oid (references pg_collation.oid)

The collation used to derive the statistics stored in the Nth “slot”. For example, a histogram slot for a collatable column would show the collation that defines the sort order of the data. Zero for noncollatable data.

stanumbersN float4[]

Numerical statistics of the appropriate kind for the Nth “slot”, or null if the slot kind does not involve numerical values

stavaluesN anyarray

Column data values of the appropriate kind for the Nth “slot”, or null if the slot kind does not store any data values. Each array's element values are actually of the specific column's data type, or a related type such as an array's element type, so there is no way to define these columns' type more specifically than anyarray.

Prev | Up | Next
52.48. pg_shseclabel | Home | 52.50. pg_statistic_ext


52.50. pg_statistic_ext
Prev Up Chapter 52. System Catalogs Home Next

52.50. pg_statistic_ext

The catalog pg_statistic_ext holds definitions of extended planner statistics. Each row in this catalog corresponds to a statistics object created with CREATE STATISTICS.

Table 52.50. pg_statistic_ext Columns

Column Type

Description

oid oid

Row identifier

stxrelid oid (references pg_class.oid)

Table containing the columns described by this object

stxname name

Name of the statistics object

stxnamespace oid (references pg_namespace.oid)

The OID of the namespace that contains this statistics object

stxowner oid (references pg_authid.oid)

Owner of the statistics object

stxstattarget int4

stxstattarget controls the level of detail of statistics accumulated for this statistics object by ANALYZE. A zero value indicates that no statistics should be collected. A negative value says to use the maximum of the statistics targets of the referenced columns, if set, or the system default statistics target. Positive values of stxstattarget determine the target number of “most common values” to collect.

stxkeys int2vector (references pg_attribute.attnum)

An array of attribute numbers, indicating which table columns are covered by this statistics object; for example a value of 1 3 would mean that the first and the third table columns are covered

stxkind char[]

An array containing codes for the enabled statistics kinds; valid values are: d for n-distinct statistics, f for functional dependency statistics, m for most common values (MCV) list statistics, and e for expression statistics

stxexprs pg_node_tree

Expression trees (in nodeToString() representation) for statistics object attributes that are not simple column references. This is a list with one element per expression. Null if all statistics object attributes are simple references.

The pg_statistic_ext entry is filled in completely during CREATE STATISTICS, but the actual statistical values are not computed then. Subsequent ANALYZE commands compute the desired values and populate an entry in the pg_statistic_ext_data catalog.

Prev | Up | Next
52.49. pg_statistic | Home | 52.51. pg_statistic_ext_data


52.51. pg_statistic_ext_data
Prev Up Chapter 52. System Catalogs Home Next

52.51. pg_statistic_ext_data

The catalog pg_statistic_ext_data holds data for extended planner statistics defined in pg_statistic_ext. Each row in this catalog corresponds to a statistics object created with CREATE STATISTICS.

Like pg_statistic, pg_statistic_ext_data should not be readable by the public, since the contents might be considered sensitive. (Example: most common combinations of values in columns might be quite interesting.) pg_stats_ext is a publicly readable view on pg_statistic_ext_data (after joining with pg_statistic_ext) that only exposes information about those tables and columns that are readable by the current user.

Table 52.51. pg_statistic_ext_data Columns

Column Type

Description

stxoid oid (references pg_statistic_ext.oid)

Extended statistics object containing the definition for this data

stxdndistinct pg_ndistinct

N-distinct counts, serialized as pg_ndistinct type

stxddependencies pg_dependencies

Functional dependency statistics, serialized as pg_dependencies type

stxdmcv pg_mcv_list

MCV (most-common values) list statistics, serialized as pg_mcv_list type

stxdexpr pg_statistic[]

Per-expression statistics, serialized as an array of pg_statistic type

Prev | Up | Next
52.50. pg_statistic_ext | Home | 52.52. pg_subscription


52.52. pg_subscription
Prev Up Chapter 52. System Catalogs Home Next

52.52. pg_subscription

The catalog pg_subscription contains all existing logical replication subscriptions. For more information about logical replication see Chapter 31.

Unlike most system catalogs, pg_subscription is shared across all databases of a cluster: there is only one copy of pg_subscription per cluster, not one per database.

Access to the column subconninfo is revoked from normal users, because it could contain plain-text passwords.

Table 52.52. pg_subscription Columns

Column Type

Description

oid oid

Row identifier

subdbid oid (references pg_database.oid)

OID of the database that the subscription resides in

subname name

Name of the subscription

subowner oid (references pg_authid.oid)

Owner of the subscription

subenabled bool

If true, the subscription is enabled and should be replicating

subbinary bool

If true, the subscription will request that the publisher send data in binary format

substream bool

If true, the subscription will allow streaming of in-progress transactions

subconninfo text

Connection string to the upstream database

subslotname name

Name of the replication slot in the upstream database (also used for the local replication origin name); null represents NONE

subsynccommit text

The synchronous_commit setting for the subscription's workers to use

subpublications text[]

Array of subscribed publication names. These reference publications defined in the upstream database. For more on publications see Section 31.1.

Prev | Up | Next
52.51. pg_statistic_ext_data | Home | 52.53. pg_subscription_rel


52.53. pg_subscription_rel
Prev Up Chapter 52. System Catalogs Home Next

52.53. pg_subscription_rel

The catalog pg_subscription_rel contains the state for each replicated relation in each subscription. This is a many-to-many mapping.

This catalog only contains tables known to the subscription after running either CREATE SUBSCRIPTION or ALTER SUBSCRIPTION ... REFRESH PUBLICATION.

Table 52.53. pg_subscription_rel Columns

Column Type

Description

srsubid oid (references pg_subscription.oid)

Reference to subscription

srrelid oid (references pg_class.oid)

Reference to relation

srsubstate char

State code: i = initialize, d = data is being copied, f = finished table copy, s = synchronized, r = ready (normal replication)

srsublsn pg_lsn

Remote LSN of the state change used for synchronization coordination when in s or r states, otherwise null

Prev | Up | Next
52.52. pg_subscription | Home | 52.54. pg_tablespace


52.54. pg_tablespace
Prev Up Chapter 52. System Catalogs Home Next

52.54. pg_tablespace

The catalog pg_tablespace stores information about the available tablespaces. Tables can be placed in particular tablespaces to aid administration of disk layout.

Unlike most system catalogs, pg_tablespace is shared across all databases of a cluster: there is only one copy of pg_tablespace per cluster, not one per database.

Table 52.54. pg_tablespace Columns

Column Type

Description

oid oid

Row identifier

spcname name

Tablespace name

spcowner oid (references pg_authid.oid)

Owner of the tablespace, usually the user who created it

spcacl aclitem[]

Access privileges; see Section 5.7 for details

spcoptions text[]

Tablespace-level options, as “keyword=value” strings

Prev | Up | Next
52.53. pg_subscription_rel | Home | 52.55. pg_transform


52.55. pg_transform
Prev Up Chapter 52. System Catalogs Home Next

52.55. pg_transform

The catalog pg_transform stores information about transforms, which are a mechanism to adapt data types to procedural languages. See CREATE TRANSFORM for more information.

Table 52.55. pg_transform Columns

Column Type

Description

oid oid

Row identifier

trftype oid (references pg_type.oid)

OID of the data type this transform is for

trflang oid (references pg_language.oid)

OID of the language this transform is for

trffromsql regproc (references pg_proc.oid)

The OID of the function to use when converting the data type for input to the procedural language (e.g., function parameters). Zero is stored if the default behavior should be used.

trftosql regproc (references pg_proc.oid)

The OID of the function to use when converting output from the procedural language (e.g., return values) to the data type. Zero is stored if the default behavior should be used.

Prev | Up | Next
52.54. pg_tablespace | Home | 52.56. pg_trigger


52.56. pg_trigger
Prev Up Chapter 52. System Catalogs Home Next

52.56. pg_trigger

The catalog pg_trigger stores triggers on tables and views. See CREATE TRIGGER for more information.

Table 52.56. pg_trigger Columns

Column Type

Description

oid oid

Row identifier

tgrelid oid (references pg_class.oid)

The table this trigger is on

tgparentid oid (references pg_trigger.oid)

Parent trigger that this trigger is cloned from (this happens when partitions are created or attached to a partitioned table); zero if not a clone

tgname name

Trigger name (must be unique among triggers of same table)

tgfoid oid (references pg_proc.oid)

The function to be called

tgtype int2

Bit mask identifying trigger firing conditions

tgenabled char

Controls in which session_replication_role modes the trigger fires. O = trigger fires in “origin” and “local” modes, D = trigger is disabled, R = trigger fires in “replica” mode, A = trigger fires always.

tgisinternal bool

True if trigger is internally generated (usually, to enforce the constraint identified by tgconstraint)

tgconstrrelid oid (references pg_class.oid)

The table referenced by a referential integrity constraint (zero if trigger is not for a referential integrity constraint)

tgconstrindid oid (references pg_class.oid)

The index supporting a unique, primary key, referential integrity, or exclusion constraint (zero if trigger is not for one of these types of constraint)

tgconstraint oid (references pg_constraint.oid)

The pg_constraint entry associated with the trigger (zero if trigger is not for a constraint)

tgdeferrable bool

True if constraint trigger is deferrable

tginitdeferred bool

True if constraint trigger is initially deferred

tgnargs int2

Number of argument strings passed to trigger function

tgattr int2vector (references pg_attribute.attnum)

Column numbers, if trigger is column-specific; otherwise an empty array

tgargs bytea

Argument strings to pass to trigger, each NULL-terminated

tgqual pg_node_tree

Expression tree (in nodeToString() representation) for the trigger's WHEN condition, or null if none

tgoldtable name

REFERENCING clause name for OLD TABLE, or null if none

tgnewtable name

REFERENCING clause name for NEW TABLE, or null if none

Currently, column-specific triggering is supported only for UPDATE events, and so tgattr is relevant only for that event type. tgtype might contain bits for other event types as well, but those are presumed to be table-wide regardless of what is in tgattr.

Note

When tgconstraint is nonzero, tgconstrrelid, tgconstrindid, tgdeferrable, and tginitdeferred are largely redundant with the referenced pg_constraint entry. However, it is possible for a non-deferrable trigger to be associated with a deferrable constraint: foreign key constraints can have some deferrable and some non-deferrable triggers.

Note

pg_class.relhastriggers must be true if a relation has any triggers in this catalog.

Prev | Up | Next
52.55. pg_transform | Home | 52.57. pg_ts_config


52.57. pg_ts_config
Prev Up Chapter 52. System Catalogs Home Next

52.57. pg_ts_config

The pg_ts_config catalog contains entries representing text search configurations. A configuration specifies a particular text search parser and a list of dictionaries to use for each of the parser's output token types. The parser is shown in the pg_ts_config entry, but the token-to-dictionary mapping is defined by subsidiary entries in pg_ts_config_map.

PostgreSQL's text search features are described at length in Chapter 12.

Table 52.57. pg_ts_config Columns

Column Type

Description

oid oid

Row identifier

cfgname name

Text search configuration name

cfgnamespace oid (references pg_namespace.oid)

The OID of the namespace that contains this configuration

cfgowner oid (references pg_authid.oid)

Owner of the configuration

cfgparser oid (references pg_ts_parser.oid)

The OID of the text search parser for this configuration

Prev | Up | Next
52.56. pg_trigger | Home | 52.58. pg_ts_config_map


52.58. pg_ts_config_map
Prev Up Chapter 52. System Catalogs Home Next

52.58. pg_ts_config_map

The pg_ts_config_map catalog contains entries showing which text search dictionaries should be consulted, and in what order, for each output token type of each text search configuration's parser.

PostgreSQL's text search features are described at length in Chapter 12.

Table 52.58. pg_ts_config_map Columns

Column Type

Description

mapcfg oid (references pg_ts_config.oid)

The OID of the pg_ts_config entry owning this map entry

maptokentype int4

A token type emitted by the configuration's parser

mapseqno int4

Order in which to consult this entry (lower mapseqnos first)

mapdict oid (references pg_ts_dict.oid)

The OID of the text search dictionary to consult

Prev | Up | Next
52.57. pg_ts_config | Home | 52.59. pg_ts_dict


52.59. pg_ts_dict
Prev Up Chapter 52. System Catalogs Home Next

52.59. pg_ts_dict

The pg_ts_dict catalog contains entries defining text search dictionaries. A dictionary depends on a text search template, which specifies all the implementation functions needed; the dictionary itself provides values for the user-settable parameters supported by the template. This division of labor allows dictionaries to be created by unprivileged users. The parameters are specified by a text string dictinitoption, whose format and meaning vary depending on the template.

PostgreSQL's text search features are described at length in Chapter 12.

Table 52.59. pg_ts_dict Columns

Column Type

Description

oid oid

Row identifier

dictname name

Text search dictionary name

dictnamespace oid (references pg_namespace.oid)

The OID of the namespace that contains this dictionary

dictowner oid (references pg_authid.oid)

Owner of the dictionary

dicttemplate oid (references pg_ts_template.oid)

The OID of the text search template for this dictionary

dictinitoption text

Initialization option string for the template

Prev | Up | Next
52.58. pg_ts_config_map | Home | 52.60. pg_ts_parser


52.60. pg_ts_parser
Prev Up Chapter 52. System Catalogs Home Next

52.60. pg_ts_parser

The pg_ts_parser catalog contains entries defining text search parsers. A parser is responsible for splitting input text into lexemes and assigning a token type to each lexeme. Since a parser must be implemented by C-language-level functions, creation of new parsers is restricted to database superusers.

PostgreSQL's text search features are described at length in Chapter 12.

Table 52.60. pg_ts_parser Columns

Column Type

Description

oid oid

Row identifier

prsname name

Text search parser name

prsnamespace oid (references pg_namespace.oid)

The OID of the namespace that contains this parser

prsstart regproc (references pg_proc.oid)

OID of the parser's startup function

prstoken regproc (references pg_proc.oid)

OID of the parser's next-token function

prsend regproc (references pg_proc.oid)

OID of the parser's shutdown function

prsheadline regproc (references pg_proc.oid)

OID of the parser's headline function (zero if none)

prslextype regproc (references pg_proc.oid)

OID of the parser's lextype function

Prev | Up | Next
52.59. pg_ts_dict | Home | 52.61. pg_ts_template


52.61. pg_ts_template
Prev Up Chapter 52. System Catalogs Home Next

52.61. pg_ts_template

The pg_ts_template catalog contains entries defining text search templates. A template is the implementation skeleton for a class of text search dictionaries. Since a template must be implemented by C-language-level functions, creation of new templates is restricted to database superusers.

PostgreSQL's text search features are described at length in Chapter 12.

Table 52.61. pg_ts_template Columns

Column Type

Description

oid oid

Row identifier

tmplname name

Text search template name

tmplnamespace oid (references pg_namespace.oid)

The OID of the namespace that contains this template

tmplinit regproc (references pg_proc.oid)

OID of the template's initialization function (zero if none)

tmpllexize regproc (references pg_proc.oid)

OID of the template's lexize function

Prev | Up | Next
52.60. pg_ts_parser | Home | 52.62. pg_type


52.62. pg_type
Prev Up Chapter 52. System Catalogs Home Next

52.62. pg_type

The catalog pg_type stores information about data types. Base types and enum types (scalar types) are created with CREATE TYPE, and domains with CREATE DOMAIN. A composite type is automatically created for each table in the database, to represent the row structure of the table. It is also possible to create composite types with CREATE TYPE AS.

Table 52.62. pg_type Columns

Column Type

Description

oid oid

Row identifier

typname name

Data type name

typnamespace oid (references pg_namespace.oid)

The OID of the namespace that contains this type

typowner oid (references pg_authid.oid)

Owner of the type

typlen int2

For a fixed-size type, typlen is the number of bytes in the internal representation of the type. But for a variable-length type, typlen is negative. -1 indicates a “varlena” type (one that has a length word), -2 indicates a null-terminated C string.

typbyval bool

typbyval determines whether internal routines pass a value of this type by value or by reference. typbyval had better be false if typlen is not 1, 2, or 4 (or 8 on machines where Datum is 8 bytes). Variable-length types are always passed by reference. Note that typbyval can be false even if the length would allow pass-by-value.

typtype char

typtype is b for a base type, c for a composite type (e.g., a table's row type), d for a domain, e for an enum type, p for a pseudo-type, r for a range type, or m for a multirange type. See also typrelid and typbasetype.

typcategory char

typcategory is an arbitrary classification of data types that is used by the parser to determine which implicit casts should be “preferred”. See Table 52.63.

typispreferred bool

True if the type is a preferred cast target within its typcategory

typisdefined bool

True if the type is defined, false if this is a placeholder entry for a not-yet-defined type. When typisdefined is false, nothing except the type name, namespace, and OID can be relied on.

typdelim char

Character that separates two values of this type when parsing array input. Note that the delimiter is associated with the array element data type, not the array data type.

typrelid oid (references pg_class.oid)

If this is a composite type (see typtype), then this column points to the pg_class entry that defines the corresponding table. (For a free-standing composite type, the pg_class entry doesn't really represent a table, but it is needed anyway for the type's pg_attribute entries to link to.) Zero for non-composite types.

typsubscript regproc (references pg_proc.oid)

Subscripting handler function's OID, or zero if this type doesn't support subscripting. Types that are “true” array types have typsubscript = array_subscript_handler, but other types may have other handler functions to implement specialized subscripting behavior.

typelem oid (references pg_type.oid)

If typelem is not zero then it identifies another row in pg_type, defining the type yielded by subscripting. This should be zero if typsubscript is zero. However, it can be zero when typsubscript isn't zero, if the handler doesn't need typelem to determine the subscripting result type. Note that a typelem dependency is considered to imply physical containment of the element type in this type; so DDL changes on the element type might be restricted by the presence of this type.

typarray oid (references pg_type.oid)

If typarray is not zero then it identifies another row in pg_type, which is the “true” array type having this type as element

typinput regproc (references pg_proc.oid)

Input conversion function (text format)

typoutput regproc (references pg_proc.oid)

Output conversion function (text format)

typreceive regproc (references pg_proc.oid)

Input conversion function (binary format), or zero if none

typsend regproc (references pg_proc.oid)

Output conversion function (binary format), or zero if none

typmodin regproc (references pg_proc.oid)

Type modifier input function, or zero if type does not support modifiers

typmodout regproc (references pg_proc.oid)

Type modifier output function, or zero to use the standard format

typanalyze regproc (references pg_proc.oid)

Custom ANALYZE function, or zero to use the standard function

typalign char

typalign is the alignment required when storing a value of this type. It applies to storage on disk as well as most representations of the value inside PostgreSQL. When multiple values are stored consecutively, such as in the representation of a complete row on disk, padding is inserted before a datum of this type so that it begins on the specified boundary. The alignment reference is the beginning of the first datum in the sequence. Possible values are:

  • c = char alignment, i.e., no alignment needed.

  • s = short alignment (2 bytes on most machines).

  • i = int alignment (4 bytes on most machines).

  • d = double alignment (8 bytes on many machines, but by no means all).

typstorage char

typstorage tells for varlena types (those with typlen = -1) if the type is prepared for toasting and what the default strategy for attributes of this type should be. Possible values are:

  • p (plain): Values must always be stored plain (non-varlena types always use this value).

  • e (external): Values can be stored in a secondary “TOAST” relation (if relation has one, see pg_class.reltoastrelid).

  • m (main): Values can be compressed and stored inline.

  • x (extended): Values can be compressed and/or moved to a secondary relation.

x is the usual choice for toast-able types. Note that m values can also be moved out to secondary storage, but only as a last resort (e and x values are moved first).

typnotnull bool

typnotnull represents a not-null constraint on a type. Used for domains only.

typbasetype oid (references pg_type.oid)

If this is a domain (see typtype), then typbasetype identifies the type that this one is based on. Zero if this type is not a domain.

typtypmod int4

Domains use typtypmod to record the typmod to be applied to their base type (-1 if base type does not use a typmod). -1 if this type is not a domain.

typndims int4

typndims is the number of array dimensions for a domain over an array (that is, typbasetype is an array type). Zero for types other than domains over array types.

typcollation oid (references pg_collation.oid)

typcollation specifies the collation of the type. If the type does not support collations, this will be zero. A base type that supports collations will have a nonzero value here, typically DEFAULT_COLLATION_OID. A domain over a collatable type can have a collation OID different from its base type's, if one was specified for the domain.

typdefaultbin pg_node_tree

If typdefaultbin is not null, it is the nodeToString() representation of a default expression for the type. This is only used for domains.

typdefault text

typdefault is null if the type has no associated default value. If typdefaultbin is not null, typdefault must contain a human-readable version of the default expression represented by typdefaultbin. If typdefaultbin is null and typdefault is not, then typdefault is the external representation of the type's default value, which can be fed to the type's input converter to produce a constant.

typacl aclitem[]

Access privileges; see Section 5.7 for details

Note

For fixed-width types used in system tables, it is critical that the size and alignment defined in pg_type agree with the way that the compiler will lay out the column in a structure representing a table row.

Table 52.63 lists the system-defined values of typcategory. Any future additions to this list will also be upper-case ASCII letters. All other ASCII characters are reserved for user-defined categories.

Table 52.63. typcategory Codes

Code | Category
A | Array types
B | Boolean types
C | Composite types
D | Date/time types
E | Enum types
G | Geometric types
I | Network address types
N | Numeric types
P | Pseudo-types
R | Range types
S | String types
T | Timespan types
U | User-defined types
V | Bit-string types
X | unknown type

Prev | Up | Next
52.61. pg_ts_template | Home | 52.63. pg_user_mapping


52.63. pg_user_mapping
Prev Up Chapter 52. System Catalogs Home Next

52.63. pg_user_mapping

The catalog pg_user_mapping stores the mappings from local user to remote. Access to this catalog is restricted from normal users, use the view pg_user_mappings instead.

Table 52.64. pg_user_mapping Columns

Column Type

Description

oid oid

Row identifier

umuser oid (references pg_authid.oid)

OID of the local role being mapped, or zero if the user mapping is public

umserver oid (references pg_foreign_server.oid)

The OID of the foreign server that contains this mapping

umoptions text[]

User mapping specific options, as “keyword=value” strings

Prev | Up | Next
52.62. pg_type | Home | 52.64. System Views


Chapter 53. Frontend/Backend Protocol
Prev Up Part VII. Internals Home Next

Chapter 53. Frontend/Backend Protocol

Table of Contents

53.1. Overview
53.1.1. Messaging Overview
53.1.2. Extended Query Overview
53.1.3. Formats and Format Codes
53.2. Message Flow
53.2.1. Start-up
53.2.2. Simple Query
53.2.3. Extended Query
53.2.4. Function Call
53.2.5. COPY Operations
53.2.6. Asynchronous Operations
53.2.7. Canceling Requests in Progress
53.2.8. Termination
53.2.9. SSL Session Encryption
53.2.10. GSSAPI Session Encryption
53.3. SASL Authentication
53.3.1. SCRAM-SHA-256 Authentication
53.4. Streaming Replication Protocol
53.5. Logical Streaming Replication Protocol
53.5.1. Logical Streaming Replication Parameters
53.5.2. Logical Replication Protocol Messages
53.5.3. Logical Replication Protocol Message Flow
53.6. Message Data Types
53.7. Message Formats
53.8. Error and Notice Message Fields
53.9. Logical Replication Message Formats
53.10. Summary of Changes since Protocol 2.0

PostgreSQL uses a message-based protocol for communication between frontends and backends (clients and servers). The protocol is supported over TCP/IP and also over Unix-domain sockets. Port number 5432 has been registered with IANA as the customary TCP port number for servers supporting this protocol, but in practice any non-privileged port number can be used.

This document describes version 3.0 of the protocol, implemented in PostgreSQL 7.4 and later. For descriptions of the earlier protocol versions, see previous releases of the PostgreSQL documentation. A single server can support multiple protocol versions. The initial startup-request message tells the server which protocol version the client is attempting to use. If the major version requested by the client is not supported by the server, the connection will be rejected (for example, this would occur if the client requested protocol version 4.0, which does not exist as of this writing). If the minor version requested by the client is not supported by the server (e.g., the client requests version 3.1, but the server supports only 3.0), the server may either reject the connection or may respond with a NegotiateProtocolVersion message containing the highest minor protocol version which it supports. The client may then choose either to continue with the connection using the specified protocol version or to abort the connection.

In order to serve multiple clients efficiently, the server launches a new “backend” process for each client. In the current implementation, a new child process is created immediately after an incoming connection is detected. This is transparent to the protocol, however. For purposes of the protocol, the terms “backend” and “server” are interchangeable; likewise “frontend” and “client” are interchangeable.

Prev | Up | Next
52.97. pg_views | Home | 53.1. Overview


53.1. Overview
Prev Up Chapter 53. Frontend/Backend Protocol Home Next

53.1. Overview

53.1.1. Messaging Overview
53.1.2. Extended Query Overview
53.1.3. Formats and Format Codes

The protocol has separate phases for startup and normal operation. In the startup phase, the frontend opens a connection to the server and authenticates itself to the satisfaction of the server. (This might involve a single message, or multiple messages depending on the authentication method being used.) If all goes well, the server then sends status information to the frontend, and finally enters normal operation. Except for the initial startup-request message, this part of the protocol is driven by the server.

During normal operation, the frontend sends queries and other commands to the backend, and the backend sends back query results and other responses. There are a few cases (such as NOTIFY) wherein the backend will send unsolicited messages, but for the most part this portion of a session is driven by frontend requests.

Termination of the session is normally by frontend choice, but can be forced by the backend in certain cases. In any case, when the backend closes the connection, it will roll back any open (incomplete) transaction before exiting.

Within normal operation, SQL commands can be executed through either of two sub-protocols. In the “simple query” protocol, the frontend just sends a textual query string, which is parsed and immediately executed by the backend. In the “extended query” protocol, processing of queries is separated into multiple steps: parsing, binding of parameter values, and execution. This offers flexibility and performance benefits, at the cost of extra complexity.

Normal operation has additional sub-protocols for special operations such as COPY.

53.1.1. Messaging Overview

All communication is through a stream of messages. The first byte of a message identifies the message type, and the next four bytes specify the length of the rest of the message (this length count includes itself, but not the message-type byte). The remaining contents of the message are determined by the message type. For historical reasons, the very first message sent by the client (the startup message) has no initial message-type byte.

To avoid losing synchronization with the message stream, both servers and clients typically read an entire message into a buffer (using the byte count) before attempting to process its contents. This allows easy recovery if an error is detected while processing the contents. In extreme situations (such as not having enough memory to buffer the message), the receiver can use the byte count to determine how much input to skip before it resumes reading messages.

Conversely, both servers and clients must take care never to send an incomplete message. This is commonly done by marshaling the entire message in a buffer before beginning to send it. If a communications failure occurs partway through sending or receiving a message, the only sensible response is to abandon the connection, since there is little hope of recovering message-boundary synchronization.

53.1.2. Extended Query Overview

In the extended-query protocol, execution of SQL commands is divided into multiple steps. The state retained between steps is represented by two types of objects: prepared statements and portals. A prepared statement represents the result of parsing and semantic analysis of a textual query string. A prepared statement is not in itself ready to execute, because it might lack specific values for parameters. A portal represents a ready-to-execute or already-partially-executed statement, with any missing parameter values filled in. (For SELECT statements, a portal is equivalent to an open cursor, but we choose to use a different term since cursors don't handle non-SELECT statements.)

The overall execution cycle consists of a parse step, which creates a prepared statement from a textual query string; a bind step, which creates a portal given a prepared statement and values for any needed parameters; and an execute step that runs a portal's query. In the case of a query that returns rows (SELECT, SHOW, etc), the execute step can be told to fetch only a limited number of rows, so that multiple execute steps might be needed to complete the operation.

The backend can keep track of multiple prepared statements and portals (but note that these exist only within a session, and are never shared across sessions). Existing prepared statements and portals are referenced by names assigned when they were created. In addition, an “unnamed” prepared statement and portal exist. Although these behave largely the same as named objects, operations on them are optimized for the case of executing a query only once and then discarding it, whereas operations on named objects are optimized on the expectation of multiple uses.

53.1.3. Formats and Format Codes

Data of a particular data type might be transmitted in any of several different formats. As of PostgreSQL 7.4 the only supported formats are “text” and “binary”, but the protocol makes provision for future extensions. The desired format for any value is specified by a format code. Clients can specify a format code for each transmitted parameter value and for each column of a query result. Text has format code zero, binary has format code one, and all other format codes are reserved for future definition.

The text representation of values is whatever strings are produced and accepted by the input/output conversion functions for the particular data type. In the transmitted representation, there is no trailing null character; the frontend must add one to received values if it wants to process them as C strings. (The text format does not allow embedded nulls, by the way.)

Binary representations for integers use network byte order (most significant byte first). For other data types consult the documentation or source code to learn about the binary representation. Keep in mind that binary representations for complex data types might change across server versions; the text format is usually the more portable choice.

Prev | Up | Next
Chapter 53. Frontend/Backend Protocol | Home | 53.2. Message Flow


53.2. Message Flow
Prev Up Chapter 53. Frontend/Backend Protocol Home Next

53.2. Message Flow

53.2.1. Start-up
53.2.2. Simple Query
53.2.3. Extended Query
53.2.4. Function Call
53.2.5. COPY Operations
53.2.6. Asynchronous Operations
53.2.7. Canceling Requests in Progress
53.2.8. Termination
53.2.9. SSL Session Encryption
53.2.10. GSSAPI Session Encryption

This section describes the message flow and the semantics of each message type. (Details of the exact representation of each message appear in Section 53.7.) There are several different sub-protocols depending on the state of the connection: start-up, query, function call, COPY, and termination. There are also special provisions for asynchronous operations (including notification responses and command cancellation), which can occur at any time after the start-up phase.

53.2.1. Start-up

To begin a session, a frontend opens a connection to the server and sends a startup message. This message includes the names of the user and of the database the user wants to connect to; it also identifies the particular protocol version to be used. (Optionally, the startup message can include additional settings for run-time parameters.) The server then uses this information and the contents of its configuration files (such as pg_hba.conf) to determine whether the connection is provisionally acceptable, and what additional authentication is required (if any).

The server then sends an appropriate authentication request message, to which the frontend must reply with an appropriate authentication response message (such as a password). For all authentication methods except GSSAPI, SSPI and SASL, there is at most one request and one response. In some methods, no response at all is needed from the frontend, and so no authentication request occurs. For GSSAPI, SSPI and SASL, multiple exchanges of packets may be needed to complete the authentication.

The authentication cycle ends with the server either rejecting the connection attempt (ErrorResponse), or sending AuthenticationOk.

The possible messages from the server in this phase are:

ErrorResponse

The connection attempt has been rejected. The server then immediately closes the connection.

AuthenticationOk

The authentication exchange is successfully completed.

AuthenticationKerberosV5

The frontend must now take part in a Kerberos V5 authentication dialog (not described here, part of the Kerberos specification) with the server. If this is successful, the server responds with an AuthenticationOk, otherwise it responds with an ErrorResponse. This is no longer supported.

AuthenticationCleartextPassword

The frontend must now send a PasswordMessage containing the password in clear-text form. If this is the correct password, the server responds with an AuthenticationOk, otherwise it responds with an ErrorResponse.

AuthenticationMD5Password

The frontend must now send a PasswordMessage containing the password (with user name) encrypted via MD5, then encrypted again using the 4-byte random salt specified in the AuthenticationMD5Password message. If this is the correct password, the server responds with an AuthenticationOk, otherwise it responds with an ErrorResponse. The actual PasswordMessage can be computed in SQL as concat('md5', md5(concat(md5(concat(password, username)), random-salt))). (Keep in mind the md5() function returns its result as a hex string.)

AuthenticationSCMCredential

This response is only possible for local Unix-domain connections on platforms that support SCM credential messages. The frontend must issue an SCM credential message and then send a single data byte. (The contents of the data byte are uninteresting; it's only used to ensure that the server waits long enough to receive the credential message.) If the credential is acceptable, the server responds with an AuthenticationOk, otherwise it responds with an ErrorResponse. (This message type is only issued by pre-9.1 servers. It may eventually be removed from the protocol specification.)

AuthenticationGSS

The frontend must now initiate a GSSAPI negotiation. The frontend will send a GSSResponse message with the first part of the GSSAPI data stream in response to this. If further messages are needed, the server will respond with AuthenticationGSSContinue.

AuthenticationSSPI

The frontend must now initiate an SSPI negotiation. The frontend will send a GSSResponse with the first part of the SSPI data stream in response to this. If further messages are needed, the server will respond with AuthenticationGSSContinue.

AuthenticationGSSContinue

This message contains the response data from the previous step of GSSAPI or SSPI negotiation (AuthenticationGSS, AuthenticationSSPI or a previous AuthenticationGSSContinue). If the GSSAPI or SSPI data in this message indicates more data is needed to complete the authentication, the frontend must send that data as another GSSResponse message. If GSSAPI or SSPI authentication is completed by this message, the server will next send AuthenticationOk to indicate successful authentication or ErrorResponse to indicate failure.

AuthenticationSASL

The frontend must now initiate a SASL negotiation, using one of the SASL mechanisms listed in the message. The frontend will send a SASLInitialResponse with the name of the selected mechanism, and the first part of the SASL data stream in response to this. If further messages are needed, the server will respond with AuthenticationSASLContinue. See Section 53.3 for details.

AuthenticationSASLContinue

This message contains challenge data from the previous step of SASL negotiation (AuthenticationSASL, or a previous AuthenticationSASLContinue). The frontend must respond with a SASLResponse message.

AuthenticationSASLFinal

SASL authentication has completed with additional mechanism-specific data for the client. The server will next send AuthenticationOk to indicate successful authentication, or an ErrorResponse to indicate failure. This message is sent only if the SASL mechanism specifies additional data to be sent from server to client at completion.

NegotiateProtocolVersion

The server does not support the minor protocol version requested by the client, but does support an earlier version of the protocol; this message indicates the highest supported minor version. This message will also be sent if the client requested unsupported protocol options (i.e., beginning with _pq_.) in the startup packet. This message will be followed by an ErrorResponse or a message indicating the success or failure of authentication.

If the frontend does not support the authentication method requested by the server, then it should immediately close the connection.

After having received AuthenticationOk, the frontend must wait for further messages from the server. In this phase a backend process is being started, and the frontend is just an interested bystander. It is still possible for the startup attempt to fail (ErrorResponse) or the server to decline support for the requested minor protocol version (NegotiateProtocolVersion), but in the normal case the backend will send some ParameterStatus messages, BackendKeyData, and finally ReadyForQuery.

During this phase the backend will attempt to apply any additional run-time parameter settings that were given in the startup message. If successful, these values become session defaults. An error causes ErrorResponse and exit.

The possible messages from the backend in this phase are:

BackendKeyData

This message provides secret-key data that the frontend must save if it wants to be able to issue cancel requests later. The frontend should not respond to this message, but should continue listening for a ReadyForQuery message.

ParameterStatus

This message informs the frontend about the current (initial) setting of backend parameters, such as client_encoding or DateStyle. The frontend can ignore this message, or record the settings for its future use; see Section 53.2.6 for more details. The frontend should not respond to this message, but should continue listening for a ReadyForQuery message.

ReadyForQuery

Start-up is completed. The frontend can now issue commands.

ErrorResponse

Start-up failed. The connection is closed after sending this message.

NoticeResponse

A warning message has been issued. The frontend should display the message but continue listening for ReadyForQuery or ErrorResponse.

The ReadyForQuery message is the same one that the backend will issue after each command cycle. Depending on the coding needs of the frontend, it is reasonable to consider ReadyForQuery as starting a command cycle, or to consider ReadyForQuery as ending the start-up phase and each subsequent command cycle.

53.2.2. Simple Query

A simple query cycle is initiated by the frontend sending a Query message to the backend. The message includes an SQL command (or commands) expressed as a text string. The backend then sends one or more response messages depending on the contents of the query command string, and finally a ReadyForQuery response message. ReadyForQuery informs the frontend that it can safely send a new command. (It is not actually necessary for the frontend to wait for ReadyForQuery before issuing another command, but the frontend must then take responsibility for figuring out what happens if the earlier command fails and already-issued later commands succeed.)

The possible response messages from the backend are:

CommandComplete

An SQL command completed normally.

CopyInResponse

The backend is ready to copy data from the frontend to a table; see Section 53.2.5.

CopyOutResponse

The backend is ready to copy data from a table to the frontend; see Section 53.2.5.

RowDescription

Indicates that rows are about to be returned in response to a SELECT, FETCH, etc query. The contents of this message describe the column layout of the rows. This will be followed by a DataRow message for each row being returned to the frontend.

DataRow

One of the set of rows returned by a SELECT, FETCH, etc query.

EmptyQueryResponse

An empty query string was recognized.

ErrorResponse

An error has occurred.

ReadyForQuery

Processing of the query string is complete. A separate message is sent to indicate this because the query string might contain multiple SQL commands. (CommandComplete marks the end of processing one SQL command, not the whole string.) ReadyForQuery will always be sent, whether processing terminates successfully or with an error.

NoticeResponse

A warning message has been issued in relation to the query. Notices are in addition to other responses, i.e., the backend will continue processing the command.

The response to a SELECT query (or other queries that return row sets, such as EXPLAIN or SHOW) normally consists of RowDescription, zero or more DataRow messages, and then CommandComplete. COPY to or from the frontend invokes special protocol as described in Section 53.2.5. All other query types normally produce only a CommandComplete message.

Since a query string could contain several queries (separated by semicolons), there might be several such response sequences before the backend finishes processing the query string. ReadyForQuery is issued when the entire string has been processed and the backend is ready to accept a new query string.

If a completely empty (no contents other than whitespace) query string is received, the response is EmptyQueryResponse followed by ReadyForQuery.

In the event of an error, ErrorResponse is issued followed by ReadyForQuery. All further processing of the query string is aborted by ErrorResponse (even if more queries remained in it). Note that this might occur partway through the sequence of messages generated by an individual query.

In simple Query mode, the format of retrieved values is always text, except when the given command is a FETCH from a cursor declared with the BINARY option. In that case, the retrieved values are in binary format. The format codes given in the RowDescription message tell which format is being used.

A frontend must be prepared to accept ErrorResponse and NoticeResponse messages whenever it is expecting any other type of message. See also Section 53.2.6 concerning messages that the backend might generate due to outside events.

Recommended practice is to code frontends in a state-machine style that will accept any message type at any time that it could make sense, rather than wiring in assumptions about the exact sequence of messages.

53.2.2.1. Multiple Statements in a Simple Query

When a simple Query message contains more than one SQL statement (separated by semicolons), those statements are executed as a single transaction, unless explicit transaction control commands are included to force a different behavior. For example, if the message contains

INSERT INTO mytable VALUES(1);
SELECT 1/0;
INSERT INTO mytable VALUES(2);

then the divide-by-zero failure in the SELECT will force rollback of the first INSERT. Furthermore, because execution of the message is abandoned at the first error, the second INSERT is never attempted at all.

If instead the message contains

BEGIN;
INSERT INTO mytable VALUES(1);
COMMIT;
INSERT INTO mytable VALUES(2);
SELECT 1/0;

then the first INSERT is committed by the explicit COMMIT command. The second INSERT and the SELECT are still treated as a single transaction, so that the divide-by-zero failure will roll back the second INSERT, but not the first one.

This behavior is implemented by running the statements in a multi-statement Query message in an implicit transaction block unless there is some explicit transaction block for them to run in. The main difference between an implicit transaction block and a regular one is that an implicit block is closed automatically at the end of the Query message, either by an implicit commit if there was no error, or an implicit rollback if there was an error. This is similar to the implicit commit or rollback that happens for a statement executed by itself (when not in a transaction block).

If the session is already in a transaction block, as a result of a BEGIN in some previous message, then the Query message simply continues that transaction block, whether the message contains one statement or several. However, if the Query message contains a COMMIT or ROLLBACK closing the existing transaction block, then any following statements are executed in an implicit transaction block. Conversely, if a BEGIN appears in a multi-statement Query message, then it starts a regular transaction block that will only be terminated by an explicit COMMIT or ROLLBACK, whether that appears in this Query message or a later one. If the BEGIN follows some statements that were executed as an implicit transaction block, those statements are not immediately committed; in effect, they are retroactively included into the new regular transaction block.

A COMMIT or ROLLBACK appearing in an implicit transaction block is executed as normal, closing the implicit block; however, a warning will be issued since a COMMIT or ROLLBACK without a previous BEGIN might represent a mistake. If more statements follow, a new implicit transaction block will be started for them.

Savepoints are not allowed in an implicit transaction block, since they would conflict with the behavior of automatically closing the block upon any error.

Remember that, regardless of any transaction control commands that may be present, execution of the Query message stops at the first error. Thus for example given

BEGIN;
SELECT 1/0;
ROLLBACK;

in a single Query message, the session will be left inside a failed regular transaction block, since the ROLLBACK is not reached after the divide-by-zero error. Another ROLLBACK will be needed to restore the session to a usable state.

Another behavior of note is that initial lexical and syntactic analysis is done on the entire query string before any of it is executed. Thus simple errors (such as a misspelled keyword) in later statements can prevent execution of any of the statements. This is normally invisible to users since the statements would all roll back anyway when done as an implicit transaction block. However, it can be visible when attempting to do multiple transactions within a multi-statement Query. For instance, if a typo turned our previous example into

BEGIN;
INSERT INTO mytable VALUES(1);
COMMIT;
INSERT INTO mytable VALUES(2);
SELCT 1/0;

then none of the statements would get run, resulting in the visible difference that the first INSERT is not committed. Errors detected at semantic analysis or later, such as a misspelled table or column name, do not have this effect.

53.2.3. Extended Query

The extended query protocol breaks down the above-described simple query protocol into multiple steps. The results of preparatory steps can be re-used multiple times for improved efficiency. Furthermore, additional features are available, such as the possibility of supplying data values as separate parameters instead of having to insert them directly into a query string.

In the extended protocol, the frontend first sends a Parse message, which contains a textual query string, optionally some information about data types of parameter placeholders, and the name of a destination prepared-statement object (an empty string selects the unnamed prepared statement). The response is either ParseComplete or ErrorResponse. Parameter data types can be specified by OID; if not given, the parser attempts to infer the data types in the same way as it would do for untyped literal string constants.

Note

A parameter data type can be left unspecified by setting it to zero, or by making the array of parameter type OIDs shorter than the number of parameter symbols ($n) used in the query string. Another special case is that a parameter's type can be specified as void (that is, the OID of the void pseudo-type). This is meant to allow parameter symbols to be used for function parameters that are actually OUT parameters. Ordinarily there is no context in which a void parameter could be used, but if such a parameter symbol appears in a function's parameter list, it is effectively ignored. For example, a function call such as foo($1,$2,$3,$4) could match a function with two IN and two OUT arguments, if $3 and $4 are specified as having type void.

Note

The query string contained in a Parse message cannot include more than one SQL statement; else a syntax error is reported. This restriction does not exist in the simple-query protocol, but it does exist in the extended protocol, because allowing prepared statements or portals to contain multiple commands would complicate the protocol unduly.

If successfully created, a named prepared-statement object lasts till the end of the current session, unless explicitly destroyed. An unnamed prepared statement lasts only until the next Parse statement specifying the unnamed statement as destination is issued. (Note that a simple Query message also destroys the unnamed statement.) Named prepared statements must be explicitly closed before they can be redefined by another Parse message, but this is not required for the unnamed statement. Named prepared statements can also be created and accessed at the SQL command level, using PREPARE and EXECUTE.

Once a prepared statement exists, it can be readied for execution using a Bind message. The Bind message gives the name of the source prepared statement (empty string denotes the unnamed prepared statement), the name of the destination portal (empty string denotes the unnamed portal), and the values to use for any parameter placeholders present in the prepared statement. The supplied parameter set must match those needed by the prepared statement. (If you declared any void parameters in the Parse message, pass NULL values for them in the Bind message.) Bind also specifies the format to use for any data returned by the query; the format can be specified overall, or per-column. The response is either BindComplete or ErrorResponse.

Note

The choice between text and binary output is determined by the format codes given in Bind, regardless of the SQL command involved. The BINARY attribute in cursor declarations is irrelevant when using extended query protocol.

Query planning typically occurs when the Bind message is processed. If the prepared statement has no parameters, or is executed repeatedly, the server might save the created plan and re-use it during subsequent Bind messages for the same prepared statement. However, it will do so only if it finds that a generic plan can be created that is not much less efficient than a plan that depends on the specific parameter values supplied. This happens transparently so far as the protocol is concerned.

If successfully created, a named portal object lasts till the end of the current transaction, unless explicitly destroyed. An unnamed portal is destroyed at the end of the transaction, or as soon as the next Bind statement specifying the unnamed portal as destination is issued. (Note that a simple Query message also destroys the unnamed portal.) Named portals must be explicitly closed before they can be redefined by another Bind message, but this is not required for the unnamed portal. Named portals can also be created and accessed at the SQL command level, using DECLARE CURSOR and FETCH.

Once a portal exists, it can be executed using an Execute message. The Execute message specifies the portal name (empty string denotes the unnamed portal) and a maximum result-row count (zero meaning “fetch all rows”). The result-row count is only meaningful for portals containing commands that return row sets; in other cases the command is always executed to completion, and the row count is ignored. The possible responses to Execute are the same as those described above for queries issued via simple query protocol, except that Execute doesn't cause ReadyForQuery or RowDescription to be issued.

If Execute terminates before completing the execution of a portal (due to reaching a nonzero result-row count), it will send a PortalSuspended message; the appearance of this message tells the frontend that another Execute should be issued against the same portal to complete the operation. The CommandComplete message indicating completion of the source SQL command is not sent until the portal's execution is completed. Therefore, an Execute phase is always terminated by the appearance of exactly one of these messages: CommandComplete, EmptyQueryResponse (if the portal was created from an empty query string), ErrorResponse, or PortalSuspended.

At completion of each series of extended-query messages, the frontend should issue a Sync message. This parameterless message causes the backend to close the current transaction if it's not inside a BEGIN/COMMIT transaction block (“close” meaning to commit if no error, or roll back if error). Then a ReadyForQuery response is issued. The purpose of Sync is to provide a resynchronization point for error recovery. When an error is detected while processing any extended-query message, the backend issues ErrorResponse, then reads and discards messages until a Sync is reached, then issues ReadyForQuery and returns to normal message processing. (But note that no skipping occurs if an error is detected while processing Sync — this ensures that there is one and only one ReadyForQuery sent for each Sync.)

Note

Sync does not cause a transaction block opened with BEGIN to be closed. It is possible to detect this situation since the ReadyForQuery message includes transaction status information.

In addition to these fundamental, required operations, there are several optional operations that can be used with extended-query protocol.

The Describe message (portal variant) specifies the name of an existing portal (or an empty string for the unnamed portal). The response is a RowDescription message describing the rows that will be returned by executing the portal; or a NoData message if the portal does not contain a query that will return rows; or ErrorResponse if there is no such portal.

The Describe message (statement variant) specifies the name of an existing prepared statement (or an empty string for the unnamed prepared statement). The response is a ParameterDescription message describing the parameters needed by the statement, followed by a RowDescription message describing the rows that will be returned when the statement is eventually executed (or a NoData message if the statement will not return rows). ErrorResponse is issued if there is no such prepared statement. Note that since Bind has not yet been issued, the formats to be used for returned columns are not yet known to the backend; the format code fields in the RowDescription message will be zeroes in this case.

Tip

In most scenarios the frontend should issue one or the other variant of Describe before issuing Execute, to ensure that it knows how to interpret the results it will get back.

The Close message closes an existing prepared statement or portal and releases resources. It is not an error to issue Close against a nonexistent statement or portal name. The response is normally CloseComplete, but could be ErrorResponse if some difficulty is encountered while releasing resources. Note that closing a prepared statement implicitly closes any open portals that were constructed from that statement.

The Flush message does not cause any specific output to be generated, but forces the backend to deliver any data pending in its output buffers. A Flush must be sent after any extended-query command except Sync, if the frontend wishes to examine the results of that command before issuing more commands. Without Flush, messages returned by the backend will be combined into the minimum possible number of packets to minimize network overhead.

Note

The simple Query message is approximately equivalent to the series Parse, Bind, portal Describe, Execute, Close, Sync, using the unnamed prepared statement and portal objects and no parameters. One difference is that it will accept multiple SQL statements in the query string, automatically performing the bind/describe/execute sequence for each one in succession. Another difference is that it will not return ParseComplete, BindComplete, CloseComplete, or NoData messages.

53.2.4. Function Call

The Function Call sub-protocol allows the client to request a direct call of any function that exists in the database's pg_proc system catalog. The client must have execute permission for the function.

Note

The Function Call sub-protocol is a legacy feature that is probably best avoided in new code. Similar results can be accomplished by setting up a prepared statement that does SELECT function($1, ...). The Function Call cycle can then be replaced with Bind/Execute.

A Function Call cycle is initiated by the frontend sending a FunctionCall message to the backend. The backend then sends one or more response messages depending on the results of the function call, and finally a ReadyForQuery response message. ReadyForQuery informs the frontend that it can safely send a new query or function call.

The possible response messages from the backend are:

ErrorResponse

An error has occurred.

FunctionCallResponse

The function call was completed and returned the result given in the message. (Note that the Function Call protocol can only handle a single scalar result, not a row type or set of results.)

ReadyForQuery

Processing of the function call is complete. ReadyForQuery will always be sent, whether processing terminates successfully or with an error.

NoticeResponse

A warning message has been issued in relation to the function call. Notices are in addition to other responses, i.e., the backend will continue processing the command.

53.2.5. COPY Operations

The COPY command allows high-speed bulk data transfer to or from the server. Copy-in and copy-out operations each switch the connection into a distinct sub-protocol, which lasts until the operation is completed.

Copy-in mode (data transfer to the server) is initiated when the backend executes a COPY FROM STDIN SQL statement. The backend sends a CopyInResponse message to the frontend. The frontend should then send zero or more CopyData messages, forming a stream of input data. (The message boundaries are not required to have anything to do with row boundaries, although that is often a reasonable choice.) The frontend can terminate the copy-in mode by sending either a CopyDone message (allowing successful termination) or a CopyFail message (which will cause the COPY SQL statement to fail with an error). The backend then reverts to the command-processing mode it was in before the COPY started, which will be either simple or extended query protocol. It will next send either CommandComplete (if successful) or ErrorResponse (if not).

In the event of a backend-detected error during copy-in mode (including receipt of a CopyFail message), the backend will issue an ErrorResponse message. If the COPY command was issued via an extended-query message, the backend will now discard frontend messages until a Sync message is received, then it will issue ReadyForQuery and return to normal processing. If the COPY command was issued in a simple Query message, the rest of that message is discarded and ReadyForQuery is issued. In either case, any subsequent CopyData, CopyDone, or CopyFail messages issued by the frontend will simply be dropped.

The backend will ignore Flush and Sync messages received during copy-in mode. Receipt of any other non-copy message type constitutes an error that will abort the copy-in state as described above. (The exception for Flush and Sync is for the convenience of client libraries that always send Flush or Sync after an Execute message, without checking whether the command to be executed is a COPY FROM STDIN.)

Copy-out mode (data transfer from the server) is initiated when the backend executes a COPY TO STDOUT SQL statement. The backend sends a CopyOutResponse message to the frontend, followed by zero or more CopyData messages (always one per row), followed by CopyDone. The backend then reverts to the command-processing mode it was in before the COPY started, and sends CommandComplete. The frontend cannot abort the transfer (except by closing the connection or issuing a Cancel request), but it can discard unwanted CopyData and CopyDone messages.

In the event of a backend-detected error during copy-out mode, the backend will issue an ErrorResponse message and revert to normal processing. The frontend should treat receipt of ErrorResponse as terminating the copy-out mode.

It is possible for NoticeResponse and ParameterStatus messages to be interspersed between CopyData messages; frontends must handle these cases, and should be prepared for other asynchronous message types as well (see Section 53.2.6). Otherwise, any message type other than CopyData or CopyDone may be treated as terminating copy-out mode.

There is another Copy-related mode called copy-both, which allows high-speed bulk data transfer to and from the server. Copy-both mode is initiated when a backend in walsender mode executes a START_REPLICATION statement. The backend sends a CopyBothResponse message to the frontend. Both the backend and the frontend may then send CopyData messages until either end sends a CopyDone message. After the client sends a CopyDone message, the connection goes from copy-both mode to copy-out mode, and the client may not send any more CopyData messages. Similarly, when the server sends a CopyDone message, the connection goes into copy-in mode, and the server may not send any more CopyData messages. After both sides have sent a CopyDone message, the copy mode is terminated, and the backend reverts to the command-processing mode. In the event of a backend-detected error during copy-both mode, the backend will issue an ErrorResponse message, discard frontend messages until a Sync message is received, and then issue ReadyForQuery and return to normal processing. The frontend should treat receipt of ErrorResponse as terminating the copy in both directions; no CopyDone should be sent in this case. See Section 53.4 for more information on the subprotocol transmitted over copy-both mode.

The CopyInResponse, CopyOutResponse and CopyBothResponse messages include fields that inform the frontend of the number of columns per row and the format codes being used for each column. (As of the present implementation, all columns in a given COPY operation will use the same format, but the message design does not assume this.)

53.2.6. Asynchronous Operations

There are several cases in which the backend will send messages that are not specifically prompted by the frontend's command stream. Frontends must be prepared to deal with these messages at any time, even when not engaged in a query. At minimum, one should check for these cases before beginning to read a query response.

It is possible for NoticeResponse messages to be generated due to outside activity; for example, if the database administrator commands a “fast” database shutdown, the backend will send a NoticeResponse indicating this fact before closing the connection. Accordingly, frontends should always be prepared to accept and display NoticeResponse messages, even when the connection is nominally idle.

ParameterStatus messages will be generated whenever the active value changes for any of the parameters the backend believes the frontend should know about. Most commonly this occurs in response to a SET SQL command executed by the frontend, and this case is effectively synchronous — but it is also possible for parameter status changes to occur because the administrator changed a configuration file and then sent the SIGHUP signal to the server. Also, if a SET command is rolled back, an appropriate ParameterStatus message will be generated to report the current effective value.

At present there is a hard-wired set of parameters for which ParameterStatus will be generated: they are server_version, server_encoding, client_encoding, application_name, default_transaction_read_only, in_hot_standby, is_superuser, session_authorization, DateStyle, IntervalStyle, TimeZone, integer_datetimes, and standard_conforming_strings. (server_encoding, TimeZone, and integer_datetimes were not reported by releases before 8.0; standard_conforming_strings was not reported by releases before 8.1; IntervalStyle was not reported by releases before 8.4; application_name was not reported by releases before 9.0; default_transaction_read_only and in_hot_standby were not reported by releases before 14.) Note that server_version, server_encoding and integer_datetimes are pseudo-parameters that cannot change after startup. This set might change in the future, or even become configurable. Accordingly, a frontend should simply ignore ParameterStatus for parameters that it does not understand or care about.

If a frontend issues a LISTEN command, then the backend will send a NotificationResponse message (not to be confused with NoticeResponse!) whenever a NOTIFY command is executed for the same channel name.

Note

At present, NotificationResponse can only be sent outside a transaction, and thus it will not occur in the middle of a command-response series, though it might occur just before ReadyForQuery. It is unwise to design frontend logic that assumes that, however. Good practice is to be able to accept NotificationResponse at any point in the protocol.

53.2.7. Canceling Requests in Progress

During the processing of a query, the frontend might request cancellation of the query. The cancel request is not sent directly on the open connection to the backend for reasons of implementation efficiency: we don't want to have the backend constantly checking for new input from the frontend during query processing. Cancel requests should be relatively infrequent, so we make them slightly cumbersome in order to avoid a penalty in the normal case.

To issue a cancel request, the frontend opens a new connection to the server and sends a CancelRequest message, rather than the StartupMessage message that would ordinarily be sent across a new connection. The server will process this request and then close the connection. For security reasons, no direct reply is made to the cancel request message.

A CancelRequest message will be ignored unless it contains the same key data (PID and secret key) passed to the frontend during connection start-up. If the request matches the PID and secret key for a currently executing backend, the processing of the current query is aborted. (In the existing implementation, this is done by sending a special signal to the backend process that is processing the query.)

The cancellation signal might or might not have any effect — for example, if it arrives after the backend has finished processing the query, then it will have no effect. If the cancellation is effective, it results in the current command being terminated early with an error message.

The upshot of all this is that for reasons of both security and efficiency, the frontend has no direct way to tell whether a cancel request has succeeded. It must continue to wait for the backend to respond to the query. Issuing a cancel simply improves the odds that the current query will finish soon, and improves the odds that it will fail with an error message instead of succeeding.

Since the cancel request is sent across a new connection to the server and not across the regular frontend/backend communication link, it is possible for the cancel request to be issued by any process, not just the frontend whose query is to be canceled. This might provide additional flexibility when building multiple-process applications. It also introduces a security risk, in that unauthorized persons might try to cancel queries. The security risk is addressed by requiring a dynamically generated secret key to be supplied in cancel requests.

53.2.8. Termination

The normal, graceful termination procedure is that the frontend sends a Terminate message and immediately closes the connection. On receipt of this message, the backend closes the connection and terminates.

In rare cases (such as an administrator-commanded database shutdown) the backend might disconnect without any frontend request to do so. In such cases the backend will attempt to send an error or notice message giving the reason for the disconnection before it closes the connection.

Other termination scenarios arise from various failure cases, such as core dump at one end or the other, loss of the communications link, loss of message-boundary synchronization, etc. If either frontend or backend sees an unexpected closure of the connection, it should clean up and terminate. The frontend has the option of launching a new backend by recontacting the server if it doesn't want to terminate itself. Closing the connection is also advisable if an unrecognizable message type is received, since this probably indicates loss of message-boundary sync.

For either normal or abnormal termination, any open transaction is rolled back, not committed. One should note however that if a frontend disconnects while a non-SELECT query is being processed, the backend will probably finish the query before noticing the disconnection. If the query is outside any transaction block (BEGIN ... COMMIT sequence) then its results might be committed before the disconnection is recognized.

53.2.9. SSL Session Encryption

If PostgreSQL was built with SSL support, frontend/backend communications can be encrypted using SSL. This provides communication security in environments where attackers might be able to capture the session traffic. For more information on encrypting PostgreSQL sessions with SSL, see Section 19.9.

To initiate an SSL-encrypted connection, the frontend initially sends an SSLRequest message rather than a StartupMessage. The server then responds with a single byte containing S or N, indicating that it is willing or unwilling to perform SSL, respectively. The frontend might close the connection at this point if it is dissatisfied with the response. To continue after S, perform an SSL startup handshake (not described here, part of the SSL specification) with the server. If this is successful, continue with sending the usual StartupMessage. In this case the StartupMessage and all subsequent data will be SSL-encrypted. To continue after N, send the usual StartupMessage and proceed without encryption. (Alternatively, it is permissible to issue a GSSENCRequest message after an N response to try to use GSSAPI encryption instead of SSL.)

The frontend should also be prepared to handle an ErrorMessage response to SSLRequest from the server. This would only occur if the server predates the addition of SSL support to PostgreSQL. (Such servers are now very ancient, and likely do not exist in the wild anymore.) In this case the connection must be closed, but the frontend might choose to open a fresh connection and proceed without requesting SSL.

When SSL encryption can be performed, the server is expected to send only the single S byte and then wait for the frontend to initiate an SSL handshake. If additional bytes are available to read at this point, it likely means that a man-in-the-middle is attempting to perform a buffer-stuffing attack (CVE-2021-23222). Frontends should be coded either to read exactly one byte from the socket before turning the socket over to their SSL library, or to treat it as a protocol violation if they find they have read additional bytes.

An initial SSLRequest can also be used in a connection that is being opened to send a CancelRequest message.

While the protocol itself does not provide a way for the server to force SSL encryption, the administrator can configure the server to reject unencrypted sessions as a byproduct of authentication checking.

53.2.10. GSSAPI Session Encryption

If PostgreSQL was built with GSSAPI support, frontend/backend communications can be encrypted using GSSAPI. This provides communication security in environments where attackers might be able to capture the session traffic. For more information on encrypting PostgreSQL sessions with GSSAPI, see Section 19.10.

To initiate a GSSAPI-encrypted connection, the frontend initially sends a GSSENCRequest message rather than a StartupMessage. The server then responds with a single byte containing G or N, indicating that it is willing or unwilling to perform GSSAPI encryption, respectively. The frontend might close the connection at this point if it is dissatisfied with the response. To continue after G, using the GSSAPI C bindings as discussed in RFC 2744 or equivalent, perform a GSSAPI initialization by calling gss_init_sec_context() in a loop and sending the result to the server, starting with an empty input and then with each result from the server, until it returns no output. When sending the results of gss_init_sec_context() to the server, prepend the length of the message as a four byte integer in network byte order. To continue after N, send the usual StartupMessage and proceed without encryption. (Alternatively, it is permissible to issue an SSLRequest message after an N response to try to use SSL encryption instead of GSSAPI.)

The frontend should also be prepared to handle an ErrorMessage response to GSSENCRequest from the server. This would only occur if the server predates the addition of GSSAPI encryption support to PostgreSQL. In this case the connection must be closed, but the frontend might choose to open a fresh connection and proceed without requesting GSSAPI encryption.

When GSSAPI encryption can be performed, the server is expected to send only the single G byte and then wait for the frontend to initiate a GSSAPI handshake. If additional bytes are available to read at this point, it likely means that a man-in-the-middle is attempting to perform a buffer-stuffing attack (CVE-2021-23222). Frontends should be coded either to read exactly one byte from the socket before turning the socket over to their GSSAPI library, or to treat it as a protocol violation if they find they have read additional bytes.

An initial GSSENCRequest can also be used in a connection that is being opened to send a CancelRequest message.

Once GSSAPI encryption has been successfully established, use gss_wrap() to encrypt the usual StartupMessage and all subsequent data, prepending the length of the result from gss_wrap() as a four byte integer in network byte order to the actual encrypted payload. Note that the server will only accept encrypted packets from the client which are less than 16kB; gss_wrap_size_limit() should be used by the client to determine the size of the unencrypted message which will fit within this limit and larger messages should be broken up into multiple gss_wrap() calls. Typical segments are 8kB of unencrypted data, resulting in encrypted packets of slightly larger than 8kB but well within the 16kB maximum. The server can be expected to not send encrypted packets of larger than 16kB to the client.

While the protocol itself does not provide a way for the server to force GSSAPI encryption, the administrator can configure the server to reject unencrypted sessions as a byproduct of authentication checking.

Prev | Up | Next
53.1. Overview | Home | 53.3. SASL Authentication


53.4. Streaming Replication Protocol
Prev Up Chapter 53. Frontend/Backend Protocol Home Next

53.4. Streaming Replication Protocol

To initiate streaming replication, the frontend sends the replication parameter in the startup message. A Boolean value of true (or on, yes, 1) tells the backend to go into physical replication walsender mode, wherein a small set of replication commands, shown below, can be issued instead of SQL statements.

Passing database as the value for the replication parameter instructs the backend to go into logical replication walsender mode, connecting to the database specified in the dbname parameter. In logical replication walsender mode, the replication commands shown below as well as normal SQL commands can be issued.

In either physical replication or logical replication walsender mode, only the simple query protocol can be used.

For the purpose of testing replication commands, you can make a replication connection via psql or any other libpq-using tool with a connection string including the replication option, e.g.:

psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"

However, it is often more useful to use pg_receivewal (for physical replication) or pg_recvlogical (for logical replication).

Replication commands are logged in the server log when log_replication_commands is enabled.

The commands accepted in replication mode are:

IDENTIFY_SYSTEM

Requests the server to identify itself. Server replies with a result set of a single row, containing four fields:

systemid (text)

The unique system identifier identifying the cluster. This can be used to check that the base backup used to initialize the standby came from the same cluster.

timeline (int4)

Current timeline ID. Also useful to check that the standby is consistent with the primary.

xlogpos (text)

Current WAL flush location. Useful to get a known location in the write-ahead log where streaming can start.

dbname (text)

Database connected to or null.

SHOW name

Requests the server to send the current setting of a run-time parameter. This is similar to the SQL command SHOW.

name

The name of a run-time parameter. Available parameters are documented in Chapter 20.

TIMELINE_HISTORY tli

Requests the server to send over the timeline history file for timeline tli. Server replies with a result set of a single row, containing two fields. While the fields are labeled as text, they effectively return raw bytes, with no encoding conversion:

filename (text)

File name of the timeline history file, e.g., 00000002.history.

content (text)

Contents of the timeline history file.

CREATE_REPLICATION_SLOT slot_name [ TEMPORARY ] { PHYSICAL [ RESERVE_WAL ] | LOGICAL output_plugin [ EXPORT_SNAPSHOT | NOEXPORT_SNAPSHOT | USE_SNAPSHOT ] }

Create a physical or logical replication slot. See Section 27.2.6 for more about replication slots.

slot_name

The name of the slot to create. Must be a valid replication slot name (see Section 27.2.6.1).

output_plugin

The name of the output plugin used for logical decoding (see Section 49.6).

TEMPORARY

Specify that this replication slot is a temporary one. Temporary slots are not saved to disk and are automatically dropped on error or when the session has finished.

RESERVE_WAL

Specify that this physical replication slot reserves WAL immediately. Otherwise, WAL is only reserved upon connection from a streaming replication client.

EXPORT_SNAPSHOT
NOEXPORT_SNAPSHOT
USE_SNAPSHOT

Decides what to do with the snapshot created during logical slot initialization. EXPORT_SNAPSHOT, which is the default, will export the snapshot for use in other sessions. This option can't be used inside a transaction. USE_SNAPSHOT will use the snapshot for the current transaction executing the command. This option must be used in a transaction, and CREATE_REPLICATION_SLOT must be the first command run in that transaction. Finally, NOEXPORT_SNAPSHOT will just use the snapshot for logical decoding as normal but won't do anything else with it.

In response to this command, the server will send a one-row result set containing the following fields:

slot_name (text)

The name of the newly-created replication slot.

consistent_point (text)

The WAL location at which the slot became consistent. This is the earliest location from which streaming can start on this replication slot.

snapshot_name (text)

The identifier of the snapshot exported by the command. The snapshot is valid until a new command is executed on this connection or the replication connection is closed. Null if the created slot is physical.

output_plugin (text)

The name of the output plugin used by the newly-created replication slot. Null if the created slot is physical.

START_REPLICATION [ SLOT slot_name ] [ PHYSICAL ] XXX/XXX [ TIMELINE tli ]

Instructs server to start streaming WAL, starting at WAL location XXX/XXX. If TIMELINE option is specified, streaming starts on timeline tli; otherwise, the server's current timeline is selected. The server can reply with an error, for example if the requested section of WAL has already been recycled. On success, the server responds with a CopyBothResponse message, and then starts to stream WAL to the frontend.

If a slot's name is provided via slot_name, it will be updated as replication progresses so that the server knows which WAL segments, and if hot_standby_feedback is on which transactions, are still needed by the standby.

If the client requests a timeline that's not the latest but is part of the history of the server, the server will stream all the WAL on that timeline starting from the requested start point up to the point where the server switched to another timeline. If the client requests streaming at exactly the end of an old timeline, the server skips COPY mode entirely.

After streaming all the WAL on a timeline that is not the latest one, the server will end streaming by exiting the COPY mode. When the client acknowledges this by also exiting COPY mode, the server sends a result set with one row and two columns, indicating the next timeline in this server's history. The first column is the next timeline's ID (type int8), and the second column is the WAL location where the switch happened (type text). Usually, the switch position is the end of the WAL that was streamed, but there are corner cases where the server can send some WAL from the old timeline that it has not itself replayed before promoting. Finally, the server sends two CommandComplete messages (one that ends the CopyData and the other ends the START_REPLICATION itself), and is ready to accept a new command.

WAL data is sent as a series of CopyData messages. (This allows other information to be intermixed; in particular the server can send an ErrorResponse message if it encounters a failure after beginning to stream.) The payload of each CopyData message from server to the client contains a message of one of the following formats:

XLogData (B)

Byte1('w')

Identifies the message as WAL data.

Int64

The starting point of the WAL data in this message.

Int64

The current end of WAL on the server.

Int64

The server's system clock at the time of transmission, as microseconds since midnight on 2000-01-01.

Byten

A section of the WAL data stream.

A single WAL record is never split across two XLogData messages. When a WAL record crosses a WAL page boundary, and is therefore already split using continuation records, it can be split at the page boundary. In other words, the first main WAL record and its continuation records can be sent in different XLogData messages.

Primary keepalive message (B)

Byte1('k')

Identifies the message as a sender keepalive.

Int64

The current end of WAL on the server.

Int64

The server's system clock at the time of transmission, as microseconds since midnight on 2000-01-01.

Byte1

1 means that the client should reply to this message as soon as possible, to avoid a timeout disconnect. 0 otherwise.

The receiving process can send replies back to the sender at any time, using one of the following message formats (also in the payload of a CopyData message):

Standby status update (F)

Byte1('r')

Identifies the message as a receiver status update.

Int64

The location of the last WAL byte + 1 received and written to disk in the standby.

Int64

The location of the last WAL byte + 1 flushed to disk in the standby.

Int64

The location of the last WAL byte + 1 applied in the standby.

Int64

The client's system clock at the time of transmission, as microseconds since midnight on 2000-01-01.

Byte1

If 1, the client requests the server to reply to this message immediately. This can be used to ping the server, to test if the connection is still healthy.

Hot Standby feedback message (F)

Byte1('h')

Identifies the message as a Hot Standby feedback message.

Int64

The client's system clock at the time of transmission, as microseconds since midnight on 2000-01-01.

Int32

The standby's current global xmin, excluding the catalog_xmin from any replication slots. If both this value and the following catalog_xmin are 0 this is treated as a notification that Hot Standby feedback will no longer be sent on this connection. Later non-zero messages may reinitiate the feedback mechanism.

Int32

The epoch of the global xmin xid on the standby.

Int32

The lowest catalog_xmin of any replication slots on the standby. Set to 0 if no catalog_xmin exists on the standby or if hot standby feedback is being disabled.

Int32

The epoch of the catalog_xmin xid on the standby.

START_REPLICATION SLOT slot_name LOGICAL XXX/XXX [ ( option_name [ option_value ] [, ...] ) ]

Instructs server to start streaming WAL for logical replication, starting at WAL location XXX/XXX. The server can reply with an error, for example if the requested section of WAL has already been recycled. On success, server responds with a CopyBothResponse message, and then starts to stream WAL to the frontend.

The messages inside the CopyBothResponse messages are of the same format documented for START_REPLICATION ... PHYSICAL, including two CommandComplete messages.

The output plugin associated with the selected slot is used to process the output for streaming.

SLOT slot_name

The name of the slot to stream changes from. This parameter is required, and must correspond to an existing logical replication slot created with CREATE_REPLICATION_SLOT in LOGICAL mode.

XXX/XXX

The WAL location to begin streaming at.

option_name

The name of an option passed to the slot's logical decoding plugin.

option_value

Optional value, in the form of a string constant, associated with the specified option.

DROP_REPLICATION_SLOT slot_name [ WAIT ]

Drops a replication slot, freeing any reserved server-side resources. If the slot is a logical slot that was created in a database other than the database the walsender is connected to, this command fails.

slot_name

The name of the slot to drop.

WAIT

This option causes the command to wait if the slot is active until it becomes inactive, instead of the default behavior of raising an error.

BASE_BACKUP [ LABEL 'label' ] [ PROGRESS ] [ FAST ] [ WAL ] [ NOWAIT ] [ MAX_RATE rate ] [ TABLESPACE_MAP ] [ NOVERIFY_CHECKSUMS ] [ MANIFEST manifest_option ] [ MANIFEST_CHECKSUMS checksum_algorithm ]

Instructs the server to start streaming a base backup. The system will automatically be put in backup mode before the backup is started, and taken out of it when the backup is complete. The following options are accepted:

LABEL 'label'

Sets the label of the backup. If none is specified, a backup label of base backup will be used. The quoting rules for the label are the same as a standard SQL string with standard_conforming_strings turned on.

PROGRESS

Request information required to generate a progress report. This will send back an approximate size in the header of each tablespace, which can be used to calculate how far along the stream is done. This is calculated by enumerating all the file sizes once before the transfer is even started, and might as such have a negative impact on the performance. In particular, it might take longer before the first data is streamed. Since the database files can change during the backup, the size is only approximate and might both grow and shrink between the time of approximation and the sending of the actual files.

FAST

Request a fast checkpoint.

WAL

Include the necessary WAL segments in the backup. This will include all the files between start and stop backup in the pg_wal directory of the base directory tar file.

NOWAIT

By default, the backup will wait until the last required WAL segment has been archived, or emit a warning if log archiving is not enabled. Specifying NOWAIT disables both the waiting and the warning, leaving the client responsible for ensuring the required log is available.

MAX_RATE rate

Limit (throttle) the maximum amount of data transferred from server to client per unit of time. The expected unit is kilobytes per second. If this option is specified, the value must either be equal to zero or it must fall within the range from 32 kB through 1 GB (inclusive). If zero is passed or the option is not specified, no restriction is imposed on the transfer.

TABLESPACE_MAP

Include information about symbolic links present in the directory pg_tblspc in a file named tablespace_map. The tablespace map file includes each symbolic link name as it exists in the directory pg_tblspc/ and the full path of that symbolic link.

NOVERIFY_CHECKSUMS

By default, checksums are verified during a base backup if they are enabled. Specifying NOVERIFY_CHECKSUMS disables this verification.

MANIFEST manifest_option

When this option is specified with a value of yes or force-encode, a backup manifest is created and sent along with the backup. The manifest is a list of every file present in the backup with the exception of any WAL files that may be included. It also stores the size, last modification time, and optionally a checksum for each file. A value of force-encode forces all filenames to be hex-encoded; otherwise, this type of encoding is performed only for files whose names are non-UTF8 octet sequences. force-encode is intended primarily for testing purposes, to be sure that clients which read the backup manifest can handle this case. For compatibility with previous releases, the default is MANIFEST 'no'.

MANIFEST_CHECKSUMS checksum_algorithm

Specifies the checksum algorithm that should be applied to each file included in the backup manifest. Currently, the available algorithms are NONE, CRC32C, SHA224, SHA256, SHA384, and SHA512. The default is CRC32C.

When the backup is started, the server will first send two ordinary result sets, followed by one or more CopyOutResponse results.

The first ordinary result set contains the starting position of the backup, in a single row with two columns. The first column contains the start position given in XLogRecPtr format, and the second column contains the corresponding timeline ID.

The second ordinary result set has one row for each tablespace. The fields in this row are:

spcoid (oid)

The OID of the tablespace, or null if it's the base directory.

spclocation (text)

The full path of the tablespace directory, or null if it's the base directory.

size (int8)

The approximate size of the tablespace, in kilobytes (1024 bytes), if progress report has been requested; otherwise it's null.

After the second regular result set, one or more CopyOutResponse results will be sent, one for the main data directory and one for each additional tablespace other than pg_default and pg_global. The data in the CopyOutResponse results will be a tar format (following the “ustar interchange format” specified in the POSIX 1003.1-2008 standard) dump of the tablespace contents, except that the two trailing blocks of zeroes specified in the standard are omitted. After the tar data is complete, and if a backup manifest was requested, another CopyOutResponse result is sent, containing the manifest data for the current base backup. In any case, a final ordinary result set will be sent, containing the WAL end position of the backup, in the same format as the start position.

The tar archive for the data directory and each tablespace will contain all files in the directories, regardless of whether they are PostgreSQL files or other files added to the same directory. The only excluded files are:

  • postmaster.pid

  • postmaster.opts

  • pg_internal.init (found in multiple directories)

  • Various temporary files and directories created during the operation of the PostgreSQL server, such as any file or directory beginning with pgsql_tmp and temporary relations.

  • Unlogged relations, except for the init fork which is required to recreate the (empty) unlogged relation on recovery.

  • pg_wal, including subdirectories. If the backup is run with WAL files included, a synthesized version of pg_wal will be included, but it will only contain the files necessary for the backup to work, not the rest of the contents.

  • pg_dynshmem, pg_notify, pg_replslot, pg_serial, pg_snapshots, pg_stat_tmp, and pg_subtrans are copied as empty directories (even if they are symbolic links).

  • Files other than regular files and directories, such as symbolic links (other than for the directories listed above) and special device files, are skipped. (Symbolic links in pg_tblspc are maintained.)

Owner, group, and file mode are set if the underlying file system on the server supports it.

Prev | Up | Next
53.3. SASL Authentication | Home | 53.5. Logical Streaming Replication Protocol


53.5. Logical Streaming Replication Protocol
Prev Up Chapter 53. Frontend/Backend Protocol Home Next

53.5. Logical Streaming Replication Protocol

53.5.1. Logical Streaming Replication Parameters
53.5.2. Logical Replication Protocol Messages
53.5.3. Logical Replication Protocol Message Flow

This section describes the logical replication protocol, which is the message flow started by the START_REPLICATION SLOT slot_name LOGICAL replication command.

The logical streaming replication protocol builds on the primitives of the physical streaming replication protocol.

53.5.1. Logical Streaming Replication Parameters

The logical replication START_REPLICATION command accepts following parameters:

proto_version

Protocol version. Currently versions 1 and 2 are supported. The version 2 is supported only for server version 14 and above, and it allows streaming of large in-progress transactions.

publication_names

Comma separated list of publication names for which to subscribe (receive changes). The individual publication names are treated as standard objects names and can be quoted the same as needed.

53.5.2. Logical Replication Protocol Messages

The individual protocol messages are discussed in the following subsections. Individual messages are described in Section 53.9.

All top-level protocol messages begin with a message type byte. While represented in code as a character, this is a signed byte with no associated encoding.

Since the streaming replication protocol supplies a message length there is no need for top-level protocol messages to embed a length in their header.

53.5.3. Logical Replication Protocol Message Flow

With the exception of the START_REPLICATION command and the replay progress messages, all information flows only from the backend to the frontend.

The logical replication protocol sends individual transactions one by one. This means that all messages between a pair of Begin and Commit messages belong to the same transaction. It also sends changes of large in-progress transactions between a pair of Stream Start and Stream Stop messages. The last stream of such a transaction contains Stream Commit or Stream Abort message.

Every sent transaction contains zero or more DML messages (Insert, Update, Delete). In case of a cascaded setup it can also contain Origin messages. The origin message indicates that the transaction originated on different replication node. Since a replication node in the scope of logical replication protocol can be pretty much anything, the only identifier is the origin name. It's downstream's responsibility to handle this as needed (if needed). The Origin message is always sent before any DML messages in the transaction.

Every DML message contains a relation OID, identifying the publisher's relation that was acted on. Before the first DML message for a given relation OID, a Relation message will be sent, describing the schema of that relation. Subsequently, a new Relation message will be sent if the relation's definition has changed since the last Relation message was sent for it. (The protocol assumes that the client is capable of remembering this metadata for as many relations as needed.)

Relation messages identify column types by their OIDs. In the case of a built-in type, it is assumed that the client can look up that type OID locally, so no additional data is needed. For a non-built-in type OID, a Type message will be sent before the Relation message, to provide the type name associated with that OID. Thus, a client that needs to specifically identify the types of relation columns should cache the contents of Type messages, and first consult that cache to see if the type OID is defined there. If not, look up the type OID locally.

Prev | Up | Next
53.4. Streaming Replication Protocol | Home | 53.6. Message Data Types


53.6. Message Data Types
Prev Up Chapter 53. Frontend/Backend Protocol Home Next

53.6. Message Data Types

This section describes the base data types used in messages.

Intn(i)

An n-bit integer in network byte order (most significant byte first). If i is specified it is the exact value that will appear, otherwise the value is variable. Eg. Int16, Int32(42).

Intn[k]

An array of k n-bit integers, each in network byte order. The array length k is always determined by an earlier field in the message. Eg. Int16[M].

String(s)

A null-terminated string (C-style string). There is no specific length limitation on strings. If s is specified it is the exact value that will appear, otherwise the value is variable. Eg. String, String("user").

Note

There is no predefined limit on the length of a string that can be returned by the backend. Good coding strategy for a frontend is to use an expandable buffer so that anything that fits in memory can be accepted. If that's not feasible, read the full string and discard trailing characters that don't fit into your fixed-size buffer.

Byten(c)

Exactly n bytes. If the field width n is not a constant, it is always determinable from an earlier field in the message. If c is specified it is the exact value. Eg. Byte2, Byte1('\n').

Prev | Up | Next
53.5. Logical Streaming Replication Protocol | Home | 53.7. Message Formats


53.7. Message Formats
Prev Up Chapter 53. Frontend/Backend Protocol Home Next

53.7. Message Formats

This section describes the detailed format of each message. Each is marked to indicate that it can be sent by a frontend (F), a backend (B), or both (F & B). Notice that although each message includes a byte count at the beginning, the message format is defined so that the message end can be found without reference to the byte count. This aids validity checking. (The CopyData message is an exception, because it forms part of a data stream; the contents of any individual CopyData message cannot be interpretable on their own.)

AuthenticationOk (B)

Byte1('R')

Identifies the message as an authentication request.

Int32(8)

Length of message contents in bytes, including self.

Int32(0)

Specifies that the authentication was successful.

AuthenticationKerberosV5 (B)

Byte1('R')

Identifies the message as an authentication request.

Int32(8)

Length of message contents in bytes, including self.

Int32(2)

Specifies that Kerberos V5 authentication is required.

AuthenticationCleartextPassword (B)

Byte1('R')

Identifies the message as an authentication request.

Int32(8)

Length of message contents in bytes, including self.

Int32(3)

Specifies that a clear-text password is required.

AuthenticationMD5Password (B)

Byte1('R')

Identifies the message as an authentication request.

Int32(12)

Length of message contents in bytes, including self.

Int32(5)

Specifies that an MD5-encrypted password is required.

Byte4

The salt to use when encrypting the password.

AuthenticationSCMCredential (B)

Byte1('R')

Identifies the message as an authentication request.

Int32(8)

Length of message contents in bytes, including self.

Int32(6)

Specifies that an SCM credentials message is required.

AuthenticationGSS (B)

Byte1('R')

Identifies the message as an authentication request.

Int32(8)

Length of message contents in bytes, including self.

Int32(7)

Specifies that GSSAPI authentication is required.

AuthenticationGSSContinue (B)

Byte1('R')

Identifies the message as an authentication request.

Int32

Length of message contents in bytes, including self.

Int32(8)

Specifies that this message contains GSSAPI or SSPI data.

Byten

GSSAPI or SSPI authentication data.

AuthenticationSSPI (B)

Byte1('R')

Identifies the message as an authentication request.

Int32(8)

Length of message contents in bytes, including self.

Int32(9)

Specifies that SSPI authentication is required.

AuthenticationSASL (B)

Byte1('R')

Identifies the message as an authentication request.

Int32

Length of message contents in bytes, including self.

Int32(10)

Specifies that SASL authentication is required.

The message body is a list of SASL authentication mechanisms, in the server's order of preference. A zero byte is required as terminator after the last authentication mechanism name. For each mechanism, there is the following:

String

Name of a SASL authentication mechanism.

AuthenticationSASLContinue (B)

Byte1('R')

Identifies the message as an authentication request.

Int32

Length of message contents in bytes, including self.

Int32(11)

Specifies that this message contains a SASL challenge.

Byten

SASL data, specific to the SASL mechanism being used.

AuthenticationSASLFinal (B)

Byte1('R')

Identifies the message as an authentication request.

Int32

Length of message contents in bytes, including self.

Int32(12)

Specifies that SASL authentication has completed.

Byten

SASL outcome "additional data", specific to the SASL mechanism being used.

BackendKeyData (B)

Byte1('K')

Identifies the message as cancellation key data. The frontend must save these values if it wishes to be able to issue CancelRequest messages later.

Int32(12)

Length of message contents in bytes, including self.

Int32

The process ID of this backend.

Int32

The secret key of this backend.

Bind (F)

Byte1('B')

Identifies the message as a Bind command.

Int32

Length of message contents in bytes, including self.

String

The name of the destination portal (an empty string selects the unnamed portal).

String

The name of the source prepared statement (an empty string selects the unnamed prepared statement).

Int16

The number of parameter format codes that follow (denoted C below). This can be zero to indicate that there are no parameters or that the parameters all use the default format (text); or one, in which case the specified format code is applied to all parameters; or it can equal the actual number of parameters.

Int16[C]

The parameter format codes. Each must presently be zero (text) or one (binary).

Int16

The number of parameter values that follow (possibly zero). This must match the number of parameters needed by the query.

Next, the following pair of fields appear for each parameter:

Int32

The length of the parameter value, in bytes (this count does not include itself). Can be zero. As a special case, -1 indicates a NULL parameter value. No value bytes follow in the NULL case.

Byten

The value of the parameter, in the format indicated by the associated format code. n is the above length.

After the last parameter, the following fields appear:

Int16

The number of result-column format codes that follow (denoted R below). This can be zero to indicate that there are no result columns or that the result columns should all use the default format (text); or one, in which case the specified format code is applied to all result columns (if any); or it can equal the actual number of result columns of the query.

Int16[R]

The result-column format codes. Each must presently be zero (text) or one (binary).

BindComplete (B)

Byte1('2')

Identifies the message as a Bind-complete indicator.

Int32(4)

Length of message contents in bytes, including self.

CancelRequest (F)

Int32(16)

Length of message contents in bytes, including self.

Int32(80877102)

The cancel request code. The value is chosen to contain 1234 in the most significant 16 bits, and 5678 in the least significant 16 bits. (To avoid confusion, this code must not be the same as any protocol version number.)

Int32

The process ID of the target backend.

Int32

The secret key for the target backend.

Close (F)

Byte1('C')

Identifies the message as a Close command.

Int32

Length of message contents in bytes, including self.

Byte1

'S' to close a prepared statement; or 'P' to close a portal.

String

The name of the prepared statement or portal to close (an empty string selects the unnamed prepared statement or portal).

CloseComplete (B)

Byte1('3')

Identifies the message as a Close-complete indicator.

Int32(4)

Length of message contents in bytes, including self.

CommandComplete (B)

Byte1('C')

Identifies the message as a command-completed response.

Int32

Length of message contents in bytes, including self.

String

The command tag. This is usually a single word that identifies which SQL command was completed.

For an INSERT command, the tag is INSERT oid rows, where rows is the number of rows inserted. oid used to be the object ID of the inserted row if rows was 1 and the target table had OIDs, but OIDs system columns are not supported anymore; therefore oid is always 0.

For a DELETE command, the tag is DELETE rows where rows is the number of rows deleted.

For an UPDATE command, the tag is UPDATE rows where rows is the number of rows updated.

For a SELECT or CREATE TABLE AS command, the tag is SELECT rows where rows is the number of rows retrieved.

For a MOVE command, the tag is MOVE rows where rows is the number of rows the cursor's position has been changed by.

For a FETCH command, the tag is FETCH rows where rows is the number of rows that have been retrieved from the cursor.

For a COPY command, the tag is COPY rows where rows is the number of rows copied. (Note: the row count appears only in PostgreSQL 8.2 and later.)

CopyData (F & B)

Byte1('d')

Identifies the message as COPY data.

Int32

Length of message contents in bytes, including self.

Byten

Data that forms part of a COPY data stream. Messages sent from the backend will always correspond to single data rows, but messages sent by frontends might divide the data stream arbitrarily.

CopyDone (F & B)

Byte1('c')

Identifies the message as a COPY-complete indicator.

Int32(4)

Length of message contents in bytes, including self.

CopyFail (F)

Byte1('f')

Identifies the message as a COPY-failure indicator.

Int32

Length of message contents in bytes, including self.

String

An error message to report as the cause of failure.

CopyInResponse (B)

Byte1('G')

Identifies the message as a Start Copy In response. The frontend must now send copy-in data (if not prepared to do so, send a CopyFail message).

Int32

Length of message contents in bytes, including self.

Int8

0 indicates the overall COPY format is textual (rows separated by newlines, columns separated by separator characters, etc). 1 indicates the overall copy format is binary (similar to DataRow format). See COPY for more information.

Int16

The number of columns in the data to be copied (denoted N below).

Int16[N]

The format codes to be used for each column. Each must presently be zero (text) or one (binary). All must be zero if the overall copy format is textual.

CopyOutResponse (B)

Byte1('H')

Identifies the message as a Start Copy Out response. This message will be followed by copy-out data.

Int32

Length of message contents in bytes, including self.

Int8

0 indicates the overall COPY format is textual (rows separated by newlines, columns separated by separator characters, etc). 1 indicates the overall copy format is binary (similar to DataRow format). See COPY for more information.

Int16

The number of columns in the data to be copied (denoted N below).

Int16[N]

The format codes to be used for each column. Each must presently be zero (text) or one (binary). All must be zero if the overall copy format is textual.

CopyBothResponse (B)

Byte1('W')

Identifies the message as a Start Copy Both response. This message is used only for Streaming Replication.

Int32

Length of message contents in bytes, including self.

Int8

0 indicates the overall COPY format is textual (rows separated by newlines, columns separated by separator characters, etc). 1 indicates the overall copy format is binary (similar to DataRow format). See COPY for more information.

Int16

The number of columns in the data to be copied (denoted N below).

Int16[N]

The format codes to be used for each column. Each must presently be zero (text) or one (binary). All must be zero if the overall copy format is textual.

DataRow (B)

Byte1('D')

Identifies the message as a data row.

Int32

Length of message contents in bytes, including self.

Int16

The number of column values that follow (possibly zero).

Next, the following pair of fields appear for each column:

Int32

The length of the column value, in bytes (this count does not include itself). Can be zero. As a special case, -1 indicates a NULL column value. No value bytes follow in the NULL case.

Byten

The value of the column, in the format indicated by the associated format code. n is the above length.

Describe (F)

Byte1('D')

Identifies the message as a Describe command.

Int32

Length of message contents in bytes, including self.

Byte1

'S' to describe a prepared statement; or 'P' to describe a portal.

String

The name of the prepared statement or portal to describe (an empty string selects the unnamed prepared statement or portal).

EmptyQueryResponse (B)

Byte1('I')

Identifies the message as a response to an empty query string. (This substitutes for CommandComplete.)

Int32(4)

Length of message contents in bytes, including self.

ErrorResponse (B)

Byte1('E')

Identifies the message as an error.

Int32

Length of message contents in bytes, including self.

The message body consists of one or more identified fields, followed by a zero byte as a terminator. Fields can appear in any order. For each field there is the following:

Byte1

A code identifying the field type; if zero, this is the message terminator and no string follows. The presently defined field types are listed in Section 53.8. Since more field types might be added in future, frontends should silently ignore fields of unrecognized type.

String

The field value.

Execute (F)

Byte1('E')

Identifies the message as an Execute command.

Int32

Length of message contents in bytes, including self.

String

The name of the portal to execute (an empty string selects the unnamed portal).

Int32

Maximum number of rows to return, if portal contains a query that returns rows (ignored otherwise). Zero denotes “no limit”.

Flush (F)

Byte1('H')

Identifies the message as a Flush command.

Int32(4)

Length of message contents in bytes, including self.

FunctionCall (F)

Byte1('F')

Identifies the message as a function call.

Int32

Length of message contents in bytes, including self.

Int32

Specifies the object ID of the function to call.

Int16

The number of argument format codes that follow (denoted C below). This can be zero to indicate that there are no arguments or that the arguments all use the default format (text); or one, in which case the specified format code is applied to all arguments; or it can equal the actual number of arguments.

Int16[C]

The argument format codes. Each must presently be zero (text) or one (binary).

Int16

Specifies the number of arguments being supplied to the function.

Next, the following pair of fields appear for each argument:

Int32

The length of the argument value, in bytes (this count does not include itself). Can be zero. As a special case, -1 indicates a NULL argument value. No value bytes follow in the NULL case.

Byten

The value of the argument, in the format indicated by the associated format code. n is the above length.

After the last argument, the following field appears:

Int16

The format code for the function result. Must presently be zero (text) or one (binary).

FunctionCallResponse (B)

Byte1('V')

Identifies the message as a function call result.

Int32

Length of message contents in bytes, including self.

Int32

The length of the function result value, in bytes (this count does not include itself). Can be zero. As a special case, -1 indicates a NULL function result. No value bytes follow in the NULL case.

Byten

The value of the function result, in the format indicated by the associated format code. n is the above length.

GSSENCRequest (F)

Int32(8)

Length of message contents in bytes, including self.

Int32(80877104)

The GSSAPI Encryption request code. The value is chosen to contain 1234 in the most significant 16 bits, and 5680 in the least significant 16 bits. (To avoid confusion, this code must not be the same as any protocol version number.)

GSSResponse (F)

Byte1('p')

Identifies the message as a GSSAPI or SSPI response. Note that this is also used for SASL and password response messages. The exact message type can be deduced from the context.

Int32

Length of message contents in bytes, including self.

Byten

GSSAPI/SSPI specific message data.

NegotiateProtocolVersion (B)

Byte1('v')

Identifies the message as a protocol version negotiation message.

Int32

Length of message contents in bytes, including self.

Int32

Newest minor protocol version supported by the server for the major protocol version requested by the client.

Int32

Number of protocol options not recognized by the server.

Then, for protocol option not recognized by the server, there is the following:

String

The option name.

NoData (B)

Byte1('n')

Identifies the message as a no-data indicator.

Int32(4)

Length of message contents in bytes, including self.

NoticeResponse (B)

Byte1('N')

Identifies the message as a notice.

Int32

Length of message contents in bytes, including self.

The message body consists of one or more identified fields, followed by a zero byte as a terminator. Fields can appear in any order. For each field there is the following:

Byte1

A code identifying the field type; if zero, this is the message terminator and no string follows. The presently defined field types are listed in Section 53.8. Since more field types might be added in future, frontends should silently ignore fields of unrecognized type.

String

The field value.

NotificationResponse (B)

Byte1('A')

Identifies the message as a notification response.

Int32

Length of message contents in bytes, including self.

Int32

The process ID of the notifying backend process.

String

The name of the channel that the notify has been raised on.

String

The “payload” string passed from the notifying process.

ParameterDescription (B)

Byte1('t')

Identifies the message as a parameter description.

Int32

Length of message contents in bytes, including self.

Int16

The number of parameters used by the statement (can be zero).

Then, for each parameter, there is the following:

Int32

Specifies the object ID of the parameter data type.

ParameterStatus (B)

Byte1('S')

Identifies the message as a run-time parameter status report.

Int32

Length of message contents in bytes, including self.

String

The name of the run-time parameter being reported.

String

The current value of the parameter.

Parse (F)

Byte1('P')

Identifies the message as a Parse command.

Int32

Length of message contents in bytes, including self.

String

The name of the destination prepared statement (an empty string selects the unnamed prepared statement).

String

The query string to be parsed.

Int16

The number of parameter data types specified (can be zero). Note that this is not an indication of the number of parameters that might appear in the query string, only the number that the frontend wants to prespecify types for.

Then, for each parameter, there is the following:

Int32

Specifies the object ID of the parameter data type. Placing a zero here is equivalent to leaving the type unspecified.

ParseComplete (B)

Byte1('1')

Identifies the message as a Parse-complete indicator.

Int32(4)

Length of message contents in bytes, including self.

PasswordMessage (F)

Byte1('p')

Identifies the message as a password response. Note that this is also used for GSSAPI, SSPI and SASL response messages. The exact message type can be deduced from the context.

Int32

Length of message contents in bytes, including self.

String

The password (encrypted, if requested).

PortalSuspended (B)

Byte1('s')

Identifies the message as a portal-suspended indicator. Note this only appears if an Execute message's row-count limit was reached.

Int32(4)

Length of message contents in bytes, including self.

Query (F)

Byte1('Q')

Identifies the message as a simple query.

Int32

Length of message contents in bytes, including self.

String

The query string itself.

ReadyForQuery (B)

Byte1('Z')

Identifies the message type. ReadyForQuery is sent whenever the backend is ready for a new query cycle.

Int32(5)

Length of message contents in bytes, including self.

Byte1

Current backend transaction status indicator. Possible values are 'I' if idle (not in a transaction block); 'T' if in a transaction block; or 'E' if in a failed transaction block (queries will be rejected until block is ended).

RowDescription (B)

Byte1('T')

Identifies the message as a row description.

Int32

Length of message contents in bytes, including self.

Int16

Specifies the number of fields in a row (can be zero).

Then, for each field, there is the following:

String

The field name.

Int32

If the field can be identified as a column of a specific table, the object ID of the table; otherwise zero.

Int16

If the field can be identified as a column of a specific table, the attribute number of the column; otherwise zero.

Int32

The object ID of the field's data type.

Int16

The data type size (see pg_type.typlen). Note that negative values denote variable-width types.

Int32

The type modifier (see pg_attribute.atttypmod). The meaning of the modifier is type-specific.

Int16

The format code being used for the field. Currently will be zero (text) or one (binary). In a RowDescription returned from the statement variant of Describe, the format code is not yet known and will always be zero.

SASLInitialResponse (F)

Byte1('p')

Identifies the message as an initial SASL response. Note that this is also used for GSSAPI, SSPI and password response messages. The exact message type is deduced from the context.

Int32

Length of message contents in bytes, including self.

String

Name of the SASL authentication mechanism that the client selected.

Int32

Length of SASL mechanism specific "Initial Client Response" that follows, or -1 if there is no Initial Response.

Byten

SASL mechanism specific "Initial Response".

SASLResponse (F)

Byte1('p')

Identifies the message as a SASL response. Note that this is also used for GSSAPI, SSPI and password response messages. The exact message type can be deduced from the context.

Int32

Length of message contents in bytes, including self.

Byten

SASL mechanism specific message data.

SSLRequest (F)

Int32(8)

Length of message contents in bytes, including self.

Int32(80877103)

The SSL request code. The value is chosen to contain 1234 in the most significant 16 bits, and 5679 in the least significant 16 bits. (To avoid confusion, this code must not be the same as any protocol version number.)

StartupMessage (F)

Int32

Length of message contents in bytes, including self.

Int32(196608)

The protocol version number. The most significant 16 bits are the major version number (3 for the protocol described here). The least significant 16 bits are the minor version number (0 for the protocol described here).

The protocol version number is followed by one or more pairs of parameter name and value strings. A zero byte is required as a terminator after the last name/value pair. Parameters can appear in any order. user is required, others are optional. Each parameter is specified as:

String

The parameter name. Currently recognized names are:

user

The database user name to connect as. Required; there is no default.

database

The database to connect to. Defaults to the user name.

options

Command-line arguments for the backend. (This is deprecated in favor of setting individual run-time parameters.) Spaces within this string are considered to separate arguments, unless escaped with a backslash (\); write \\ to represent a literal backslash.

replication

Used to connect in streaming replication mode, where a small set of replication commands can be issued instead of SQL statements. Value can be true, false, or database, and the default is false. See Section 53.4 for details.

In addition to the above, other parameters may be listed. Parameter names beginning with _pq_. are reserved for use as protocol extensions, while others are treated as run-time parameters to be set at backend start time. Such settings will be applied during backend start (after parsing the command-line arguments if any) and will act as session defaults.

String

The parameter value.

Sync (F)

Byte1('S')

Identifies the message as a Sync command.

Int32(4)

Length of message contents in bytes, including self.

Terminate (F)

Byte1('X')

Identifies the message as a termination.

Int32(4)

Length of message contents in bytes, including self.

Prev | Up | Next
53.6. Message Data Types | Home | 53.8. Error and Notice Message Fields


53.8. Error and Notice Message Fields
Prev Up Chapter 53. Frontend/Backend Protocol Home Next

53.8. Error and Notice Message Fields

This section describes the fields that can appear in ErrorResponse and NoticeResponse messages. Each field type has a single-byte identification token. Note that any given field type should appear at most once per message.

S

Severity: the field contents are ERROR, FATAL, or PANIC (in an error message), or WARNING, NOTICE, DEBUG, INFO, or LOG (in a notice message), or a localized translation of one of these. Always present.

V

Severity: the field contents are ERROR, FATAL, or PANIC (in an error message), or WARNING, NOTICE, DEBUG, INFO, or LOG (in a notice message). This is identical to the S field except that the contents are never localized. This is present only in messages generated by PostgreSQL versions 9.6 and later.

C

Code: the SQLSTATE code for the error (see Appendix A). Not localizable. Always present.

M

Message: the primary human-readable error message. This should be accurate but terse (typically one line). Always present.

D

Detail: an optional secondary error message carrying more detail about the problem. Might run to multiple lines.

H

Hint: an optional suggestion what to do about the problem. This is intended to differ from Detail in that it offers advice (potentially inappropriate) rather than hard facts. Might run to multiple lines.

P

Position: the field value is a decimal ASCII integer, indicating an error cursor position as an index into the original query string. The first character has index 1, and positions are measured in characters not bytes.

p

Internal position: this is defined the same as the P field, but it is used when the cursor position refers to an internally generated command rather than the one submitted by the client. The q field will always appear when this field appears.

q

Internal query: the text of a failed internally-generated command. This could be, for example, an SQL query issued by a PL/pgSQL function.

W

Where: an indication of the context in which the error occurred. Presently this includes a call stack traceback of active procedural language functions and internally-generated queries. The trace is one entry per line, most recent first.

s

Schema name: if the error was associated with a specific database object, the name of the schema containing that object, if any.

t

Table name: if the error was associated with a specific table, the name of the table. (Refer to the schema name field for the name of the table's schema.)

c

Column name: if the error was associated with a specific table column, the name of the column. (Refer to the schema and table name fields to identify the table.)

d

Data type name: if the error was associated with a specific data type, the name of the data type. (Refer to the schema name field for the name of the data type's schema.)

n

Constraint name: if the error was associated with a specific constraint, the name of the constraint. Refer to fields listed above for the associated table or domain. (For this purpose, indexes are treated as constraints, even if they weren't created with constraint syntax.)

F

File: the file name of the source-code location where the error was reported.

L

Line: the line number of the source-code location where the error was reported.

R

Routine: the name of the source-code routine reporting the error.

Note

The fields for schema name, table name, column name, data type name, and constraint name are supplied only for a limited number of error types; see Appendix A. Frontends should not assume that the presence of any of these fields guarantees the presence of another field. Core error sources observe the interrelationships noted above, but user-defined functions may use these fields in other ways. In the same vein, clients should not assume that these fields denote contemporary objects in the current database.

The client is responsible for formatting displayed information to meet its needs; in particular it should break long lines as needed. Newline characters appearing in the error message fields should be treated as paragraph breaks, not line breaks.

Prev | Up | Next
53.7. Message Formats | Home | 53.9. Logical Replication Message Formats


53.9. Logical Replication Message Formats
Prev Up Chapter 53. Frontend/Backend Protocol Home Next

53.9. Logical Replication Message Formats

This section describes the detailed format of each logical replication message. These messages are returned either by the replication slot SQL interface or are sent by a walsender. In case of a walsender they are encapsulated inside the replication protocol WAL messages as described in Section 53.4 and generally obey same message flow as physical replication.

Begin

Byte1('B')

Identifies the message as a begin message.

Int64

The final LSN of the transaction.

Int64

Commit timestamp of the transaction. The value is in number of microseconds since PostgreSQL epoch (2000-01-01).

Int32

Xid of the transaction.

Message

Byte1('M')

Identifies the message as a logical decoding message.

Int32

Xid of the transaction (only present for streamed transactions). This field is available since protocol version 2.

Int8

Flags; Either 0 for no flags or 1 if the logical decoding message is transactional.

Int64

The LSN of the logical decoding message.

String

The prefix of the logical decoding message.

Int32

Length of the content.

Byten

The content of the logical decoding message.

Commit

Byte1('C')

Identifies the message as a commit message.

Int8

Flags; currently unused (must be 0).

Int64

The LSN of the commit.

Int64

The end LSN of the transaction.

Int64

Commit timestamp of the transaction. The value is in number of microseconds since PostgreSQL epoch (2000-01-01).

Origin

Byte1('O')

Identifies the message as an origin message.

Int64

The LSN of the commit on the origin server.

String

Name of the origin.

Note that there can be multiple Origin messages inside a single transaction.

Relation

Byte1('R')

Identifies the message as a relation message.

Int32

Xid of the transaction (only present for streamed transactions). This field is available since protocol version 2.

Int32

ID of the relation.

String

Namespace (empty string for pg_catalog).

String

Relation name.

Int8

Replica identity setting for the relation (same as relreplident in pg_class).

Int16

Number of columns.

Next, the following message part appears for each column (except generated columns):

Int8

Flags for the column. Currently can be either 0 for no flags or 1 which marks the column as part of the key.

String

Name of the column.

Int32

ID of the column's data type.

Int32

Type modifier of the column (atttypmod).

Type

Byte1('Y')

Identifies the message as a type message.

Int32

Xid of the transaction (only present for streamed transactions). This field is available since protocol version 2.

Int32

ID of the data type.

String

Namespace (empty string for pg_catalog).

String

Name of the data type.

Insert

Byte1('I')

Identifies the message as an insert message.

Int32

Xid of the transaction (only present for streamed transactions). This field is available since protocol version 2.

Int32

ID of the relation corresponding to the ID in the relation message.

Byte1('N')

Identifies the following TupleData message as a new tuple.

TupleData

TupleData message part representing the contents of new tuple.

Update

Byte1('U')

Identifies the message as an update message.

Int32

Xid of the transaction (only present for streamed transactions). This field is available since protocol version 2.

Int32

ID of the relation corresponding to the ID in the relation message.

Byte1('K')

Identifies the following TupleData submessage as a key. This field is optional and is only present if the update changed data in any of the column(s) that are part of the REPLICA IDENTITY index.

Byte1('O')

Identifies the following TupleData submessage as an old tuple. This field is optional and is only present if table in which the update happened has REPLICA IDENTITY set to FULL.

TupleData

TupleData message part representing the contents of the old tuple or primary key. Only present if the previous 'O' or 'K' part is present.

Byte1('N')

Identifies the following TupleData message as a new tuple.

TupleData

TupleData message part representing the contents of a new tuple.

The Update message may contain either a 'K' message part or an 'O' message part or neither of them, but never both of them.

Delete

Byte1('D')

Identifies the message as a delete message.

Int32

Xid of the transaction (only present for streamed transactions). This field is available since protocol version 2.

Int32

ID of the relation corresponding to the ID in the relation message.

Byte1('K')

Identifies the following TupleData submessage as a key. This field is present if the table in which the delete has happened uses an index as REPLICA IDENTITY.

Byte1('O')

Identifies the following TupleData message as an old tuple. This field is present if the table in which the delete happened has REPLICA IDENTITY set to FULL.

TupleData

TupleData message part representing the contents of the old tuple or primary key, depending on the previous field.

The Delete message may contain either a 'K' message part or an 'O' message part, but never both of them.

Truncate

Byte1('T')

Identifies the message as a truncate message.

Int32

Xid of the transaction (only present for streamed transactions). This field is available since protocol version 2.

Int32

Number of relations

Int8

Option bits for TRUNCATE: 1 for CASCADE, 2 for RESTART IDENTITY

Int32

ID of the relation corresponding to the ID in the relation message. This field is repeated for each relation.

The following messages (Stream Start, Stream Stop, Stream Commit, and Stream Abort) are available since protocol version 2.

Stream Start

Byte1('S')

Identifies the message as a stream start message.

Int32

Xid of the transaction.

Int8

A value of 1 indicates this is the first stream segment for this XID, 0 for any other stream segment.

Stream Stop

Byte1('E')

Identifies the message as a stream stop message.

Stream Commit

Byte1('c')

Identifies the message as a stream commit message.

Int32

Xid of the transaction.

Int8

Flags; currently unused (must be 0).

Int64

The LSN of the commit.

Int64

The end LSN of the transaction.

Int64

Commit timestamp of the transaction. The value is in number of microseconds since PostgreSQL epoch (2000-01-01).

Stream Abort

Byte1('A')

Identifies the message as a stream abort message.

Int32

Xid of the transaction.

Int32

Xid of the subtransaction (will be same as xid of the transaction for top-level transactions).

The following message parts are shared by the above messages.

TupleData

Int16

Number of columns.

Next, one of the following submessages appears for each column (except generated columns):

Byte1('n')

Identifies the data as NULL value.

Or

Byte1('u')

Identifies unchanged TOASTed value (the actual value is not sent).

Or

Byte1('t')

Identifies the data as text formatted value.

Or

Byte1('b')

Identifies the data as binary formatted value.

Int32

Length of the column value.

Byten

The value of the column, either in binary or in text format. (As specified in the preceding format byte). n is the above length.

Prev | Up | Next
53.8. Error and Notice Message Fields | Home | 53.10. Summary of Changes since Protocol 2.0


53.10. Summary of Changes since Protocol 2.0
Prev Up Chapter 53. Frontend/Backend Protocol Home Next

53.10. Summary of Changes since Protocol 2.0

This section provides a quick checklist of changes, for the benefit of developers trying to update existing client libraries to protocol 3.0.

The initial startup packet uses a flexible list-of-strings format instead of a fixed format. Notice that session default values for run-time parameters can now be specified directly in the startup packet. (Actually, you could do that before using the options field, but given the limited width of options and the lack of any way to quote whitespace in the values, it wasn't a very safe technique.)

All messages now have a length count immediately following the message type byte (except for startup packets, which have no type byte). Also note that PasswordMessage now has a type byte.

ErrorResponse and NoticeResponse ('E' and 'N') messages now contain multiple fields, from which the client code can assemble an error message of the desired level of verbosity. Note that individual fields will typically not end with a newline, whereas the single string sent in the older protocol always did.

The ReadyForQuery ('Z') message includes a transaction status indicator.

The distinction between BinaryRow and DataRow message types is gone; the single DataRow message type serves for returning data in all formats. Note that the layout of DataRow has changed to make it easier to parse. Also, the representation of binary values has changed: it is no longer directly tied to the server's internal representation.

There is a new “extended query” sub-protocol, which adds the frontend message types Parse, Bind, Execute, Describe, Close, Flush, and Sync, and the backend message types ParseComplete, BindComplete, PortalSuspended, ParameterDescription, NoData, and CloseComplete. Existing clients do not have to concern themselves with this sub-protocol, but making use of it might allow improvements in performance or functionality.

COPY data is now encapsulated into CopyData and CopyDone messages. There is a well-defined way to recover from errors during COPY. The special “\.” last line is not needed anymore, and is not sent during COPY OUT. (It is still recognized as a terminator during COPY IN, but its use is deprecated and will eventually be removed.) Binary COPY is supported. The CopyInResponse and CopyOutResponse messages include fields indicating the number of columns and the format of each column.

The layout of FunctionCall and FunctionCallResponse messages has changed. FunctionCall can now support passing NULL arguments to functions. It also can handle passing parameters and retrieving results in either text or binary format. There is no longer any reason to consider FunctionCall a potential security hole, since it does not offer direct access to internal server data representations.

The backend sends ParameterStatus ('S') messages during connection startup for all parameters it considers interesting to the client library. Subsequently, a ParameterStatus message is sent whenever the active value changes for any of these parameters.

The RowDescription ('T') message carries new table OID and column number fields for each column of the described row. It also shows the format code for each column.

The CursorResponse ('P') message is no longer generated by the backend.

The NotificationResponse ('A') message has an additional string field, which can carry a “payload” string passed from the NOTIFY event sender.

The EmptyQueryResponse ('I') message used to include an empty string parameter; this has been removed.

Prev | Up | Next
53.9. Logical Replication Message Formats | Home | Chapter 54. PostgreSQL Coding Conventions


54.2. Reporting Errors Within the Server
Prev Up Chapter 54. PostgreSQL Coding Conventions Home Next

54.2. Reporting Errors Within the Server

Error, warning, and log messages generated within the server code should be created using ereport, or its older cousin elog. The use of this function is complex enough to require some explanation.

There are two required elements for every message: a severity level (ranging from DEBUG to PANIC) and a primary message text. In addition there are optional elements, the most common of which is an error identifier code that follows the SQL spec's SQLSTATE conventions. ereport itself is just a shell macro that exists mainly for the syntactic convenience of making message generation look like a single function call in the C source code. The only parameter accepted directly by ereport is the severity level. The primary message text and any optional message elements are generated by calling auxiliary functions, such as errmsg, within the ereport call.

A typical call to ereport might look like this:

ereport(ERROR,
        errcode(ERRCODE_DIVISION_BY_ZERO),
        errmsg("division by zero"));

This specifies error severity level ERROR (a run-of-the-mill error). The errcode call specifies the SQLSTATE error code using a macro defined in src/include/utils/errcodes.h. The errmsg call provides the primary message text.

You will also frequently see this older style, with an extra set of parentheses surrounding the auxiliary function calls:

ereport(ERROR,
        (errcode(ERRCODE_DIVISION_BY_ZERO),
         errmsg("division by zero")));

The extra parentheses were required before PostgreSQL version 12, but are now optional.

Here is a more complex example:

ereport(ERROR,
        errcode(ERRCODE_AMBIGUOUS_FUNCTION),
        errmsg("function %s is not unique",
               func_signature_string(funcname, nargs,
                                     NIL, actual_arg_types)),
        errhint("Unable to choose a best candidate function. "
                "You might need to add explicit typecasts."));

This illustrates the use of format codes to embed run-time values into a message text. Also, an optional “hint” message is provided. The auxiliary function calls can be written in any order, but conventionally errcode and errmsg appear first.

If the severity level is ERROR or higher, ereport aborts execution of the current query and does not return to the caller. If the severity level is lower than ERROR, ereport returns normally.

The available auxiliary routines for ereport are:

  • errcode(sqlerrcode) specifies the SQLSTATE error identifier code for the condition. If this routine is not called, the error identifier defaults to ERRCODE_INTERNAL_ERROR when the error severity level is ERROR or higher, ERRCODE_WARNING when the error level is WARNING, otherwise (for NOTICE and below) ERRCODE_SUCCESSFUL_COMPLETION. While these defaults are often convenient, always think whether they are appropriate before omitting the errcode() call.

  • errmsg(const char *msg, ...) specifies the primary error message text, and possibly run-time values to insert into it. Insertions are specified by sprintf-style format codes. In addition to the standard format codes accepted by sprintf, the format code %m can be used to insert the error message returned by strerror for the current value of errno. [16] %m does not require any corresponding entry in the parameter list for errmsg. Note that the message string will be run through gettext for possible localization before format codes are processed.

  • errmsg_internal(const char *msg, ...) is the same as errmsg, except that the message string will not be translated nor included in the internationalization message dictionary. This should be used for “cannot happen” cases that are probably not worth expending translation effort on.

  • errmsg_plural(const char *fmt_singular, const char *fmt_plural, unsigned long n, ...) is like errmsg, but with support for various plural forms of the message. fmt_singular is the English singular format, fmt_plural is the English plural format, n is the integer value that determines which plural form is needed, and the remaining arguments are formatted according to the selected format string. For more information see Section 55.2.2.

  • errdetail(const char *msg, ...) supplies an optional “detail” message; this is to be used when there is additional information that seems inappropriate to put in the primary message. The message string is processed in just the same way as for errmsg.

  • errdetail_internal(const char *msg, ...) is the same as errdetail, except that the message string will not be translated nor included in the internationalization message dictionary. This should be used for detail messages that are not worth expending translation effort on, for instance because they are too technical to be useful to most users.

  • errdetail_plural(const char *fmt_singular, const char *fmt_plural, unsigned long n, ...) is like errdetail, but with support for various plural forms of the message. For more information see Section 55.2.2.

  • errdetail_log(const char *msg, ...) is the same as errdetail except that this string goes only to the server log, never to the client. If both errdetail (or one of its equivalents above) and errdetail_log are used then one string goes to the client and the other to the log. This is useful for error details that are too security-sensitive or too bulky to include in the report sent to the client.

  • errdetail_log_plural(const char *fmt_singular, const char *fmt_plural, unsigned long n, ...) is like errdetail_log, but with support for various plural forms of the message. For more information see Section 55.2.2.

  • errhint(const char *msg, ...) supplies an optional “hint” message; this is to be used when offering suggestions about how to fix the problem, as opposed to factual details about what went wrong. The message string is processed in just the same way as for errmsg.

  • errhint_plural(const char *fmt_singular, const char *fmt_plural, unsigned long n, ...) is like errhint, but with support for various plural forms of the message. For more information see Section 55.2.2.

  • errcontext(const char *msg, ...) is not normally called directly from an ereport message site; rather it is used in error_context_stack callback functions to provide information about the context in which an error occurred, such as the current location in a PL function. The message string is processed in just the same way as for errmsg. Unlike the other auxiliary functions, this can be called more than once per ereport call; the successive strings thus supplied are concatenated with separating newlines.

  • errposition(int cursorpos) specifies the textual location of an error within a query string. Currently it is only useful for errors detected in the lexical and syntactic analysis phases of query processing.

  • errtable(Relation rel) specifies a relation whose name and schema name should be included as auxiliary fields in the error report.

  • errtablecol(Relation rel, int attnum) specifies a column whose name, table name, and schema name should be included as auxiliary fields in the error report.

  • errtableconstraint(Relation rel, const char *conname) specifies a table constraint whose name, table name, and schema name should be included as auxiliary fields in the error report. Indexes should be considered to be constraints for this purpose, whether or not they have an associated pg_constraint entry. Be careful to pass the underlying heap relation, not the index itself, as rel.

  • errdatatype(Oid datatypeOid) specifies a data type whose name and schema name should be included as auxiliary fields in the error report.

  • errdomainconstraint(Oid datatypeOid, const char *conname) specifies a domain constraint whose name, domain name, and schema name should be included as auxiliary fields in the error report.

  • errcode_for_file_access() is a convenience function that selects an appropriate SQLSTATE error identifier for a failure in a file-access-related system call. It uses the saved errno to determine which error code to generate. Usually this should be used in combination with %m in the primary error message text.

  • errcode_for_socket_access() is a convenience function that selects an appropriate SQLSTATE error identifier for a failure in a socket-related system call.

  • errhidestmt(bool hide_stmt) can be called to specify suppression of the STATEMENT: portion of a message in the postmaster log. Generally this is appropriate if the message text includes the current statement already.

  • errhidecontext(bool hide_ctx) can be called to specify suppression of the CONTEXT: portion of a message in the postmaster log. This should only be used for verbose debugging messages where the repeated inclusion of context would bloat the log too much.

Note

At most one of the functions errtable, errtablecol, errtableconstraint, errdatatype, or errdomainconstraint should be used in an ereport call. These functions exist to allow applications to extract the name of a database object associated with the error condition without having to examine the potentially-localized error message text. These functions should be used in error reports for which it's likely that applications would wish to have automatic error handling. As of PostgreSQL 9.3, complete coverage exists only for errors in SQLSTATE class 23 (integrity constraint violation), but this is likely to be expanded in future.

There is an older function elog that is still heavily used. An elog call:

elog(level, "format string", ...);

is exactly equivalent to:

ereport(level, errmsg_internal("format string", ...));

Notice that the SQLSTATE error code is always defaulted, and the message string is not subject to translation. Therefore, elog should be used only for internal errors and low-level debug logging. Any message that is likely to be of interest to ordinary users should go through ereport. Nonetheless, there are enough internal “cannot happen” error checks in the system that elog is still widely used; it is preferred for those messages for its notational simplicity.

Advice about writing good error messages can be found in Section 54.3.

[16] That is, the value that was current when the ereport call was reached; changes of errno within the auxiliary reporting routines will not affect it. That would not be true if you were to write strerror(errno) explicitly in errmsg's parameter list; accordingly, do not do so.

Prev | Up | Next
54.1. Formatting | Home | 54.3. Error Message Style Guide


54.3. Error Message Style Guide
Prev Up Chapter 54. PostgreSQL Coding Conventions Home Next

54.3. Error Message Style Guide

This style guide is offered in the hope of maintaining a consistent, user-friendly style throughout all the messages generated by PostgreSQL.

What Goes Where

The primary message should be short, factual, and avoid reference to implementation details such as specific function names. “Short” means “should fit on one line under normal conditions”. Use a detail message if needed to keep the primary message short, or if you feel a need to mention implementation details such as the particular system call that failed. Both primary and detail messages should be factual. Use a hint message for suggestions about what to do to fix the problem, especially if the suggestion might not always be applicable.

For example, instead of:

IpcMemoryCreate: shmget(key=%d, size=%u, 0%o) failed: %m
(plus a long addendum that is basically a hint)

write:

Primary:    could not create shared memory segment: %m
Detail:     Failed syscall was shmget(key=%d, size=%u, 0%o).
Hint:       the addendum

Rationale: keeping the primary message short helps keep it to the point, and lets clients lay out screen space on the assumption that one line is enough for error messages. Detail and hint messages can be relegated to a verbose mode, or perhaps a pop-up error-details window. Also, details and hints would normally be suppressed from the server log to save space. Reference to implementation details is best avoided since users aren't expected to know the details.

Formatting

Don't put any specific assumptions about formatting into the message texts. Expect clients and the server log to wrap lines to fit their own needs. In long messages, newline characters (\n) can be used to indicate suggested paragraph breaks. Don't end a message with a newline. Don't use tabs or other formatting characters. (In error context displays, newlines are automatically added to separate levels of context such as function calls.)

Rationale: Messages are not necessarily displayed on terminal-type displays. In GUI displays or browsers these formatting instructions are at best ignored.

Quotation Marks

English text should use double quotes when quoting is appropriate. Text in other languages should consistently use one kind of quotes that is consistent with publishing customs and computer output of other programs.

Rationale: The choice of double quotes over single quotes is somewhat arbitrary, but tends to be the preferred use. Some have suggested choosing the kind of quotes depending on the type of object according to SQL conventions (namely, strings single quoted, identifiers double quoted). But this is a language-internal technical issue that many users aren't even familiar with, it won't scale to other kinds of quoted terms, it doesn't translate to other languages, and it's pretty pointless, too.

Use of Quotes

Always use quotes to delimit file names, user-supplied identifiers, and other variables that might contain words. Do not use them to mark up variables that will not contain words (for example, operator names).

There are functions in the backend that will double-quote their own output as needed (for example, format_type_be()). Do not put additional quotes around the output of such functions.

Rationale: Objects can have names that create ambiguity when embedded in a message. Be consistent about denoting where a plugged-in name starts and ends. But don't clutter messages with unnecessary or duplicate quote marks.

Grammar and Punctuation

The rules are different for primary error messages and for detail/hint messages:

Primary error messages: Do not capitalize the first letter. Do not end a message with a period. Do not even think about ending a message with an exclamation point.

Detail and hint messages: Use complete sentences, and end each with a period. Capitalize the first word of sentences. Put two spaces after the period if another sentence follows (for English text; might be inappropriate in other languages).

Error context strings: Do not capitalize the first letter and do not end the string with a period. Context strings should normally not be complete sentences.

Rationale: Avoiding punctuation makes it easier for client applications to embed the message into a variety of grammatical contexts. Often, primary messages are not grammatically complete sentences anyway. (And if they're long enough to be more than one sentence, they should be split into primary and detail parts.) However, detail and hint messages are longer and might need to include multiple sentences. For consistency, they should follow complete-sentence style even when there's only one sentence.

Upper Case vs. Lower Case

Use lower case for message wording, including the first letter of a primary error message. Use upper case for SQL commands and key words if they appear in the message.

Rationale: It's easier to make everything look more consistent this way, since some messages are complete sentences and some not.

Avoid Passive Voice

Use the active voice. Use complete sentences when there is an acting subject (“A could not do B”). Use telegram style without subject if the subject would be the program itself; do not use “I” for the program.

Rationale: The program is not human. Don't pretend otherwise.

Present vs. Past Tense

Use past tense if an attempt to do something failed, but could perhaps succeed next time (perhaps after fixing some problem). Use present tense if the failure is certainly permanent.

There is a nontrivial semantic difference between sentences of the form:

could not open file "%s": %m

and:

cannot open file "%s"

The first one means that the attempt to open the file failed. The message should give a reason, such as “disk full” or “file doesn't exist”. The past tense is appropriate because next time the disk might not be full anymore or the file in question might exist.

The second form indicates that the functionality of opening the named file does not exist at all in the program, or that it's conceptually impossible. The present tense is appropriate because the condition will persist indefinitely.

Rationale: Granted, the average user will not be able to draw great conclusions merely from the tense of the message, but since the language provides us with a grammar we should use it correctly.

Type of the Object

When citing the name of an object, state what kind of object it is.

Rationale: Otherwise no one will know what “foo.bar.baz” refers to.

Brackets

Square brackets are only to be used (1) in command synopses to denote optional arguments, or (2) to denote an array subscript.

Rationale: Anything else does not correspond to widely-known customary usage and will confuse people.

Assembling Error Messages

When a message includes text that is generated elsewhere, embed it in this style:

could not open file %s: %m

Rationale: It would be difficult to account for all possible error codes to paste this into a single smooth sentence, so some sort of punctuation is needed. Putting the embedded text in parentheses has also been suggested, but it's unnatural if the embedded text is likely to be the most important part of the message, as is often the case.

Reasons for Errors

Messages should always state the reason why an error occurred. For example:

BAD:    could not open file %s
BETTER: could not open file %s (I/O failure)

If no reason is known you better fix the code.

Function Names

Don't include the name of the reporting routine in the error text. We have other mechanisms for finding that out when needed, and for most users it's not helpful information. If the error text doesn't make as much sense without the function name, reword it.

BAD:    pg_strtoint32: error in "z": cannot parse "z"
BETTER: invalid input syntax for type integer: "z"

Avoid mentioning called function names, either; instead say what the code was trying to do:

BAD:    open() failed: %m
BETTER: could not open file %s: %m

If it really seems necessary, mention the system call in the detail message. (In some cases, providing the actual values passed to the system call might be appropriate information for the detail message.)

Rationale: Users don't know what all those functions do.

Tricky Words to Avoid

Unable. “Unable” is nearly the passive voice. Better use “cannot” or “could not”, as appropriate.

Bad. Error messages like “bad result” are really hard to interpret intelligently. It's better to write why the result is “bad”, e.g., “invalid format”.

Illegal. “Illegal” stands for a violation of the law, the rest is “invalid”. Better yet, say why it's invalid.

Unknown. Try to avoid “unknown”. Consider “error: unknown response”. If you don't know what the response is, how do you know it's erroneous? “Unrecognized” is often a better choice. Also, be sure to include the value being complained of.

BAD:    unknown node type
BETTER: unrecognized node type: 42

Find vs. Exists. If the program uses a nontrivial algorithm to locate a resource (e.g., a path search) and that algorithm fails, it is fair to say that the program couldn't “find” the resource. If, on the other hand, the expected location of the resource is known but the program cannot access it there then say that the resource doesn't “exist”. Using “find” in this case sounds weak and confuses the issue.

May vs. Can vs. Might. “May” suggests permission (e.g., "You may borrow my rake."), and has little use in documentation or error messages. “Can” suggests ability (e.g., "I can lift that log."), and “might” suggests possibility (e.g., "It might rain today."). Using the proper word clarifies meaning and assists translation.

Contractions. Avoid contractions, like “can't”; use “cannot” instead.

Non-negative. Avoid “non-negative” as it is ambiguous about whether it accepts zero. It's better to use “greater than zero” or “greater than or equal to zero”.

Proper Spelling

Spell out words in full. For instance, avoid:

  • spec

  • stats

  • parens

  • auth

  • xact

Rationale: This will improve consistency.

Localization

Keep in mind that error message texts need to be translated into other languages. Follow the guidelines in Section 55.2.2 to avoid making life difficult for translators.

Prev | Up | Next
54.2. Reporting Errors Within the Server | Home | 54.4. Miscellaneous Coding Conventions


Chapter 55. Native Language Support
Prev Up Part VII. Internals Home Next

Chapter 55. Native Language Support

Table of Contents

55.1. For the Translator
55.1.1. Requirements
55.1.2. Concepts
55.1.3. Creating and Maintaining Message Catalogs
55.1.4. Editing the PO Files
55.2. For the Programmer
55.2.1. Mechanics
55.2.2. Message-Writing Guidelines
Prev | Up | Next
54.4. Miscellaneous Coding Conventions | Home | 55.1. For the Translator


55.1. For the Translator
Prev Up Chapter 55. Native Language Support Home Next

55.1. For the Translator

55.1.1. Requirements
55.1.2. Concepts
55.1.3. Creating and Maintaining Message Catalogs
55.1.4. Editing the PO Files

PostgreSQL programs (server and client) can issue their messages in your favorite language — if the messages have been translated. Creating and maintaining translated message sets needs the help of people who speak their own language well and want to contribute to the PostgreSQL effort. You do not have to be a programmer at all to do this. This section explains how to help.

55.1.1. Requirements

We won't judge your language skills — this section is about software tools. Theoretically, you only need a text editor. But this is only in the unlikely event that you do not want to try out your translated messages. When you configure your source tree, be sure to use the --enable-nls option. This will also check for the libintl library and the msgfmt program, which all end users will need anyway. To try out your work, follow the applicable portions of the installation instructions.

If you want to start a new translation effort or want to do a message catalog merge (described later), you will need the programs xgettext and msgmerge, respectively, in a GNU-compatible implementation. Later, we will try to arrange it so that if you use a packaged source distribution, you won't need xgettext. (If working from Git, you will still need it.) GNU Gettext 0.10.36 or later is currently recommended.

Your local gettext implementation should come with its own documentation. Some of that is probably duplicated in what follows, but for additional details you should look there.

55.1.2. Concepts

The pairs of original (English) messages and their (possibly) translated equivalents are kept in message catalogs, one for each program (although related programs can share a message catalog) and for each target language. There are two file formats for message catalogs: The first is the “PO” file (for Portable Object), which is a plain text file with special syntax that translators edit. The second is the “MO” file (for Machine Object), which is a binary file generated from the respective PO file and is used while the internationalized program is run. Translators do not deal with MO files; in fact hardly anyone does.

The extension of the message catalog file is to no surprise either .po or .mo. The base name is either the name of the program it accompanies, or the language the file is for, depending on the situation. This is a bit confusing. Examples are psql.po (PO file for psql) or fr.mo (MO file in French).

The file format of the PO files is illustrated here:

# comment

msgid "original string"
msgstr "translated string"

msgid "more original"
msgstr "another translated"
"string can be broken up like this"

...

The msgid lines are extracted from the program source. (They need not be, but this is the most common way.) The msgstr lines are initially empty and are filled in with useful strings by the translator. The strings can contain C-style escape characters and can be continued across lines as illustrated. (The next line must start at the beginning of the line.)

The # character introduces a comment. If whitespace immediately follows the # character, then this is a comment maintained by the translator. There can also be automatic comments, which have a non-whitespace character immediately following the #. These are maintained by the various tools that operate on the PO files and are intended to aid the translator.

#. automatic comment
#: filename.c:1023
#, flags, flags

The #. style comments are extracted from the source file where the message is used. Possibly the programmer has inserted information for the translator, such as about expected alignment. The #: comments indicate the exact locations where the message is used in the source. The translator need not look at the program source, but can if there is doubt about the correct translation. The #, comments contain flags that describe the message in some way. There are currently two flags: fuzzy is set if the message has possibly been outdated because of changes in the program source. The translator can then verify this and possibly remove the fuzzy flag. Note that fuzzy messages are not made available to the end user. The other flag is c-format, which indicates that the message is a printf-style format template. This means that the translation should also be a format string with the same number and type of placeholders. There are tools that can verify this, which key off the c-format flag.

55.1.3. Creating and Maintaining Message Catalogs

OK, so how does one create a “blank” message catalog? First, go into the directory that contains the program whose messages you want to translate. If there is a file nls.mk, then this program has been prepared for translation.

If there are already some .po files, then someone has already done some translation work. The files are named language.po, where language is the ISO 639-1 two-letter language code (in lower case), e.g., fr.po for French. If there is really a need for more than one translation effort per language then the files can also be named language_region.po where region is the ISO 3166-1 two-letter country code (in upper case), e.g., pt_BR.po for Portuguese in Brazil. If you find the language you wanted you can just start working on that file.

If you need to start a new translation effort, then first run the command:

make init-po

This will create a file progname.pot. (.pot to distinguish it from PO files that are “in production”. The T stands for “template”.) Copy this file to language.po and edit it. To make it known that the new language is available, also edit the file nls.mk and add the language (or language and country) code to the line that looks like:

AVAIL_LANGUAGES := de fr

(Other languages can appear, of course.)

As the underlying program or library changes, messages might be changed or added by the programmers. In this case you do not need to start from scratch. Instead, run the command:

make update-po

which will create a new blank message catalog file (the pot file you started with) and will merge it with the existing PO files. If the merge algorithm is not sure about a particular message it marks it “fuzzy” as explained above. The new PO file is saved with a .po.new extension.

55.1.4. Editing the PO Files

The PO files can be edited with a regular text editor. The translator should only change the area between the quotes after the msgstr directive, add comments, and alter the fuzzy flag. There is (unsurprisingly) a PO mode for Emacs, which I find quite useful.

The PO files need not be completely filled in. The software will automatically fall back to the original string if no translation (or an empty translation) is available. It is no problem to submit incomplete translations for inclusions in the source tree; that gives room for other people to pick up your work. However, you are encouraged to give priority to removing fuzzy entries after doing a merge. Remember that fuzzy entries will not be installed; they only serve as reference for what might be the right translation.

Here are some things to keep in mind while editing the translations:

  • Make sure that if the original ends with a newline, the translation does, too. Similarly for tabs, etc.

  • If the original is a printf format string, the translation also needs to be. The translation also needs to have the same format specifiers in the same order. Sometimes the natural rules of the language make this impossible or at least awkward. In that case you can modify the format specifiers like this:

    msgstr "Die Datei %2$s hat %1$u Zeichen."

    Then the first placeholder will actually use the second argument from the list. The digits$ needs to follow the % immediately, before any other format manipulators. (This feature really exists in the printf family of functions. You might not have heard of it before because there is little use for it outside of message internationalization.)

  • If the original string contains a linguistic mistake, report that (or fix it yourself in the program source) and translate normally. The corrected string can be merged in when the program sources have been updated. If the original string contains a factual mistake, report that (or fix it yourself) and do not translate it. Instead, you can mark the string with a comment in the PO file.

  • Maintain the style and tone of the original string. Specifically, messages that are not sentences (cannot open file %s) should probably not start with a capital letter (if your language distinguishes letter case) or end with a period (if your language uses punctuation marks). It might help to read Section 54.3.

  • If you don't know what a message means, or if it is ambiguous, ask on the developers' mailing list. Chances are that English speaking end users might also not understand it or find it ambiguous, so it's best to improve the message.

Prev | Up | Next
Chapter 55. Native Language Support | Home | 55.2. For the Programmer


55.2. For the Programmer
Prev Up Chapter 55. Native Language Support Home Next

55.2. For the Programmer

55.2.1. Mechanics
55.2.2. Message-Writing Guidelines

55.2.1. Mechanics

This section describes how to implement native language support in a program or library that is part of the PostgreSQL distribution. Currently, it only applies to C programs.

Adding NLS Support to a Program

  1. Insert this code into the start-up sequence of the program:

    #ifdef ENABLE_NLS
#include <locale.h>
#endif

...

#ifdef ENABLE_NLS
setlocale(LC_ALL, "");
bindtextdomain("progname", LOCALEDIR);
textdomain("progname");
#endif

    (The progname can actually be chosen freely.)

  2. Wherever a message that is a candidate for translation is found, a call to gettext() needs to be inserted. E.g.:

    fprintf(stderr, "panic level %d\n", lvl);

    would be changed to:

    fprintf(stderr, gettext("panic level %d\n"), lvl);

    (gettext is defined as a no-op if NLS support is not configured.)

    This tends to add a lot of clutter. One common shortcut is to use:

    #define _(x) gettext(x)

    Another solution is feasible if the program does much of its communication through one or a few functions, such as ereport() in the backend. Then you make this function call gettext internally on all input strings.

  3. Add a file nls.mk in the directory with the program sources. This file will be read as a makefile. The following variable assignments need to be made here:

    CATALOG_NAME

    The program name, as provided in the textdomain() call.

    AVAIL_LANGUAGES

    List of provided translations — initially empty.

    GETTEXT_FILES

    List of files that contain translatable strings, i.e., those marked with gettext or an alternative solution. Eventually, this will include nearly all source files of the program. If this list gets too long you can make the first “file” be a + and the second word be a file that contains one file name per line.

    GETTEXT_TRIGGERS

    The tools that generate message catalogs for the translators to work on need to know what function calls contain translatable strings. By default, only gettext() calls are known. If you used _ or other identifiers you need to list them here. If the translatable string is not the first argument, the item needs to be of the form func:2 (for the second argument). If you have a function that supports pluralized messages, the item should look like func:1,2 (identifying the singular and plural message arguments).

The build system will automatically take care of building and installing the message catalogs.

55.2.2. Message-Writing Guidelines

Here are some guidelines for writing messages that are easily translatable.

  • Do not construct sentences at run-time, like:

    printf("Files were %s.\n", flag ? "copied" : "removed");

    The word order within the sentence might be different in other languages. Also, even if you remember to call gettext() on each fragment, the fragments might not translate well separately. It's better to duplicate a little code so that each message to be translated is a coherent whole. Only numbers, file names, and such-like run-time variables should be inserted at run time into a message text.

  • For similar reasons, this won't work:

    printf("copied %d file%s", n, n!=1 ? "s" : "");

    because it assumes how the plural is formed. If you figured you could solve it like this:

    if (n==1)
    printf("copied 1 file");
else
    printf("copied %d files", n):

    then be disappointed. Some languages have more than two forms, with some peculiar rules. It's often best to design the message to avoid the issue altogether, for instance like this:

    printf("number of copied files: %d", n);

    If you really want to construct a properly pluralized message, there is support for this, but it's a bit awkward. When generating a primary or detail error message in ereport(), you can write something like this:

    errmsg_plural("copied %d file",
              "copied %d files",
              n,
              n)

    The first argument is the format string appropriate for English singular form, the second is the format string appropriate for English plural form, and the third is the integer control value that determines which plural form to use. Subsequent arguments are formatted per the format string as usual. (Normally, the pluralization control value will also be one of the values to be formatted, so it has to be written twice.) In English it only matters whether n is 1 or not 1, but in other languages there can be many different plural forms. The translator sees the two English forms as a group and has the opportunity to supply multiple substitute strings, with the appropriate one being selected based on the run-time value of n.

    If you need to pluralize a message that isn't going directly to an errmsg or errdetail report, you have to use the underlying function ngettext. See the gettext documentation.

  • If you want to communicate something to the translator, such as about how a message is intended to line up with other output, precede the occurrence of the string with a comment that starts with translator, e.g.:

    /* translator: This message is not what it seems to be. */

    These comments are copied to the message catalog files so that the translators can see them.

Prev | Up | Next
55.1. For the Translator | Home | Chapter 56. Writing a Procedural Language Handler


Chapter 56. Writing a Procedural Language Handler
Prev Up Part VII. Internals Home Next

Chapter 56. Writing a Procedural Language Handler

All calls to functions that are written in a language other than the current “version 1” interface for compiled languages (this includes functions in user-defined procedural languages and functions written in SQL) go through a call handler function for the specific language. It is the responsibility of the call handler to execute the function in a meaningful way, such as by interpreting the supplied source text. This chapter outlines how a new procedural language's call handler can be written.

The call handler for a procedural language is a “normal” function that must be written in a compiled language such as C, using the version-1 interface, and registered with PostgreSQL as taking no arguments and returning the type language_handler. This special pseudo-type identifies the function as a call handler and prevents it from being called directly in SQL commands. For more details on C language calling conventions and dynamic loading, see Section 38.10.

The call handler is called in the same way as any other function: It receives a pointer to a FunctionCallInfoBaseData struct containing argument values and information about the called function, and it is expected to return a Datum result (and possibly set the isnull field of the FunctionCallInfoBaseData structure, if it wishes to return an SQL null result). The difference between a call handler and an ordinary callee function is that the flinfo->fn_oid field of the FunctionCallInfoBaseData structure will contain the OID of the actual function to be called, not of the call handler itself. The call handler must use this field to determine which function to execute. Also, the passed argument list has been set up according to the declaration of the target function, not of the call handler.

It's up to the call handler to fetch the entry of the function from the pg_proc system catalog and to analyze the argument and return types of the called function. The AS clause from the CREATE FUNCTION command for the function will be found in the prosrc column of the pg_proc row. This is commonly source text in the procedural language, but in theory it could be something else, such as a path name to a file, or anything else that tells the call handler what to do in detail.

Often, the same function is called many times per SQL statement. A call handler can avoid repeated lookups of information about the called function by using the flinfo->fn_extra field. This will initially be NULL, but can be set by the call handler to point at information about the called function. On subsequent calls, if flinfo->fn_extra is already non-NULL then it can be used and the information lookup step skipped. The call handler must make sure that flinfo->fn_extra is made to point at memory that will live at least until the end of the current query, since an FmgrInfo data structure could be kept that long. One way to do this is to allocate the extra data in the memory context specified by flinfo->fn_mcxt; such data will normally have the same lifespan as the FmgrInfo itself. But the handler could also choose to use a longer-lived memory context so that it can cache function definition information across queries.

When a procedural-language function is invoked as a trigger, no arguments are passed in the usual way, but the FunctionCallInfoBaseData's context field points at a TriggerData structure, rather than being NULL as it is in a plain function call. A language handler should provide mechanisms for procedural-language functions to get at the trigger information.

A template for a procedural-language handler written as a C extension is provided in src/test/modules/plsample. This is a working sample demonstrating one way to create a procedural-language handler, process parameters, and return a value.

Although providing a call handler is sufficient to create a minimal procedural language, there are two other functions that can optionally be provided to make the language more convenient to use. These are a validator and an inline handler. A validator can be provided to allow language-specific checking to be done during CREATE FUNCTION. An inline handler can be provided to allow the language to support anonymous code blocks executed via the DO command.

If a validator is provided by a procedural language, it must be declared as a function taking a single parameter of type oid. The validator's result is ignored, so it is customarily declared to return void. The validator will be called at the end of a CREATE FUNCTION command that has created or updated a function written in the procedural language. The passed-in OID is the OID of the function's pg_proc row. The validator must fetch this row in the usual way, and do whatever checking is appropriate. First, call CheckFunctionValidatorAccess() to diagnose explicit calls to the validator that the user could not achieve through CREATE FUNCTION. Typical checks then include verifying that the function's argument and result types are supported by the language, and that the function's body is syntactically correct in the language. If the validator finds the function to be okay, it should just return. If it finds an error, it should report that via the normal ereport() error reporting mechanism. Throwing an error will force a transaction rollback and thus prevent the incorrect function definition from being committed.

Validator functions should typically honor the check_function_bodies parameter: if it is turned off then any expensive or context-sensitive checking should be skipped. If the language provides for code execution at compilation time, the validator must suppress checks that would induce such execution. In particular, this parameter is turned off by pg_dump so that it can load procedural language functions without worrying about side effects or dependencies of the function bodies on other database objects. (Because of this requirement, the call handler should avoid assuming that the validator has fully checked the function. The point of having a validator is not to let the call handler omit checks, but to notify the user immediately if there are obvious errors in a CREATE FUNCTION command.) While the choice of exactly what to check is mostly left to the discretion of the validator function, note that the core CREATE FUNCTION code only executes SET clauses attached to a function when check_function_bodies is on. Therefore, checks whose results might be affected by GUC parameters definitely should be skipped when check_function_bodies is off, to avoid false failures when reloading a dump.

If an inline handler is provided by a procedural language, it must be declared as a function taking a single parameter of type internal. The inline handler's result is ignored, so it is customarily declared to return void. The inline handler will be called when a DO statement is executed specifying the procedural language. The parameter actually passed is a pointer to an InlineCodeBlock struct, which contains information about the DO statement's parameters, in particular the text of the anonymous code block to be executed. The inline handler should execute this code and return.

It's recommended that you wrap all these function declarations, as well as the CREATE LANGUAGE command itself, into an extension so that a simple CREATE EXTENSION command is sufficient to install the language. See Section 38.17 for information about writing extensions.

The procedural languages included in the standard distribution are good references when trying to write your own language handler. Look into the src/pl subdirectory of the source tree. The CREATE LANGUAGE reference page also has some useful details.

Prev | Up | Next
55.2. For the Programmer | Home | Chapter 57. Writing a Foreign Data Wrapper


Chapter 57. Writing a Foreign Data Wrapper
Prev Up Part VII. Internals Home Next

Chapter 57. Writing a Foreign Data Wrapper

Table of Contents

57.1. Foreign Data Wrapper Functions
57.2. Foreign Data Wrapper Callback Routines
57.2.1. FDW Routines for Scanning Foreign Tables
57.2.2. FDW Routines for Scanning Foreign Joins
57.2.3. FDW Routines for Planning Post-Scan/Join Processing
57.2.4. FDW Routines for Updating Foreign Tables
57.2.5. FDW Routines for TRUNCATE
57.2.6. FDW Routines for Row Locking
57.2.7. FDW Routines for EXPLAIN
57.2.8. FDW Routines for ANALYZE
57.2.9. FDW Routines for IMPORT FOREIGN SCHEMA
57.2.10. FDW Routines for Parallel Execution
57.2.11. FDW Routines for Asynchronous Execution
57.2.12. FDW Routines for Reparameterization of Paths
57.3. Foreign Data Wrapper Helper Functions
57.4. Foreign Data Wrapper Query Planning
57.5. Row Locking in Foreign Data Wrappers

All operations on a foreign table are handled through its foreign data wrapper, which consists of a set of functions that the core server calls. The foreign data wrapper is responsible for fetching data from the remote data source and returning it to the PostgreSQL executor. If updating foreign tables is to be supported, the wrapper must handle that, too. This chapter outlines how to write a new foreign data wrapper.

The foreign data wrappers included in the standard distribution are good references when trying to write your own. Look into the contrib subdirectory of the source tree. The CREATE FOREIGN DATA WRAPPER reference page also has some useful details.

Note

The SQL standard specifies an interface for writing foreign data wrappers. However, PostgreSQL does not implement that API, because the effort to accommodate it into PostgreSQL would be large, and the standard API hasn't gained wide adoption anyway.

Prev | Up | Next
Chapter 56. Writing a Procedural Language Handler | Home | 57.1. Foreign Data Wrapper Functions


57.1. Foreign Data Wrapper Functions
Prev Up Chapter 57. Writing a Foreign Data Wrapper Home Next

57.1. Foreign Data Wrapper Functions

The FDW author needs to implement a handler function, and optionally a validator function. Both functions must be written in a compiled language such as C, using the version-1 interface. For details on C language calling conventions and dynamic loading, see Section 38.10.

The handler function simply returns a struct of function pointers to callback functions that will be called by the planner, executor, and various maintenance commands. Most of the effort in writing an FDW is in implementing these callback functions. The handler function must be registered with PostgreSQL as taking no arguments and returning the special pseudo-type fdw_handler. The callback functions are plain C functions and are not visible or callable at the SQL level. The callback functions are described in Section 57.2.

The validator function is responsible for validating options given in CREATE and ALTER commands for its foreign data wrapper, as well as foreign servers, user mappings, and foreign tables using the wrapper. The validator function must be registered as taking two arguments, a text array containing the options to be validated, and an OID representing the type of object the options are associated with (in the form of the OID of the system catalog the object would be stored in, either ForeignDataWrapperRelationId, ForeignServerRelationId, UserMappingRelationId, or ForeignTableRelationId). If no validator function is supplied, options are not checked at object creation time or object alteration time.

Prev | Up | Next
Chapter 57. Writing a Foreign Data Wrapper | Home | 57.2. Foreign Data Wrapper Callback Routines


57.2. Foreign Data Wrapper Callback Routines
Prev Up Chapter 57. Writing a Foreign Data Wrapper Home Next

57.2. Foreign Data Wrapper Callback Routines

57.2.1. FDW Routines for Scanning Foreign Tables
57.2.2. FDW Routines for Scanning Foreign Joins
57.2.3. FDW Routines for Planning Post-Scan/Join Processing
57.2.4. FDW Routines for Updating Foreign Tables
57.2.5. FDW Routines for TRUNCATE
57.2.6. FDW Routines for Row Locking
57.2.7. FDW Routines for EXPLAIN
57.2.8. FDW Routines for ANALYZE
57.2.9. FDW Routines for IMPORT FOREIGN SCHEMA
57.2.10. FDW Routines for Parallel Execution
57.2.11. FDW Routines for Asynchronous Execution
57.2.12. FDW Routines for Reparameterization of Paths

The FDW handler function returns a palloc'd FdwRoutine struct containing pointers to the callback functions described below. The scan-related functions are required, the rest are optional.

The FdwRoutine struct type is declared in src/include/foreign/fdwapi.h, which see for additional details.

57.2.1. FDW Routines for Scanning Foreign Tables

void
GetForeignRelSize(PlannerInfo *root,
                  RelOptInfo *baserel,
                  Oid foreigntableid);

Obtain relation size estimates for a foreign table. This is called at the beginning of planning for a query that scans a foreign table. root is the planner's global information about the query; baserel is the planner's information about this table; and foreigntableid is the pg_class OID of the foreign table. (foreigntableid could be obtained from the planner data structures, but it's passed explicitly to save effort.)

This function should update baserel->rows to be the expected number of rows returned by the table scan, after accounting for the filtering done by the restriction quals. The initial value of baserel->rows is just a constant default estimate, which should be replaced if at all possible. The function may also choose to update baserel->width if it can compute a better estimate of the average result row width. (The initial value is based on column data types and on column average-width values measured by the last ANALYZE.) Also, this function may update baserel->tuples if it can compute a better estimate of the foreign table's total row count. (The initial value is from pg_class.reltuples which represents the total row count seen by the last ANALYZE; it will be -1 if no ANALYZE has been done on this foreign table.)

See Section 57.4 for additional information.

void
GetForeignPaths(PlannerInfo *root,
                RelOptInfo *baserel,
                Oid foreigntableid);

Create possible access paths for a scan on a foreign table. This is called during query planning. The parameters are the same as for GetForeignRelSize, which has already been called.

This function must generate at least one access path (ForeignPath node) for a scan on the foreign table and must call add_path to add each such path to baserel->pathlist. It's recommended to use create_foreignscan_path to build the ForeignPath nodes. The function can generate multiple access paths, e.g., a path which has valid pathkeys to represent a pre-sorted result. Each access path must contain cost estimates, and can contain any FDW-private information that is needed to identify the specific scan method intended.

See Section 57.4 for additional information.

ForeignScan *
GetForeignPlan(PlannerInfo *root,
               RelOptInfo *baserel,
               Oid foreigntableid,
               ForeignPath *best_path,
               List *tlist,
               List *scan_clauses,
               Plan *outer_plan);

Create a ForeignScan plan node from the selected foreign access path. This is called at the end of query planning. The parameters are as for GetForeignRelSize, plus the selected ForeignPath (previously produced by GetForeignPaths, GetForeignJoinPaths, or GetForeignUpperPaths), the target list to be emitted by the plan node, the restriction clauses to be enforced by the plan node, and the outer subplan of the ForeignScan, which is used for rechecks performed by RecheckForeignScan. (If the path is for a join rather than a base relation, foreigntableid is InvalidOid.)

This function must create and return a ForeignScan plan node; it's recommended to use make_foreignscan to build the ForeignScan node.

See Section 57.4 for additional information.

void
BeginForeignScan(ForeignScanState *node,
                 int eflags);

Begin executing a foreign scan. This is called during executor startup. It should perform any initialization needed before the scan can start, but not start executing the actual scan (that should be done upon the first call to IterateForeignScan). The ForeignScanState node has already been created, but its fdw_state field is still NULL. Information about the table to scan is accessible through the ForeignScanState node (in particular, from the underlying ForeignScan plan node, which contains any FDW-private information provided by GetForeignPlan). eflags contains flag bits describing the executor's operating mode for this plan node.

Note that when (eflags & EXEC_FLAG_EXPLAIN_ONLY) is true, this function should not perform any externally-visible actions; it should only do the minimum required to make the node state valid for ExplainForeignScan and EndForeignScan.

TupleTableSlot *
IterateForeignScan(ForeignScanState *node);

Fetch one row from the foreign source, returning it in a tuple table slot (the node's ScanTupleSlot should be used for this purpose). Return NULL if no more rows are available. The tuple table slot infrastructure allows either a physical or virtual tuple to be returned; in most cases the latter choice is preferable from a performance standpoint. Note that this is called in a short-lived memory context that will be reset between invocations. Create a memory context in BeginForeignScan if you need longer-lived storage, or use the es_query_cxt of the node's EState.

The rows returned must match the fdw_scan_tlist target list if one was supplied, otherwise they must match the row type of the foreign table being scanned. If you choose to optimize away fetching columns that are not needed, you should insert nulls in those column positions, or else generate a fdw_scan_tlist list with those columns omitted.

Note that PostgreSQL's executor doesn't care whether the rows returned violate any constraints that were defined on the foreign table — but the planner does care, and may optimize queries incorrectly if there are rows visible in the foreign table that do not satisfy a declared constraint. If a constraint is violated when the user has declared that the constraint should hold true, it may be appropriate to raise an error (just as you would need to do in the case of a data type mismatch).

void
ReScanForeignScan(ForeignScanState *node);

Restart the scan from the beginning. Note that any parameters the scan depends on may have changed value, so the new scan does not necessarily return exactly the same rows.

void
EndForeignScan(ForeignScanState *node);

End the scan and release resources. It is normally not important to release palloc'd memory, but for example open files and connections to remote servers should be cleaned up.

57.2.2. FDW Routines for Scanning Foreign Joins

If an FDW supports performing foreign joins remotely (rather than by fetching both tables' data and doing the join locally), it should provide this callback function:

void
GetForeignJoinPaths(PlannerInfo *root,
                    RelOptInfo *joinrel,
                    RelOptInfo *outerrel,
                    RelOptInfo *innerrel,
                    JoinType jointype,
                    JoinPathExtraData *extra);

Create possible access paths for a join of two (or more) foreign tables that all belong to the same foreign server. This optional function is called during query planning. As with GetForeignPaths, this function should generate ForeignPath path(s) for the supplied joinrel (use create_foreign_join_path to build them), and call add_path to add these paths to the set of paths considered for the join. But unlike GetForeignPaths, it is not necessary that this function succeed in creating at least one path, since paths involving local joining are always possible.

Note that this function will be invoked repeatedly for the same join relation, with different combinations of inner and outer relations; it is the responsibility of the FDW to minimize duplicated work.

If a ForeignPath path is chosen for the join, it will represent the entire join process; paths generated for the component tables and subsidiary joins will not be used. Subsequent processing of the join path proceeds much as it does for a path scanning a single foreign table. One difference is that the scanrelid of the resulting ForeignScan plan node should be set to zero, since there is no single relation that it represents; instead, the fs_relids field of the ForeignScan node represents the set of relations that were joined. (The latter field is set up automatically by the core planner code, and need not be filled by the FDW.) Another difference is that, because the column list for a remote join cannot be found from the system catalogs, the FDW must fill fdw_scan_tlist with an appropriate list of TargetEntry nodes, representing the set of columns it will supply at run time in the tuples it returns.

See Section 57.4 for additional information.

57.2.3. FDW Routines for Planning Post-Scan/Join Processing

If an FDW supports performing remote post-scan/join processing, such as remote aggregation, it should provide this callback function:

void
GetForeignUpperPaths(PlannerInfo *root,
                     UpperRelationKind stage,
                     RelOptInfo *input_rel,
                     RelOptInfo *output_rel,
                     void *extra);

Create possible access paths for upper relation processing, which is the planner's term for all post-scan/join query processing, such as aggregation, window functions, sorting, and table updates. This optional function is called during query planning. Currently, it is called only if all base relation(s) involved in the query belong to the same FDW. This function should generate ForeignPath path(s) for any post-scan/join processing that the FDW knows how to perform remotely (use create_foreign_upper_path to build them), and call add_path to add these paths to the indicated upper relation. As with GetForeignJoinPaths, it is not necessary that this function succeed in creating any paths, since paths involving local processing are always possible.

The stage parameter identifies which post-scan/join step is currently being considered. output_rel is the upper relation that should receive paths representing computation of this step, and input_rel is the relation representing the input to this step. The extra parameter provides additional details, currently, it is set only for UPPERREL_PARTIAL_GROUP_AGG or UPPERREL_GROUP_AGG, in which case it points to a GroupPathExtraData structure; or for UPPERREL_FINAL, in which case it points to a FinalPathExtraData structure. (Note that ForeignPath paths added to output_rel would typically not have any direct dependency on paths of the input_rel, since their processing is expected to be done externally. However, examining paths previously generated for the previous processing step can be useful to avoid redundant planning work.)

See Section 57.4 for additional information.

57.2.4. FDW Routines for Updating Foreign Tables

If an FDW supports writable foreign tables, it should provide some or all of the following callback functions depending on the needs and capabilities of the FDW:

void
AddForeignUpdateTargets(PlannerInfo *root,
                        Index rtindex,
                        RangeTblEntry *target_rte,
                        Relation target_relation);

UPDATE and DELETE operations are performed against rows previously fetched by the table-scanning functions. The FDW may need extra information, such as a row ID or the values of primary-key columns, to ensure that it can identify the exact row to update or delete. To support that, this function can add extra hidden, or “junk”, target columns to the list of columns that are to be retrieved from the foreign table during an UPDATE or DELETE.

To do that, construct a Var representing an extra value you need, and pass it to add_row_identity_var, along with a name for the junk column. (You can do this more than once if several columns are needed.) You must choose a distinct junk column name for each different Var you need, except that Vars that are identical except for the varno field can and should share a column name. The core system uses the junk column names tableoid for a table's tableoid column, ctid or ctidN for ctid, wholerow for a whole-row Var marked with vartype = RECORD, and wholerowN for a whole-row Var with vartype equal to the table's declared row type. Re-use these names when you can (the planner will combine duplicate requests for identical junk columns). If you need another kind of junk column besides these, it might be wise to choose a name prefixed with your extension name, to avoid conflicts against other FDWs.

If the AddForeignUpdateTargets pointer is set to NULL, no extra target expressions are added. (This will make it impossible to implement DELETE operations, though UPDATE may still be feasible if the FDW relies on an unchanging primary key to identify rows.)

List *
PlanForeignModify(PlannerInfo *root,
                  ModifyTable *plan,
                  Index resultRelation,
                  int subplan_index);

Perform any additional planning actions needed for an insert, update, or delete on a foreign table. This function generates the FDW-private information that will be attached to the ModifyTable plan node that performs the update action. This private information must have the form of a List, and will be delivered to BeginForeignModify during the execution stage.

root is the planner's global information about the query. plan is the ModifyTable plan node, which is complete except for the fdwPrivLists field. resultRelation identifies the target foreign table by its range table index. subplan_index identifies which target of the ModifyTable plan node this is, counting from zero; use this if you want to index into per-target-relation substructures of the plan node.

See Section 57.4 for additional information.

If the PlanForeignModify pointer is set to NULL, no additional plan-time actions are taken, and the fdw_private list delivered to BeginForeignModify will be NIL.

void
BeginForeignModify(ModifyTableState *mtstate,
                   ResultRelInfo *rinfo,
                   List *fdw_private,
                   int subplan_index,
                   int eflags);

Begin executing a foreign table modification operation. This routine is called during executor startup. It should perform any initialization needed prior to the actual table modifications. Subsequently, ExecForeignInsert/ExecForeignBatchInsert, ExecForeignUpdate or ExecForeignDelete will be called for tuple(s) to be inserted, updated, or deleted.

mtstate is the overall state of the ModifyTable plan node being executed; global data about the plan and execution state is available via this structure. rinfo is the ResultRelInfo struct describing the target foreign table. (The ri_FdwState field of ResultRelInfo is available for the FDW to store any private state it needs for this operation.) fdw_private contains the private data generated by PlanForeignModify, if any. subplan_index identifies which target of the ModifyTable plan node this is. eflags contains flag bits describing the executor's operating mode for this plan node.

Note that when (eflags & EXEC_FLAG_EXPLAIN_ONLY) is true, this function should not perform any externally-visible actions; it should only do the minimum required to make the node state valid for ExplainForeignModify and EndForeignModify.

If the BeginForeignModify pointer is set to NULL, no action is taken during executor startup.

TupleTableSlot *
ExecForeignInsert(EState *estate,
                  ResultRelInfo *rinfo,
                  TupleTableSlot *slot,
                  TupleTableSlot *planSlot);

Insert one tuple into the foreign table. estate is global execution state for the query. rinfo is the ResultRelInfo struct describing the target foreign table. slot contains the tuple to be inserted; it will match the row-type definition of the foreign table. planSlot contains the tuple that was generated by the ModifyTable plan node's subplan; it differs from slot in possibly containing additional “junk” columns. (The planSlot is typically of little interest for INSERT cases, but is provided for completeness.)

The return value is either a slot containing the data that was actually inserted (this might differ from the data supplied, for example as a result of trigger actions), or NULL if no row was actually inserted (again, typically as a result of triggers). The passed-in slot can be re-used for this purpose.

The data in the returned slot is used only if the INSERT statement has a RETURNING clause or involves a view WITH CHECK OPTION; or if the foreign table has an AFTER ROW trigger. Triggers require all columns, but the FDW could choose to optimize away returning some or all columns depending on the contents of the RETURNING clause or WITH CHECK OPTION constraints. Regardless, some slot must be returned to indicate success, or the query's reported row count will be wrong.

If the ExecForeignInsert pointer is set to NULL, attempts to insert into the foreign table will fail with an error message.

Note that this function is also called when inserting routed tuples into a foreign-table partition or executing COPY FROM on a foreign table, in which case it is called in a different way than it is in the INSERT case. See the callback functions described below that allow the FDW to support that.

TupleTableSlot **
ExecForeignBatchInsert(EState *estate,
                       ResultRelInfo *rinfo,
                       TupleTableSlot **slots,
                       TupleTableSlot **planSlots,
                       int *numSlots);

Insert multiple tuples in bulk into the foreign table. The parameters are the same for ExecForeignInsert except slots and planSlots contain multiple tuples and *numSlots specifies the number of tuples in those arrays.

The return value is an array of slots containing the data that was actually inserted (this might differ from the data supplied, for example as a result of trigger actions.) The passed-in slots can be re-used for this purpose. The number of successfully inserted tuples is returned in *numSlots.

The data in the returned slot is used only if the INSERT statement involves a view WITH CHECK OPTION; or if the foreign table has an AFTER ROW trigger. Triggers require all columns, but the FDW could choose to optimize away returning some or all columns depending on the contents of the WITH CHECK OPTION constraints.

If the ExecForeignBatchInsert or GetForeignModifyBatchSize pointer is set to NULL, attempts to insert into the foreign table will use ExecForeignInsert. This function is not used if the INSERT has the RETURNING clause.

Note that this function is also called when inserting routed tuples into a foreign-table partition. See the callback functions described below that allow the FDW to support that.

int
GetForeignModifyBatchSize(ResultRelInfo *rinfo);

Report the maximum number of tuples that a single ExecForeignBatchInsert call can handle for the specified foreign table. The executor passes at most the given number of tuples to ExecForeignBatchInsert. rinfo is the ResultRelInfo struct describing the target foreign table. The FDW is expected to provide a foreign server and/or foreign table option for the user to set this value, or some hard-coded value.

If the ExecForeignBatchInsert or GetForeignModifyBatchSize pointer is set to NULL, attempts to insert into the foreign table will use ExecForeignInsert.

TupleTableSlot *
ExecForeignUpdate(EState *estate,
                  ResultRelInfo *rinfo,
                  TupleTableSlot *slot,
                  TupleTableSlot *planSlot);

Update one tuple in the foreign table. estate is global execution state for the query. rinfo is the ResultRelInfo struct describing the target foreign table. slot contains the new data for the tuple; it will match the row-type definition of the foreign table. planSlot contains the tuple that was generated by the ModifyTable plan node's subplan. Unlike slot, this tuple contains only the new values for columns changed by the query, so do not rely on attribute numbers of the foreign table to index into planSlot. Also, planSlot typically contains additional “junk” columns. In particular, any junk columns that were requested by AddForeignUpdateTargets will be available from this slot.

The return value is either a slot containing the row as it was actually updated (this might differ from the data supplied, for example as a result of trigger actions), or NULL if no row was actually updated (again, typically as a result of triggers). The passed-in slot can be re-used for this purpose.

The data in the returned slot is used only if the UPDATE statement has a RETURNING clause or involves a view WITH CHECK OPTION; or if the foreign table has an AFTER ROW trigger. Triggers require all columns, but the FDW could choose to optimize away returning some or all columns depending on the contents of the RETURNING clause or WITH CHECK OPTION constraints. Regardless, some slot must be returned to indicate success, or the query's reported row count will be wrong.

If the ExecForeignUpdate pointer is set to NULL, attempts to update the foreign table will fail with an error message.

TupleTableSlot *
ExecForeignDelete(EState *estate,
                  ResultRelInfo *rinfo,
                  TupleTableSlot *slot,
                  TupleTableSlot *planSlot);

Delete one tuple from the foreign table. estate is global execution state for the query. rinfo is the ResultRelInfo struct describing the target foreign table. slot contains nothing useful upon call, but can be used to hold the returned tuple. planSlot contains the tuple that was generated by the ModifyTable plan node's subplan; in particular, it will carry any junk columns that were requested by AddForeignUpdateTargets. The junk column(s) must be used to identify the tuple to be deleted.

The return value is either a slot containing the row that was deleted, or NULL if no row was deleted (typically as a result of triggers). The passed-in slot can be used to hold the tuple to be returned.

The data in the returned slot is used only if the DELETE query has a RETURNING clause or the foreign table has an AFTER ROW trigger. Triggers require all columns, but the FDW could choose to optimize away returning some or all columns depending on the contents of the RETURNING clause. Regardless, some slot must be returned to indicate success, or the query's reported row count will be wrong.

If the ExecForeignDelete pointer is set to NULL, attempts to delete from the foreign table will fail with an error message.

void
EndForeignModify(EState *estate,
                 ResultRelInfo *rinfo);

End the table update and release resources. It is normally not important to release palloc'd memory, but for example open files and connections to remote servers should be cleaned up.

If the EndForeignModify pointer is set to NULL, no action is taken during executor shutdown.

Tuples inserted into a partitioned table by INSERT or COPY FROM are routed to partitions. If an FDW supports routable foreign-table partitions, it should also provide the following callback functions. These functions are also called when COPY FROM is executed on a foreign table.

void
BeginForeignInsert(ModifyTableState *mtstate,
                   ResultRelInfo *rinfo);

Begin executing an insert operation on a foreign table. This routine is called right before the first tuple is inserted into the foreign table in both cases when it is the partition chosen for tuple routing and the target specified in a COPY FROM command. It should perform any initialization needed prior to the actual insertion. Subsequently, ExecForeignInsert or ExecForeignBatchInsert will be called for tuple(s) to be inserted into the foreign table.

mtstate is the overall state of the ModifyTable plan node being executed; global data about the plan and execution state is available via this structure. rinfo is the ResultRelInfo struct describing the target foreign table. (The ri_FdwState field of ResultRelInfo is available for the FDW to store any private state it needs for this operation.)

When this is called by a COPY FROM command, the plan-related global data in mtstate is not provided and the planSlot parameter of ExecForeignInsert subsequently called for each inserted tuple is NULL, whether the foreign table is the partition chosen for tuple routing or the target specified in the command.

If the BeginForeignInsert pointer is set to NULL, no action is taken for the initialization.

Note that if the FDW does not support routable foreign-table partitions and/or executing COPY FROM on foreign tables, this function or ExecForeignInsert/ExecForeignBatchInsert subsequently called must throw error as needed.

void
EndForeignInsert(EState *estate,
                 ResultRelInfo *rinfo);

End the insert operation and release resources. It is normally not important to release palloc'd memory, but for example open files and connections to remote servers should be cleaned up.

If the EndForeignInsert pointer is set to NULL, no action is taken for the termination.

int
IsForeignRelUpdatable(Relation rel);

Report which update operations the specified foreign table supports. The return value should be a bit mask of rule event numbers indicating which operations are supported by the foreign table, using the CmdType enumeration; that is, (1 << CMD_UPDATE) = 4 for UPDATE, (1 << CMD_INSERT) = 8 for INSERT, and (1 << CMD_DELETE) = 16 for DELETE.

If the IsForeignRelUpdatable pointer is set to NULL, foreign tables are assumed to be insertable, updatable, or deletable if the FDW provides ExecForeignInsert, ExecForeignUpdate, or ExecForeignDelete respectively. This function is only needed if the FDW supports some tables that are updatable and some that are not. (Even then, it's permissible to throw an error in the execution routine instead of checking in this function. However, this function is used to determine updatability for display in the information_schema views.)

Some inserts, updates, and deletes to foreign tables can be optimized by implementing an alternative set of interfaces. The ordinary interfaces for inserts, updates, and deletes fetch rows from the remote server and then modify those rows one at a time. In some cases, this row-by-row approach is necessary, but it can be inefficient. If it is possible for the foreign server to determine which rows should be modified without actually retrieving them, and if there are no local structures which would affect the operation (row-level local triggers, stored generated columns, or WITH CHECK OPTION constraints from parent views), then it is possible to arrange things so that the entire operation is performed on the remote server. The interfaces described below make this possible.

bool
PlanDirectModify(PlannerInfo *root,
                 ModifyTable *plan,
                 Index resultRelation,
                 int subplan_index);

Decide whether it is safe to execute a direct modification on the remote server. If so, return true after performing planning actions needed for that. Otherwise, return false. This optional function is called during query planning. If this function succeeds, BeginDirectModify, IterateDirectModify and EndDirectModify will be called at the execution stage, instead. Otherwise, the table modification will be executed using the table-updating functions described above. The parameters are the same as for PlanForeignModify.

To execute the direct modification on the remote server, this function must rewrite the target subplan with a ForeignScan plan node that executes the direct modification on the remote server. The operation and resultRelation fields of the ForeignScan must be set appropriately. operation must be set to the CmdType enumeration corresponding to the statement kind (that is, CMD_UPDATE for UPDATE, CMD_INSERT for INSERT, and CMD_DELETE for DELETE), and the resultRelation argument must be copied to the resultRelation field.

See Section 57.4 for additional information.

If the PlanDirectModify pointer is set to NULL, no attempts to execute a direct modification on the remote server are taken.

void
BeginDirectModify(ForeignScanState *node,
                  int eflags);

Prepare to execute a direct modification on the remote server. This is called during executor startup. It should perform any initialization needed prior to the direct modification (that should be done upon the first call to IterateDirectModify). The ForeignScanState node has already been created, but its fdw_state field is still NULL. Information about the table to modify is accessible through the ForeignScanState node (in particular, from the underlying ForeignScan plan node, which contains any FDW-private information provided by PlanDirectModify). eflags contains flag bits describing the executor's operating mode for this plan node.

Note that when (eflags & EXEC_FLAG_EXPLAIN_ONLY) is true, this function should not perform any externally-visible actions; it should only do the minimum required to make the node state valid for ExplainDirectModify and EndDirectModify.

If the BeginDirectModify pointer is set to NULL, no attempts to execute a direct modification on the remote server are taken.

TupleTableSlot *
IterateDirectModify(ForeignScanState *node);

When the INSERT, UPDATE or DELETE query doesn't have a RETURNING clause, just return NULL after a direct modification on the remote server. When the query has the clause, fetch one result containing the data needed for the RETURNING calculation, returning it in a tuple table slot (the node's ScanTupleSlot should be used for this purpose). The data that was actually inserted, updated or deleted must be stored in node->resultRelInfo->ri_projectReturning->pi_exprContext->ecxt_scantuple. Return NULL if no more rows are available. Note that this is called in a short-lived memory context that will be reset between invocations. Create a memory context in BeginDirectModify if you need longer-lived storage, or use the es_query_cxt of the node's EState.

The rows returned must match the fdw_scan_tlist target list if one was supplied, otherwise they must match the row type of the foreign table being updated. If you choose to optimize away fetching columns that are not needed for the RETURNING calculation, you should insert nulls in those column positions, or else generate a fdw_scan_tlist list with those columns omitted.

Whether the query has the clause or not, the query's reported row count must be incremented by the FDW itself. When the query doesn't have the clause, the FDW must also increment the row count for the ForeignScanState node in the EXPLAIN ANALYZE case.

If the IterateDirectModify pointer is set to NULL, no attempts to execute a direct modification on the remote server are taken.

void
EndDirectModify(ForeignScanState *node);

Clean up following a direct modification on the remote server. It is normally not important to release palloc'd memory, but for example open files and connections to the remote server should be cleaned up.

If the EndDirectModify pointer is set to NULL, no attempts to execute a direct modification on the remote server are taken.

57.2.5. FDW Routines for TRUNCATE

void
ExecForeignTruncate(List *rels,
                    DropBehavior behavior,
                    bool restart_seqs);

Truncate foreign tables. This function is called when TRUNCATE is executed on a foreign table. rels is a list of Relation data structures of foreign tables to truncate.

behavior is either DROP_RESTRICT or DROP_CASCADE indicating that the RESTRICT or CASCADE option was requested in the original TRUNCATE command, respectively.

If restart_seqs is true, the original TRUNCATE command requested the RESTART IDENTITY behavior, otherwise the CONTINUE IDENTITY behavior was requested.

Note that the ONLY options specified in the original TRUNCATE command are not passed to ExecForeignTruncate. This behavior is similar to the callback functions of SELECT, UPDATE and DELETE on a foreign table.

ExecForeignTruncate is invoked once per foreign server for which foreign tables are to be truncated. This means that all foreign tables included in rels must belong to the same server.

If the ExecForeignTruncate pointer is set to NULL, attempts to truncate foreign tables will fail with an error message.

57.2.6. FDW Routines for Row Locking

If an FDW wishes to support late row locking (as described in Section 57.5), it must provide the following callback functions:

RowMarkType
GetForeignRowMarkType(RangeTblEntry *rte,
                      LockClauseStrength strength);

Report which row-marking option to use for a foreign table. rte is the RangeTblEntry node for the table and strength describes the lock strength requested by the relevant FOR UPDATE/SHARE clause, if any. The result must be a member of the RowMarkType enum type.

This function is called during query planning for each foreign table that appears in an UPDATE, DELETE, or SELECT FOR UPDATE/SHARE query and is not the target of UPDATE or DELETE.

If the GetForeignRowMarkType pointer is set to NULL, the ROW_MARK_COPY option is always used. (This implies that RefetchForeignRow will never be called, so it need not be provided either.)

See Section 57.5 for more information.

void
RefetchForeignRow(EState *estate,
                  ExecRowMark *erm,
                  Datum rowid,
                  TupleTableSlot *slot,
                  bool *updated);

Re-fetch one tuple slot from the foreign table, after locking it if required. estate is global execution state for the query. erm is the ExecRowMark struct describing the target foreign table and the row lock type (if any) to acquire. rowid identifies the tuple to be fetched. slot contains nothing useful upon call, but can be used to hold the returned tuple. updated is an output parameter.

This function should store the tuple into the provided slot, or clear it if the row lock couldn't be obtained. The row lock type to acquire is defined by erm->markType, which is the value previously returned by GetForeignRowMarkType. (ROW_MARK_REFERENCE means to just re-fetch the tuple without acquiring any lock, and ROW_MARK_COPY will never be seen by this routine.)

In addition, *updated should be set to true if what was fetched was an updated version of the tuple rather than the same version previously obtained. (If the FDW cannot be sure about this, always returning true is recommended.)

Note that by default, failure to acquire a row lock should result in raising an error; returning with an empty slot is only appropriate if the SKIP LOCKED option is specified by erm->waitPolicy.

The rowid is the ctid value previously read for the row to be re-fetched. Although the rowid value is passed as a Datum, it can currently only be a tid. The function API is chosen in hopes that it may be possible to allow other data types for row IDs in future.

If the RefetchForeignRow pointer is set to NULL, attempts to re-fetch rows will fail with an error message.

See Section 57.5 for more information.

bool
RecheckForeignScan(ForeignScanState *node,
                   TupleTableSlot *slot);

Recheck that a previously-returned tuple still matches the relevant scan and join qualifiers, and possibly provide a modified version of the tuple. For foreign data wrappers which do not perform join pushdown, it will typically be more convenient to set this to NULL and instead set fdw_recheck_quals appropriately. When outer joins are pushed down, however, it isn't sufficient to reapply the checks relevant to all the base tables to the result tuple, even if all needed attributes are present, because failure to match some qualifier might result in some attributes going to NULL, rather than in no tuple being returned. RecheckForeignScan can recheck qualifiers and return true if they are still satisfied and false otherwise, but it can also store a replacement tuple into the supplied slot.

To implement join pushdown, a foreign data wrapper will typically construct an alternative local join plan which is used only for rechecks; this will become the outer subplan of the ForeignScan. When a recheck is required, this subplan can be executed and the resulting tuple can be stored in the slot. This plan need not be efficient since no base table will return more than one row; for example, it may implement all joins as nested loops. The function GetExistingLocalJoinPath may be used to search existing paths for a suitable local join path, which can be used as the alternative local join plan. GetExistingLocalJoinPath searches for an unparameterized path in the path list of the specified join relation. (If it does not find such a path, it returns NULL, in which case a foreign data wrapper may build the local path by itself or may choose not to create access paths for that join.)

57.2.7. FDW Routines for EXPLAIN

void
ExplainForeignScan(ForeignScanState *node,
                   ExplainState *es);

Print additional EXPLAIN output for a foreign table scan. This function can call ExplainPropertyText and related functions to add fields to the EXPLAIN output. The flag fields in es can be used to determine what to print, and the state of the ForeignScanState node can be inspected to provide run-time statistics in the EXPLAIN ANALYZE case.

If the ExplainForeignScan pointer is set to NULL, no additional information is printed during EXPLAIN.

void
ExplainForeignModify(ModifyTableState *mtstate,
                     ResultRelInfo *rinfo,
                     List *fdw_private,
                     int subplan_index,
                     struct ExplainState *es);

Print additional EXPLAIN output for a foreign table update. This function can call ExplainPropertyText and related functions to add fields to the EXPLAIN output. The flag fields in es can be used to determine what to print, and the state of the ModifyTableState node can be inspected to provide run-time statistics in the EXPLAIN ANALYZE case. The first four arguments are the same as for BeginForeignModify.

If the ExplainForeignModify pointer is set to NULL, no additional information is printed during EXPLAIN.

void
ExplainDirectModify(ForeignScanState *node,
                    ExplainState *es);

Print additional EXPLAIN output for a direct modification on the remote server. This function can call ExplainPropertyText and related functions to add fields to the EXPLAIN output. The flag fields in es can be used to determine what to print, and the state of the ForeignScanState node can be inspected to provide run-time statistics in the EXPLAIN ANALYZE case.

If the ExplainDirectModify pointer is set to NULL, no additional information is printed during EXPLAIN.

57.2.8. FDW Routines for ANALYZE

bool
AnalyzeForeignTable(Relation relation,
                    AcquireSampleRowsFunc *func,
                    BlockNumber *totalpages);

This function is called when ANALYZE is executed on a foreign table. If the FDW can collect statistics for this foreign table, it should return true, and provide a pointer to a function that will collect sample rows from the table in func, plus the estimated size of the table in pages in totalpages. Otherwise, return false.

If the FDW does not support collecting statistics for any tables, the AnalyzeForeignTable pointer can be set to NULL.

If provided, the sample collection function must have the signature

int
AcquireSampleRowsFunc(Relation relation,
                      int elevel,
                      HeapTuple *rows,
                      int targrows,
                      double *totalrows,
                      double *totaldeadrows);

A random sample of up to targrows rows should be collected from the table and stored into the caller-provided rows array. The actual number of rows collected must be returned. In addition, store estimates of the total numbers of live and dead rows in the table into the output parameters totalrows and totaldeadrows. (Set totaldeadrows to zero if the FDW does not have any concept of dead rows.)

57.2.9. FDW Routines for IMPORT FOREIGN SCHEMA

List *
ImportForeignSchema(ImportForeignSchemaStmt *stmt, Oid serverOid);

Obtain a list of foreign table creation commands. This function is called when executing IMPORT FOREIGN SCHEMA, and is passed the parse tree for that statement, as well as the OID of the foreign server to use. It should return a list of C strings, each of which must contain a CREATE FOREIGN TABLE command. These strings will be parsed and executed by the core server.

Within the ImportForeignSchemaStmt struct, remote_schema is the name of the remote schema from which tables are to be imported. list_type identifies how to filter table names: FDW_IMPORT_SCHEMA_ALL means that all tables in the remote schema should be imported (in this case table_list is empty), FDW_IMPORT_SCHEMA_LIMIT_TO means to include only tables listed in table_list, and FDW_IMPORT_SCHEMA_EXCEPT means to exclude the tables listed in table_list. options is a list of options used for the import process. The meanings of the options are up to the FDW. For example, an FDW could use an option to define whether the NOT NULL attributes of columns should be imported. These options need not have anything to do with those supported by the FDW as database object options.

The FDW may ignore the local_schema field of the ImportForeignSchemaStmt, because the core server will automatically insert that name into the parsed CREATE FOREIGN TABLE commands.

The FDW does not have to concern itself with implementing the filtering specified by list_type and table_list, either, as the core server will automatically skip any returned commands for tables excluded according to those options. However, it's often useful to avoid the work of creating commands for excluded tables in the first place. The function IsImportableForeignTable() may be useful to test whether a given foreign-table name will pass the filter.

If the FDW does not support importing table definitions, the ImportForeignSchema pointer can be set to NULL.

57.2.10. FDW Routines for Parallel Execution

A ForeignScan node can, optionally, support parallel execution. A parallel ForeignScan will be executed in multiple processes and must return each row exactly once across all cooperating processes. To do this, processes can coordinate through fixed-size chunks of dynamic shared memory. This shared memory is not guaranteed to be mapped at the same address in every process, so it must not contain pointers. The following functions are all optional, but most are required if parallel execution is to be supported.

bool
IsForeignScanParallelSafe(PlannerInfo *root, RelOptInfo *rel,
                          RangeTblEntry *rte);

Test whether a scan can be performed within a parallel worker. This function will only be called when the planner believes that a parallel plan might be possible, and should return true if it is safe for that scan to run within a parallel worker. This will generally not be the case if the remote data source has transaction semantics, unless the worker's connection to the data can somehow be made to share the same transaction context as the leader.

If this function is not defined, it is assumed that the scan must take place within the parallel leader. Note that returning true does not mean that the scan itself can be done in parallel, only that the scan can be performed within a parallel worker. Therefore, it can be useful to define this method even when parallel execution is not supported.

Size
EstimateDSMForeignScan(ForeignScanState *node, ParallelContext *pcxt);

Estimate the amount of dynamic shared memory that will be required for parallel operation. This may be higher than the amount that will actually be used, but it must not be lower. The return value is in bytes. This function is optional, and can be omitted if not needed; but if it is omitted, the next three functions must be omitted as well, because no shared memory will be allocated for the FDW's use.

void
InitializeDSMForeignScan(ForeignScanState *node, ParallelContext *pcxt,
                         void *coordinate);

Initialize the dynamic shared memory that will be required for parallel operation. coordinate points to a shared memory area of size equal to the return value of EstimateDSMForeignScan. This function is optional, and can be omitted if not needed.

void
ReInitializeDSMForeignScan(ForeignScanState *node, ParallelContext *pcxt,
                           void *coordinate);

Re-initialize the dynamic shared memory required for parallel operation when the foreign-scan plan node is about to be re-scanned. This function is optional, and can be omitted if not needed. Recommended practice is that this function reset only shared state, while the ReScanForeignScan function resets only local state. Currently, this function will be called before ReScanForeignScan, but it's best not to rely on that ordering.

void
InitializeWorkerForeignScan(ForeignScanState *node, shm_toc *toc,
                            void *coordinate);

Initialize a parallel worker's local state based on the shared state set up by the leader during InitializeDSMForeignScan. This function is optional, and can be omitted if not needed.

void
ShutdownForeignScan(ForeignScanState *node);

Release resources when it is anticipated the node will not be executed to completion. This is not called in all cases; sometimes, EndForeignScan may be called without this function having been called first. Since the DSM segment used by parallel query is destroyed just after this callback is invoked, foreign data wrappers that wish to take some action before the DSM segment goes away should implement this method.

57.2.11. FDW Routines for Asynchronous Execution

A ForeignScan node can, optionally, support asynchronous execution as described in src/backend/executor/README. The following functions are all optional, but are all required if asynchronous execution is to be supported.

bool
IsForeignPathAsyncCapable(ForeignPath *path);

Test whether a given ForeignPath path can scan the underlying foreign relation asynchronously. This function will only be called at the end of query planning when the given path is a direct child of an AppendPath path and when the planner believes that asynchronous execution improves performance, and should return true if the given path is able to scan the foreign relation asynchronously.

If this function is not defined, it is assumed that the given path scans the foreign relation using IterateForeignScan. (This implies that the callback functions described below will never be called, so they need not be provided either.)

void
ForeignAsyncRequest(AsyncRequest *areq);

Produce one tuple asynchronously from the ForeignScan node. areq is the AsyncRequest struct describing the ForeignScan node and the parent Append node that requested the tuple from it. This function should store the tuple into the slot specified by areq->result, and set areq->request_complete to true; or if it needs to wait on an event external to the core server such as network I/O, and cannot produce any tuple immediately, set the flag to false, and set areq->callback_pending to true for the ForeignScan node to get a callback from the callback functions described below. If no more tuples are available, set the slot to NULL or an empty slot, and the areq->request_complete flag to true. It's recommended to use ExecAsyncRequestDone or ExecAsyncRequestPending to set the output parameters in the areq.

void
ForeignAsyncConfigureWait(AsyncRequest *areq);

Configure a file descriptor event for which the ForeignScan node wishes to wait. This function will only be called when the ForeignScan node has the areq->callback_pending flag set, and should add the event to the as_eventset of the parent Append node described by the areq. See the comments for ExecAsyncConfigureWait in src/backend/executor/execAsync.c for additional information. When the file descriptor event occurs, ForeignAsyncNotify will be called.

void
ForeignAsyncNotify(AsyncRequest *areq);

Process a relevant event that has occurred, then produce one tuple asynchronously from the ForeignScan node. This function should set the output parameters in the areq in the same way as ForeignAsyncRequest.

57.2.12. FDW Routines for Reparameterization of Paths

List *
ReparameterizeForeignPathByChild(PlannerInfo *root, List *fdw_private,
                                 RelOptInfo *child_rel);

This function is called while converting a path parameterized by the top-most parent of the given child relation child_rel to be parameterized by the child relation. The function is used to reparameterize any paths or translate any expression nodes saved in the given fdw_private member of a ForeignPath. The callback may use reparameterize_path_by_child, adjust_appendrel_attrs or adjust_appendrel_attrs_multilevel as required.

Prev | Up | Next
57.1. Foreign Data Wrapper Functions | Home | 57.3. Foreign Data Wrapper Helper Functions


57.3. Foreign Data Wrapper Helper Functions
Prev Up Chapter 57. Writing a Foreign Data Wrapper Home Next

57.3. Foreign Data Wrapper Helper Functions

Several helper functions are exported from the core server so that authors of foreign data wrappers can get easy access to attributes of FDW-related objects, such as FDW options. To use any of these functions, you need to include the header file foreign/foreign.h in your source file. That header also defines the struct types that are returned by these functions.

ForeignDataWrapper *
GetForeignDataWrapperExtended(Oid fdwid, bits16 flags);

This function returns a ForeignDataWrapper object for the foreign-data wrapper with the given OID. A ForeignDataWrapper object contains properties of the FDW (see foreign/foreign.h for details). flags is a bitwise-or'd bit mask indicating an extra set of options. It can take the value FDW_MISSING_OK, in which case a NULL result is returned to the caller instead of an error for an undefined object.

ForeignDataWrapper *
GetForeignDataWrapper(Oid fdwid);

This function returns a ForeignDataWrapper object for the foreign-data wrapper with the given OID. A ForeignDataWrapper object contains properties of the FDW (see foreign/foreign.h for details).

ForeignServer *
GetForeignServerExtended(Oid serverid, bits16 flags);

This function returns a ForeignServer object for the foreign server with the given OID. A ForeignServer object contains properties of the server (see foreign/foreign.h for details). flags is a bitwise-or'd bit mask indicating an extra set of options. It can take the value FSV_MISSING_OK, in which case a NULL result is returned to the caller instead of an error for an undefined object.

ForeignServer *
GetForeignServer(Oid serverid);

This function returns a ForeignServer object for the foreign server with the given OID. A ForeignServer object contains properties of the server (see foreign/foreign.h for details).

UserMapping *
GetUserMapping(Oid userid, Oid serverid);

This function returns a UserMapping object for the user mapping of the given role on the given server. (If there is no mapping for the specific user, it will return the mapping for PUBLIC, or throw error if there is none.) A UserMapping object contains properties of the user mapping (see foreign/foreign.h for details).

ForeignTable *
GetForeignTable(Oid relid);

This function returns a ForeignTable object for the foreign table with the given OID. A ForeignTable object contains properties of the foreign table (see foreign/foreign.h for details).

List *
GetForeignColumnOptions(Oid relid, AttrNumber attnum);

This function returns the per-column FDW options for the column with the given foreign table OID and attribute number, in the form of a list of DefElem. NIL is returned if the column has no options.

Some object types have name-based lookup functions in addition to the OID-based ones:

ForeignDataWrapper *
GetForeignDataWrapperByName(const char *name, bool missing_ok);

This function returns a ForeignDataWrapper object for the foreign-data wrapper with the given name. If the wrapper is not found, return NULL if missing_ok is true, otherwise raise an error.

ForeignServer *
GetForeignServerByName(const char *name, bool missing_ok);

This function returns a ForeignServer object for the foreign server with the given name. If the server is not found, return NULL if missing_ok is true, otherwise raise an error.

Prev | Up | Next
57.2. Foreign Data Wrapper Callback Routines | Home | 57.4. Foreign Data Wrapper Query Planning


57.4. Foreign Data Wrapper Query Planning
Prev Up Chapter 57. Writing a Foreign Data Wrapper Home Next

57.4. Foreign Data Wrapper Query Planning

The FDW callback functions GetForeignRelSize, GetForeignPaths, GetForeignPlan, PlanForeignModify, GetForeignJoinPaths, GetForeignUpperPaths, and PlanDirectModify must fit into the workings of the PostgreSQL planner. Here are some notes about what they must do.

The information in root and baserel can be used to reduce the amount of information that has to be fetched from the foreign table (and therefore reduce the cost). baserel->baserestrictinfo is particularly interesting, as it contains restriction quals (WHERE clauses) that should be used to filter the rows to be fetched. (The FDW itself is not required to enforce these quals, as the core executor can check them instead.) baserel->reltarget->exprs can be used to determine which columns need to be fetched; but note that it only lists columns that have to be emitted by the ForeignScan plan node, not columns that are used in qual evaluation but not output by the query.

Various private fields are available for the FDW planning functions to keep information in. Generally, whatever you store in FDW private fields should be palloc'd, so that it will be reclaimed at the end of planning.

baserel->fdw_private is a void pointer that is available for FDW planning functions to store information relevant to the particular foreign table. The core planner does not touch it except to initialize it to NULL when the RelOptInfo node is created. It is useful for passing information forward from GetForeignRelSize to GetForeignPaths and/or GetForeignPaths to GetForeignPlan, thereby avoiding recalculation.

GetForeignPaths can identify the meaning of different access paths by storing private information in the fdw_private field of ForeignPath nodes. fdw_private is declared as a List pointer, but could actually contain anything since the core planner does not touch it. However, best practice is to use a representation that's dumpable by nodeToString, for use with debugging support available in the backend.

GetForeignPlan can examine the fdw_private field of the selected ForeignPath node, and can generate fdw_exprs and fdw_private lists to be placed in the ForeignScan plan node, where they will be available at execution time. Both of these lists must be represented in a form that copyObject knows how to copy. The fdw_private list has no other restrictions and is not interpreted by the core backend in any way. The fdw_exprs list, if not NIL, is expected to contain expression trees that are intended to be executed at run time. These trees will undergo post-processing by the planner to make them fully executable.

In GetForeignPlan, generally the passed-in target list can be copied into the plan node as-is. The passed scan_clauses list contains the same clauses as baserel->baserestrictinfo, but may be re-ordered for better execution efficiency. In simple cases the FDW can just strip RestrictInfo nodes from the scan_clauses list (using extract_actual_clauses) and put all the clauses into the plan node's qual list, which means that all the clauses will be checked by the executor at run time. More complex FDWs may be able to check some of the clauses internally, in which case those clauses can be removed from the plan node's qual list so that the executor doesn't waste time rechecking them.

As an example, the FDW might identify some restriction clauses of the form foreign_variable = sub_expression, which it determines can be executed on the remote server given the locally-evaluated value of the sub_expression. The actual identification of such a clause should happen during GetForeignPaths, since it would affect the cost estimate for the path. The path's fdw_private field would probably include a pointer to the identified clause's RestrictInfo node. Then GetForeignPlan would remove that clause from scan_clauses, but add the sub_expression to fdw_exprs to ensure that it gets massaged into executable form. It would probably also put control information into the plan node's fdw_private field to tell the execution functions what to do at run time. The query transmitted to the remote server would involve something like WHERE foreign_variable = $1, with the parameter value obtained at run time from evaluation of the fdw_exprs expression tree.

Any clauses removed from the plan node's qual list must instead be added to fdw_recheck_quals or rechecked by RecheckForeignScan in order to ensure correct behavior at the READ COMMITTED isolation level. When a concurrent update occurs for some other table involved in the query, the executor may need to verify that all of the original quals are still satisfied for the tuple, possibly against a different set of parameter values. Using fdw_recheck_quals is typically easier than implementing checks inside RecheckForeignScan, but this method will be insufficient when outer joins have been pushed down, since the join tuples in that case might have some fields go to NULL without rejecting the tuple entirely.

Another ForeignScan field that can be filled by FDWs is fdw_scan_tlist, which describes the tuples returned by the FDW for this plan node. For simple foreign table scans this can be set to NIL, implying that the returned tuples have the row type declared for the foreign table. A non-NIL value must be a target list (list of TargetEntrys) containing Vars and/or expressions representing the returned columns. This might be used, for example, to show that the FDW has omitted some columns that it noticed won't be needed for the query. Also, if the FDW can compute expressions used by the query more cheaply than can be done locally, it could add those expressions to fdw_scan_tlist. Note that join plans (created from paths made by GetForeignJoinPaths) must always supply fdw_scan_tlist to describe the set of columns they will return.

The FDW should always construct at least one path that depends only on the table's restriction clauses. In join queries, it might also choose to construct path(s) that depend on join clauses, for example foreign_variable = local_variable. Such clauses will not be found in baserel->baserestrictinfo but must be sought in the relation's join lists. A path using such a clause is called a “parameterized path”. It must identify the other relations used in the selected join clause(s) with a suitable value of param_info; use get_baserel_parampathinfo to compute that value. In GetForeignPlan, the local_variable portion of the join clause would be added to fdw_exprs, and then at run time the case works the same as for an ordinary restriction clause.

If an FDW supports remote joins, GetForeignJoinPaths should produce ForeignPaths for potential remote joins in much the same way as GetForeignPaths works for base tables. Information about the intended join can be passed forward to GetForeignPlan in the same ways described above. However, baserestrictinfo is not relevant for join relations; instead, the relevant join clauses for a particular join are passed to GetForeignJoinPaths as a separate parameter (extra->restrictlist).

An FDW might additionally support direct execution of some plan actions that are above the level of scans and joins, such as grouping or aggregation. To offer such options, the FDW should generate paths and insert them into the appropriate upper relation. For example, a path representing remote aggregation should be inserted into the UPPERREL_GROUP_AGG relation, using add_path. This path will be compared on a cost basis with local aggregation performed by reading a simple scan path for the foreign relation (note that such a path must also be supplied, else there will be an error at plan time). If the remote-aggregation path wins, which it usually would, it will be converted into a plan in the usual way, by calling GetForeignPlan. The recommended place to generate such paths is in the GetForeignUpperPaths callback function, which is called for each upper relation (i.e., each post-scan/join processing step), if all the base relations of the query come from the same FDW.

PlanForeignModify and the other callbacks described in Section 57.2.4 are designed around the assumption that the foreign relation will be scanned in the usual way and then individual row updates will be driven by a local ModifyTable plan node. This approach is necessary for the general case where an update requires reading local tables as well as foreign tables. However, if the operation could be executed entirely by the foreign server, the FDW could generate a path representing that and insert it into the UPPERREL_FINAL upper relation, where it would compete against the ModifyTable approach. This approach could also be used to implement remote SELECT FOR UPDATE, rather than using the row locking callbacks described in Section 57.2.6. Keep in mind that a path inserted into UPPERREL_FINAL is responsible for implementing all behavior of the query.

When planning an UPDATE or DELETE, PlanForeignModify and PlanDirectModify can look up the RelOptInfo struct for the foreign table and make use of the baserel->fdw_private data previously created by the scan-planning functions. However, in INSERT the target table is not scanned so there is no RelOptInfo for it. The List returned by PlanForeignModify has the same restrictions as the fdw_private list of a ForeignScan plan node, that is it must contain only structures that copyObject knows how to copy.

INSERT with an ON CONFLICT clause does not support specifying the conflict target, as unique constraints or exclusion constraints on remote tables are not locally known. This in turn implies that ON CONFLICT DO UPDATE is not supported, since the specification is mandatory there.

Prev | Up | Next
57.3. Foreign Data Wrapper Helper Functions | Home | 57.5. Row Locking in Foreign Data Wrappers


57.5. Row Locking in Foreign Data Wrappers
Prev Up Chapter 57. Writing a Foreign Data Wrapper Home Next

57.5. Row Locking in Foreign Data Wrappers

If an FDW's underlying storage mechanism has a concept of locking individual rows to prevent concurrent updates of those rows, it is usually worthwhile for the FDW to perform row-level locking with as close an approximation as practical to the semantics used in ordinary PostgreSQL tables. There are multiple considerations involved in this.

One key decision to be made is whether to perform early locking or late locking. In early locking, a row is locked when it is first retrieved from the underlying store, while in late locking, the row is locked only when it is known that it needs to be locked. (The difference arises because some rows may be discarded by locally-checked restriction or join conditions.) Early locking is much simpler and avoids extra round trips to a remote store, but it can cause locking of rows that need not have been locked, resulting in reduced concurrency or even unexpected deadlocks. Also, late locking is only possible if the row to be locked can be uniquely re-identified later. Preferably the row identifier should identify a specific version of the row, as PostgreSQL TIDs do.

By default, PostgreSQL ignores locking considerations when interfacing to FDWs, but an FDW can perform early locking without any explicit support from the core code. The API functions described in Section 57.2.6, which were added in PostgreSQL 9.5, allow an FDW to use late locking if it wishes.

An additional consideration is that in READ COMMITTED isolation mode, PostgreSQL may need to re-check restriction and join conditions against an updated version of some target tuple. Rechecking join conditions requires re-obtaining copies of the non-target rows that were previously joined to the target tuple. When working with standard PostgreSQL tables, this is done by including the TIDs of the non-target tables in the column list projected through the join, and then re-fetching non-target rows when required. This approach keeps the join data set compact, but it requires inexpensive re-fetch capability, as well as a TID that can uniquely identify the row version to be re-fetched. By default, therefore, the approach used with foreign tables is to include a copy of the entire row fetched from a foreign table in the column list projected through the join. This puts no special demands on the FDW but can result in reduced performance of merge and hash joins. An FDW that is capable of meeting the re-fetch requirements can choose to do it the first way.

For an UPDATE or DELETE on a foreign table, it is recommended that the ForeignScan operation on the target table perform early locking on the rows that it fetches, perhaps via the equivalent of SELECT FOR UPDATE. An FDW can detect whether a table is an UPDATE/DELETE target at plan time by comparing its relid to root->parse->resultRelation, or at execution time by using ExecRelationIsTargetRelation(). An alternative possibility is to perform late locking within the ExecForeignUpdate or ExecForeignDelete callback, but no special support is provided for this.

For foreign tables that are specified to be locked by a SELECT FOR UPDATE/SHARE command, the ForeignScan operation can again perform early locking by fetching tuples with the equivalent of SELECT FOR UPDATE/SHARE. To perform late locking instead, provide the callback functions defined in Section 57.2.6. In GetForeignRowMarkType, select rowmark option ROW_MARK_EXCLUSIVE, ROW_MARK_NOKEYEXCLUSIVE, ROW_MARK_SHARE, or ROW_MARK_KEYSHARE depending on the requested lock strength. (The core code will act the same regardless of which of these four options you choose.) Elsewhere, you can detect whether a foreign table was specified to be locked by this type of command by using get_plan_rowmark at plan time, or ExecFindRowMark at execution time; you must check not only whether a non-null rowmark struct is returned, but that its strength field is not LCS_NONE.

Lastly, for foreign tables that are used in an UPDATE, DELETE or SELECT FOR UPDATE/SHARE command but are not specified to be row-locked, you can override the default choice to copy entire rows by having GetForeignRowMarkType select option ROW_MARK_REFERENCE when it sees lock strength LCS_NONE. This will cause RefetchForeignRow to be called with that value for markType; it should then re-fetch the row without acquiring any new lock. (If you have a GetForeignRowMarkType function but don't wish to re-fetch unlocked rows, select option ROW_MARK_COPY for LCS_NONE.)

See src/include/nodes/lockoptions.h, the comments for RowMarkType and PlanRowMark in src/include/nodes/plannodes.h, and the comments for ExecRowMark in src/include/nodes/execnodes.h for additional information.

Prev | Up | Next
57.4. Foreign Data Wrapper Query Planning | Home | Chapter 58. Writing a Table Sampling Method


Chapter 59. Writing a Custom Scan Provider
Prev Up Part VII. Internals Home Next

Chapter 59. Writing a Custom Scan Provider

Table of Contents

59.1. Creating Custom Scan Paths
59.1.1. Custom Scan Path Callbacks
59.2. Creating Custom Scan Plans
59.2.1. Custom Scan Plan Callbacks
59.3. Executing Custom Scans
59.3.1. Custom Scan Execution Callbacks

PostgreSQL supports a set of experimental facilities which are intended to allow extension modules to add new scan types to the system. Unlike a foreign data wrapper, which is only responsible for knowing how to scan its own foreign tables, a custom scan provider can provide an alternative method of scanning any relation in the system. Typically, the motivation for writing a custom scan provider will be to allow the use of some optimization not supported by the core system, such as caching or some form of hardware acceleration. This chapter outlines how to write a new custom scan provider.

Implementing a new type of custom scan is a three-step process. First, during planning, it is necessary to generate access paths representing a scan using the proposed strategy. Second, if one of those access paths is selected by the planner as the optimal strategy for scanning a particular relation, the access path must be converted to a plan. Finally, it must be possible to execute the plan and generate the same results that would have been generated for any other access path targeting the same relation.

Prev | Up | Next
58.1. Sampling Method Support Functions | Home | 59.1. Creating Custom Scan Paths


59.1. Creating Custom Scan Paths
Prev Up Chapter 59. Writing a Custom Scan Provider Home Next

59.1. Creating Custom Scan Paths

59.1.1. Custom Scan Path Callbacks

A custom scan provider will typically add paths for a base relation by setting the following hook, which is called after the core code has generated all the access paths it can for the relation (except for Gather paths, which are made after this call so that they can use partial paths added by the hook):

typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,
                                            RelOptInfo *rel,
                                            Index rti,
                                            RangeTblEntry *rte);
extern PGDLLIMPORT set_rel_pathlist_hook_type set_rel_pathlist_hook;

Although this hook function can be used to examine, modify, or remove paths generated by the core system, a custom scan provider will typically confine itself to generating CustomPath objects and adding them to rel using add_path. The custom scan provider is responsible for initializing the CustomPath object, which is declared like this:

typedef struct CustomPath
{
    Path      path;
    uint32    flags;
    List     *custom_paths;
    List     *custom_private;
    const CustomPathMethods *methods;
} CustomPath;

path must be initialized as for any other path, including the row-count estimate, start and total cost, and sort ordering provided by this path. flags is a bit mask, which should include CUSTOMPATH_SUPPORT_BACKWARD_SCAN if the custom path can support a backward scan and CUSTOMPATH_SUPPORT_MARK_RESTORE if it can support mark and restore. Both capabilities are optional. An optional custom_paths is a list of Path nodes used by this custom-path node; these will be transformed into Plan nodes by planner. custom_private can be used to store the custom path's private data. Private data should be stored in a form that can be handled by nodeToString, so that debugging routines that attempt to print the custom path will work as designed. methods must point to a (usually statically allocated) object implementing the required custom path methods, of which there is currently only one.

A custom scan provider can also provide join paths. Just as for base relations, such a path must produce the same output as would normally be produced by the join it replaces. To do this, the join provider should set the following hook, and then within the hook function, create CustomPath path(s) for the join relation.

typedef void (*set_join_pathlist_hook_type) (PlannerInfo *root,
                                             RelOptInfo *joinrel,
                                             RelOptInfo *outerrel,
                                             RelOptInfo *innerrel,
                                             JoinType jointype,
                                             JoinPathExtraData *extra);
extern PGDLLIMPORT set_join_pathlist_hook_type set_join_pathlist_hook;

This hook will be invoked repeatedly for the same join relation, with different combinations of inner and outer relations; it is the responsibility of the hook to minimize duplicated work.

59.1.1. Custom Scan Path Callbacks

Plan *(*PlanCustomPath) (PlannerInfo *root,
                         RelOptInfo *rel,
                         CustomPath *best_path,
                         List *tlist,
                         List *clauses,
                         List *custom_plans);

Convert a custom path to a finished plan. The return value will generally be a CustomScan object, which the callback must allocate and initialize. See Section 59.2 for more details.

Prev | Up | Next
Chapter 59. Writing a Custom Scan Provider | Home | 59.2. Creating Custom Scan Plans


59.2. Creating Custom Scan Plans
Prev Up Chapter 59. Writing a Custom Scan Provider Home Next

59.2. Creating Custom Scan Plans

59.2.1. Custom Scan Plan Callbacks

A custom scan is represented in a finished plan tree using the following structure:

typedef struct CustomScan
{
    Scan      scan;
    uint32    flags;
    List     *custom_plans;
    List     *custom_exprs;
    List     *custom_private;
    List     *custom_scan_tlist;
    Bitmapset *custom_relids;
    const CustomScanMethods *methods;
} CustomScan;

scan must be initialized as for any other scan, including estimated costs, target lists, qualifications, and so on. flags is a bit mask with the same meaning as in CustomPath. custom_plans can be used to store child Plan nodes. custom_exprs should be used to store expression trees that will need to be fixed up by setrefs.c and subselect.c, while custom_private should be used to store other private data that is only used by the custom scan provider itself. custom_scan_tlist can be NIL when scanning a base relation, indicating that the custom scan returns scan tuples that match the base relation's row type. Otherwise it is a target list describing the actual scan tuples. custom_scan_tlist must be provided for joins, and could be provided for scans if the custom scan provider can compute some non-Var expressions. custom_relids is set by the core code to the set of relations (range table indexes) that this scan node handles; except when this scan is replacing a join, it will have only one member. methods must point to a (usually statically allocated) object implementing the required custom scan methods, which are further detailed below.

When a CustomScan scans a single relation, scan.scanrelid must be the range table index of the table to be scanned. When it replaces a join, scan.scanrelid should be zero.

Plan trees must be able to be duplicated using copyObject, so all the data stored within the “custom” fields must consist of nodes that that function can handle. Furthermore, custom scan providers cannot substitute a larger structure that embeds a CustomScan for the structure itself, as would be possible for a CustomPath or CustomScanState.

59.2.1. Custom Scan Plan Callbacks

Node *(*CreateCustomScanState) (CustomScan *cscan);

Allocate a CustomScanState for this CustomScan. The actual allocation will often be larger than required for an ordinary CustomScanState, because many providers will wish to embed that as the first field of a larger structure. The value returned must have the node tag and methods set appropriately, but other fields should be left as zeroes at this stage; after ExecInitCustomScan performs basic initialization, the BeginCustomScan callback will be invoked to give the custom scan provider a chance to do whatever else is needed.

Prev | Up | Next
59.1. Creating Custom Scan Paths | Home | 59.3. Executing Custom Scans


59.3. Executing Custom Scans
Prev Up Chapter 59. Writing a Custom Scan Provider Home Next

59.3. Executing Custom Scans

59.3.1. Custom Scan Execution Callbacks

When a CustomScan is executed, its execution state is represented by a CustomScanState, which is declared as follows:

typedef struct CustomScanState
{
    ScanState ss;
    uint32    flags;
    const CustomExecMethods *methods;
} CustomScanState;

ss is initialized as for any other scan state, except that if the scan is for a join rather than a base relation, ss.ss_currentRelation is left NULL. flags is a bit mask with the same meaning as in CustomPath and CustomScan. methods must point to a (usually statically allocated) object implementing the required custom scan state methods, which are further detailed below. Typically, a CustomScanState, which need not support copyObject, will actually be a larger structure embedding the above as its first member.

59.3.1. Custom Scan Execution Callbacks

void (*BeginCustomScan) (CustomScanState *node,
                         EState *estate,
                         int eflags);

Complete initialization of the supplied CustomScanState. Standard fields have been initialized by ExecInitCustomScan, but any private fields should be initialized here.

TupleTableSlot *(*ExecCustomScan) (CustomScanState *node);

Fetch the next scan tuple. If any tuples remain, it should fill ps_ResultTupleSlot with the next tuple in the current scan direction, and then return the tuple slot. If not, NULL or an empty slot should be returned.

void (*EndCustomScan) (CustomScanState *node);

Clean up any private data associated with the CustomScanState. This method is required, but it does not need to do anything if there is no associated data or it will be cleaned up automatically.

void (*ReScanCustomScan) (CustomScanState *node);

Rewind the current scan to the beginning and prepare to rescan the relation.

void (*MarkPosCustomScan) (CustomScanState *node);

Save the current scan position so that it can subsequently be restored by the RestrPosCustomScan callback. This callback is optional, and need only be supplied if the CUSTOMPATH_SUPPORT_MARK_RESTORE flag is set.

void (*RestrPosCustomScan) (CustomScanState *node);

Restore the previous scan position as saved by the MarkPosCustomScan callback. This callback is optional, and need only be supplied if the CUSTOMPATH_SUPPORT_MARK_RESTORE flag is set.

Size (*EstimateDSMCustomScan) (CustomScanState *node,
                               ParallelContext *pcxt);

Estimate the amount of dynamic shared memory that will be required for parallel operation. This may be higher than the amount that will actually be used, but it must not be lower. The return value is in bytes. This callback is optional, and need only be supplied if this custom scan provider supports parallel execution.

void (*InitializeDSMCustomScan) (CustomScanState *node,
                                 ParallelContext *pcxt,
                                 void *coordinate);

Initialize the dynamic shared memory that will be required for parallel operation. coordinate points to a shared memory area of size equal to the return value of EstimateDSMCustomScan. This callback is optional, and need only be supplied if this custom scan provider supports parallel execution.

void (*ReInitializeDSMCustomScan) (CustomScanState *node,
                                   ParallelContext *pcxt,
                                   void *coordinate);

Re-initialize the dynamic shared memory required for parallel operation when the custom-scan plan node is about to be re-scanned. This callback is optional, and need only be supplied if this custom scan provider supports parallel execution. Recommended practice is that this callback reset only shared state, while the ReScanCustomScan callback resets only local state. Currently, this callback will be called before ReScanCustomScan, but it's best not to rely on that ordering.

void (*InitializeWorkerCustomScan) (CustomScanState *node,
                                    shm_toc *toc,
                                    void *coordinate);

Initialize a parallel worker's local state based on the shared state set up by the leader during InitializeDSMCustomScan. This callback is optional, and need only be supplied if this custom scan provider supports parallel execution.

void (*ShutdownCustomScan) (CustomScanState *node);

Release resources when it is anticipated the node will not be executed to completion. This is not called in all cases; sometimes, EndCustomScan may be called without this function having been called first. Since the DSM segment used by parallel query is destroyed just after this callback is invoked, custom scan providers that wish to take some action before the DSM segment goes away should implement this method.

void (*ExplainCustomScan) (CustomScanState *node,
                           List *ancestors,
                           ExplainState *es);

Output additional information for EXPLAIN of a custom-scan plan node. This callback is optional. Common data stored in the ScanState, such as the target list and scan relation, will be shown even without this callback, but the callback allows the display of additional, private state.

Prev | Up | Next
59.2. Creating Custom Scan Plans | Home | Chapter 60. Genetic Query Optimizer


Chapter 60. Genetic Query Optimizer
Prev Up Part VII. Internals Home Next

Chapter 60. Genetic Query Optimizer

Table of Contents

60.1. Query Handling as a Complex Optimization Problem
60.2. Genetic Algorithms
60.3. Genetic Query Optimization (GEQO) in PostgreSQL
60.3.1. Generating Possible Plans with GEQO
60.3.2. Future Implementation Tasks for PostgreSQL GEQO
60.4. Further Reading

Author

Written by Martin Utesch (<utesch@aut.tu-freiberg.de>) for the Institute of Automatic Control at the University of Mining and Technology in Freiberg, Germany.

Prev | Up | Next
59.3. Executing Custom Scans | Home | 60.1. Query Handling as a Complex Optimization Problem


60.1. Query Handling as a Complex Optimization Problem
Prev Up Chapter 60. Genetic Query Optimizer Home Next

60.1. Query Handling as a Complex Optimization Problem

Among all relational operators the most difficult one to process and optimize is the join. The number of possible query plans grows exponentially with the number of joins in the query. Further optimization effort is caused by the support of a variety of join methods (e.g., nested loop, hash join, merge join in PostgreSQL) to process individual joins and a diversity of indexes (e.g., B-tree, hash, GiST and GIN in PostgreSQL) as access paths for relations.

The normal PostgreSQL query optimizer performs a near-exhaustive search over the space of alternative strategies. This algorithm, first introduced in IBM's System R database, produces a near-optimal join order, but can take an enormous amount of time and memory space when the number of joins in the query grows large. This makes the ordinary PostgreSQL query optimizer inappropriate for queries that join a large number of tables.

The Institute of Automatic Control at the University of Mining and Technology, in Freiberg, Germany, encountered some problems when it wanted to use PostgreSQL as the backend for a decision support knowledge based system for the maintenance of an electrical power grid. The DBMS needed to handle large join queries for the inference machine of the knowledge based system. The number of joins in these queries made using the normal query optimizer infeasible.

In the following we describe the implementation of a genetic algorithm to solve the join ordering problem in a manner that is efficient for queries involving large numbers of joins.

Prev | Up | Next
Chapter 60. Genetic Query Optimizer | Home | 60.2. Genetic Algorithms


60.2. Genetic Algorithms
Prev Up Chapter 60. Genetic Query Optimizer Home Next

60.2. Genetic Algorithms

The genetic algorithm (GA) is a heuristic optimization method which operates through randomized search. The set of possible solutions for the optimization problem is considered as a population of individuals. The degree of adaptation of an individual to its environment is specified by its fitness.

The coordinates of an individual in the search space are represented by chromosomes, in essence a set of character strings. A gene is a subsection of a chromosome which encodes the value of a single parameter being optimized. Typical encodings for a gene could be binary or integer.

Through simulation of the evolutionary operations recombination, mutation, and selection new generations of search points are found that show a higher average fitness than their ancestors. Figure 60.1 illustrates these steps.

Figure 60.1. Structure of a Genetic Algorithm

According to the comp.ai.genetic FAQ it cannot be stressed too strongly that a GA is not a pure random search for a solution to a problem. A GA uses stochastic processes, but the result is distinctly non-random (better than random).

Prev | Up | Next
60.1. Query Handling as a Complex Optimization Problem | Home | 60.3. Genetic Query Optimization (GEQO) in PostgreSQL


60.3. Genetic Query Optimization (GEQO) in PostgreSQL
Prev Up Chapter 60. Genetic Query Optimizer Home Next

60.3. Genetic Query Optimization (GEQO) in PostgreSQL

60.3.1. Generating Possible Plans with GEQO
60.3.2. Future Implementation Tasks for PostgreSQL GEQO

The GEQO module approaches the query optimization problem as though it were the well-known traveling salesman problem (TSP). Possible query plans are encoded as integer strings. Each string represents the join order from one relation of the query to the next. For example, the join tree

   /\
  /\ 2
 /\ 3
4  1

is encoded by the integer string '4-1-3-2', which means, first join relation '4' and '1', then '3', and then '2', where 1, 2, 3, 4 are relation IDs within the PostgreSQL optimizer.

Specific characteristics of the GEQO implementation in PostgreSQL are:

  • Usage of a steady state GA (replacement of the least fit individuals in a population, not whole-generational replacement) allows fast convergence towards improved query plans. This is essential for query handling with reasonable time;

  • Usage of edge recombination crossover which is especially suited to keep edge losses low for the solution of the TSP by means of a GA;

  • Mutation as genetic operator is deprecated so that no repair mechanisms are needed to generate legal TSP tours.

Parts of the GEQO module are adapted from D. Whitley's Genitor algorithm.

The GEQO module allows the PostgreSQL query optimizer to support large join queries effectively through non-exhaustive search.

60.3.1. Generating Possible Plans with GEQO

The GEQO planning process uses the standard planner code to generate plans for scans of individual relations. Then join plans are developed using the genetic approach. As shown above, each candidate join plan is represented by a sequence in which to join the base relations. In the initial stage, the GEQO code simply generates some possible join sequences at random. For each join sequence considered, the standard planner code is invoked to estimate the cost of performing the query using that join sequence. (For each step of the join sequence, all three possible join strategies are considered; and all the initially-determined relation scan plans are available. The estimated cost is the cheapest of these possibilities.) Join sequences with lower estimated cost are considered “more fit” than those with higher cost. The genetic algorithm discards the least fit candidates. Then new candidates are generated by combining genes of more-fit candidates — that is, by using randomly-chosen portions of known low-cost join sequences to create new sequences for consideration. This process is repeated until a preset number of join sequences have been considered; then the best one found at any time during the search is used to generate the finished plan.

This process is inherently nondeterministic, because of the randomized choices made during both the initial population selection and subsequent “mutation” of the best candidates. To avoid surprising changes of the selected plan, each run of the GEQO algorithm restarts its random number generator with the current geqo_seed parameter setting. As long as geqo_seed and the other GEQO parameters are kept fixed, the same plan will be generated for a given query (and other planner inputs such as statistics). To experiment with different search paths, try changing geqo_seed.

60.3.2. Future Implementation Tasks for PostgreSQL GEQO

Work is still needed to improve the genetic algorithm parameter settings. In file src/backend/optimizer/geqo/geqo_main.c, routines gimme_pool_size and gimme_number_generations, we have to find a compromise for the parameter settings to satisfy two competing demands:

  • Optimality of the query plan

  • Computing time

In the current implementation, the fitness of each candidate join sequence is estimated by running the standard planner's join selection and cost estimation code from scratch. To the extent that different candidates use similar sub-sequences of joins, a great deal of work will be repeated. This could be made significantly faster by retaining cost estimates for sub-joins. The problem is to avoid expending unreasonable amounts of memory on retaining that state.

At a more basic level, it is not clear that solving query optimization with a GA algorithm designed for TSP is appropriate. In the TSP case, the cost associated with any substring (partial tour) is independent of the rest of the tour, but this is certainly not true for query optimization. Thus it is questionable whether edge recombination crossover is the most effective mutation procedure.

Prev | Up | Next
60.2. Genetic Algorithms | Home | 60.4. Further Reading


60.4. Further Reading
Prev Up Chapter 60. Genetic Query Optimizer Home Next

60.4. Further Reading

The following resources contain additional information about genetic algorithms:

  • The Hitch-Hiker's Guide to Evolutionary Computation, (FAQ for news://comp.ai.genetic)

  • Evolutionary Computation and its application to art and design, by Craig Reynolds

  • [elma04]

  • [fong]

Prev | Up | Next
60.3. Genetic Query Optimization (GEQO) in PostgreSQL | Home | Chapter 61. Table Access Method Interface Definition


Chapter 62. Index Access Method Interface Definition
Prev Up Part VII. Internals Home Next

Chapter 62. Index Access Method Interface Definition

Table of Contents

62.1. Basic API Structure for Indexes
62.2. Index Access Method Functions
62.3. Index Scanning
62.4. Index Locking Considerations
62.5. Index Uniqueness Checks
62.6. Index Cost Estimation Functions

This chapter defines the interface between the core PostgreSQL system and index access methods, which manage individual index types. The core system knows nothing about indexes beyond what is specified here, so it is possible to develop entirely new index types by writing add-on code.

All indexes in PostgreSQL are what are known technically as secondary indexes; that is, the index is physically separate from the table file that it describes. Each index is stored as its own physical relation and so is described by an entry in the pg_class catalog. The contents of an index are entirely under the control of its index access method. In practice, all index access methods divide indexes into standard-size pages so that they can use the regular storage manager and buffer manager to access the index contents. (All the existing index access methods furthermore use the standard page layout described in Section 70.6, and most use the same format for index tuple headers; but these decisions are not forced on an access method.)

An index is effectively a mapping from some data key values to tuple identifiers, or TIDs, of row versions (tuples) in the index's parent table. A TID consists of a block number and an item number within that block (see Section 70.6). This is sufficient information to fetch a particular row version from the table. Indexes are not directly aware that under MVCC, there might be multiple extant versions of the same logical row; to an index, each tuple is an independent object that needs its own index entry. Thus, an update of a row always creates all-new index entries for the row, even if the key values did not change. (HOT tuples are an exception to this statement; but indexes do not deal with those, either.) Index entries for dead tuples are reclaimed (by vacuuming) when the dead tuples themselves are reclaimed.

Prev | Up | Next
Chapter 61. Table Access Method Interface Definition | Home | 62.1. Basic API Structure for Indexes


62.1. Basic API Structure for Indexes
Prev Up Chapter 62. Index Access Method Interface Definition Home Next

62.1. Basic API Structure for Indexes

Each index access method is described by a row in the pg_am system catalog. The pg_am entry specifies a name and a handler function for the index access method. These entries can be created and deleted using the CREATE ACCESS METHOD and DROP ACCESS METHOD SQL commands.

An index access method handler function must be declared to accept a single argument of type internal and to return the pseudo-type index_am_handler. The argument is a dummy value that simply serves to prevent handler functions from being called directly from SQL commands. The result of the function must be a palloc'd struct of type IndexAmRoutine, which contains everything that the core code needs to know to make use of the index access method. The IndexAmRoutine struct, also called the access method's API struct, includes fields specifying assorted fixed properties of the access method, such as whether it can support multicolumn indexes. More importantly, it contains pointers to support functions for the access method, which do all of the real work to access indexes. These support functions are plain C functions and are not visible or callable at the SQL level. The support functions are described in Section 62.2.

The structure IndexAmRoutine is defined thus:

typedef struct IndexAmRoutine
{
    NodeTag     type;

    /*
     * Total number of strategies (operators) by which we can traverse/search
     * this AM.  Zero if AM does not have a fixed set of strategy assignments.
     */
    uint16      amstrategies;
    /* total number of support functions that this AM uses */
    uint16      amsupport;
    /* opclass options support function number or 0 */
    uint16      amoptsprocnum;
    /* does AM support ORDER BY indexed column's value? */
    bool        amcanorder;
    /* does AM support ORDER BY result of an operator on indexed column? */
    bool        amcanorderbyop;
    /* does AM support backward scanning? */
    bool        amcanbackward;
    /* does AM support UNIQUE indexes? */
    bool        amcanunique;
    /* does AM support multi-column indexes? */
    bool        amcanmulticol;
    /* does AM require scans to have a constraint on the first index column? */
    bool        amoptionalkey;
    /* does AM handle ScalarArrayOpExpr quals? */
    bool        amsearcharray;
    /* does AM handle IS NULL/IS NOT NULL quals? */
    bool        amsearchnulls;
    /* can index storage data type differ from column data type? */
    bool        amstorage;
    /* can an index of this type be clustered on? */
    bool        amclusterable;
    /* does AM handle predicate locks? */
    bool        ampredlocks;
    /* does AM support parallel scan? */
    bool        amcanparallel;
    /* does AM support columns included with clause INCLUDE? */
    bool        amcaninclude;
    /* does AM use maintenance_work_mem? */
    bool        amusemaintenanceworkmem;
    /* OR of parallel vacuum flags */
    uint8       amparallelvacuumoptions;
    /* type of data stored in index, or InvalidOid if variable */
    Oid         amkeytype;

    /* interface functions */
    ambuild_function ambuild;
    ambuildempty_function ambuildempty;
    aminsert_function aminsert;
    ambulkdelete_function ambulkdelete;
    amvacuumcleanup_function amvacuumcleanup;
    amcanreturn_function amcanreturn;   /* can be NULL */
    amcostestimate_function amcostestimate;
    amoptions_function amoptions;
    amproperty_function amproperty;     /* can be NULL */
    ambuildphasename_function ambuildphasename;   /* can be NULL */
    amvalidate_function amvalidate;
    amadjustmembers_function amadjustmembers; /* can be NULL */
    ambeginscan_function ambeginscan;
    amrescan_function amrescan;
    amgettuple_function amgettuple;     /* can be NULL */
    amgetbitmap_function amgetbitmap;   /* can be NULL */
    amendscan_function amendscan;
    ammarkpos_function ammarkpos;       /* can be NULL */
    amrestrpos_function amrestrpos;     /* can be NULL */

    /* interface functions to support parallel index scans */
    amestimateparallelscan_function amestimateparallelscan;    /* can be NULL */
    aminitparallelscan_function aminitparallelscan;    /* can be NULL */
    amparallelrescan_function amparallelrescan;    /* can be NULL */
} IndexAmRoutine;

To be useful, an index access method must also have one or more operator families and operator classes defined in pg_opfamily, pg_opclass, pg_amop, and pg_amproc. These entries allow the planner to determine what kinds of query qualifications can be used with indexes of this access method. Operator families and classes are described in Section 38.16, which is prerequisite material for reading this chapter.

An individual index is defined by a pg_class entry that describes it as a physical relation, plus a pg_index entry that shows the logical content of the index — that is, the set of index columns it has and the semantics of those columns, as captured by the associated operator classes. The index columns (key values) can be either simple columns of the underlying table or expressions over the table rows. The index access method normally has no interest in where the index key values come from (it is always handed precomputed key values) but it will be very interested in the operator class information in pg_index. Both of these catalog entries can be accessed as part of the Relation data structure that is passed to all operations on the index.

Some of the flag fields of IndexAmRoutine have nonobvious implications. The requirements of amcanunique are discussed in Section 62.5. The amcanmulticol flag asserts that the access method supports multi-key-column indexes, while amoptionalkey asserts that it allows scans where no indexable restriction clause is given for the first index column. When amcanmulticol is false, amoptionalkey essentially says whether the access method supports full-index scans without any restriction clause. Access methods that support multiple index columns must support scans that omit restrictions on any or all of the columns after the first; however they are permitted to require some restriction to appear for the first index column, and this is signaled by setting amoptionalkey false. One reason that an index AM might set amoptionalkey false is if it doesn't index null values. Since most indexable operators are strict and hence cannot return true for null inputs, it is at first sight attractive to not store index entries for null values: they could never be returned by an index scan anyway. However, this argument fails when an index scan has no restriction clause for a given index column. In practice this means that indexes that have amoptionalkey true must index nulls, since the planner might decide to use such an index with no scan keys at all. A related restriction is that an index access method that supports multiple index columns must support indexing null values in columns after the first, because the planner will assume the index can be used for queries that do not restrict these columns. For example, consider an index on (a,b) and a query with WHERE a = 4. The system will assume the index can be used to scan for rows with a = 4, which is wrong if the index omits rows where b is null. It is, however, OK to omit rows where the first indexed column is null. An index access method that does index nulls may also set amsearchnulls, indicating that it supports IS NULL and IS NOT NULL clauses as search conditions.

The amcaninclude flag indicates whether the access method supports “included” columns, that is it can store (without processing) additional columns beyond the key column(s). The requirements of the preceding paragraph apply only to the key columns. In particular, the combination of amcanmulticol=false and amcaninclude=true is sensible: it means that there can only be one key column, but there can also be included column(s). Also, included columns must be allowed to be null, independently of amoptionalkey.

Prev | Up | Next
Chapter 62. Index Access Method Interface Definition | Home | 62.2. Index Access Method Functions


62.2. Index Access Method Functions
Prev Up Chapter 62. Index Access Method Interface Definition Home Next

62.2. Index Access Method Functions

The index construction and maintenance functions that an index access method must provide in IndexAmRoutine are:

IndexBuildResult *
ambuild (Relation heapRelation,
         Relation indexRelation,
         IndexInfo *indexInfo);

Build a new index. The index relation has been physically created, but is empty. It must be filled in with whatever fixed data the access method requires, plus entries for all tuples already existing in the table. Ordinarily the ambuild function will call table_index_build_scan() to scan the table for existing tuples and compute the keys that need to be inserted into the index. The function must return a palloc'd struct containing statistics about the new index.

void
ambuildempty (Relation indexRelation);

Build an empty index, and write it to the initialization fork (INIT_FORKNUM) of the given relation. This method is called only for unlogged indexes; the empty index written to the initialization fork will be copied over the main relation fork on each server restart.

bool
aminsert (Relation indexRelation,
          Datum *values,
          bool *isnull,
          ItemPointer heap_tid,
          Relation heapRelation,
          IndexUniqueCheck checkUnique,
          bool indexUnchanged,
          IndexInfo *indexInfo);

Insert a new tuple into an existing index. The values and isnull arrays give the key values to be indexed, and heap_tid is the TID to be indexed. If the access method supports unique indexes (its amcanunique flag is true) then checkUnique indicates the type of uniqueness check to perform. This varies depending on whether the unique constraint is deferrable; see Section 62.5 for details. Normally the access method only needs the heapRelation parameter when performing uniqueness checking (since then it will have to look into the heap to verify tuple liveness).

The indexUnchanged Boolean value gives a hint about the nature of the tuple to be indexed. When it is true, the tuple is a duplicate of some existing tuple in the index. The new tuple is a logically unchanged successor MVCC tuple version. This happens when an UPDATE takes place that does not modify any columns covered by the index, but nevertheless requires a new version in the index. The index AM may use this hint to decide to apply bottom-up index deletion in parts of the index where many versions of the same logical row accumulate. Note that updating a non-key column does not affect the value of indexUnchanged.

The function's Boolean result value is significant only when checkUnique is UNIQUE_CHECK_PARTIAL. In this case a true result means the new entry is known unique, whereas false means it might be non-unique (and a deferred uniqueness check must be scheduled). For other cases a constant false result is recommended.

Some indexes might not index all tuples. If the tuple is not to be indexed, aminsert should just return without doing anything.

If the index AM wishes to cache data across successive index insertions within an SQL statement, it can allocate space in indexInfo->ii_Context and store a pointer to the data in indexInfo->ii_AmCache (which will be NULL initially).

IndexBulkDeleteResult *
ambulkdelete (IndexVacuumInfo *info,
              IndexBulkDeleteResult *stats,
              IndexBulkDeleteCallback callback,
              void *callback_state);

Delete tuple(s) from the index. This is a “bulk delete” operation that is intended to be implemented by scanning the whole index and checking each entry to see if it should be deleted. The passed-in callback function must be called, in the style callback(TID, callback_state) returns bool, to determine whether any particular index entry, as identified by its referenced TID, is to be deleted. Must return either NULL or a palloc'd struct containing statistics about the effects of the deletion operation. It is OK to return NULL if no information needs to be passed on to amvacuumcleanup.

Because of limited maintenance_work_mem, ambulkdelete might need to be called more than once when many tuples are to be deleted. The stats argument is the result of the previous call for this index (it is NULL for the first call within a VACUUM operation). This allows the AM to accumulate statistics across the whole operation. Typically, ambulkdelete will modify and return the same struct if the passed stats is not null.

IndexBulkDeleteResult *
amvacuumcleanup (IndexVacuumInfo *info,
                 IndexBulkDeleteResult *stats);

Clean up after a VACUUM operation (zero or more ambulkdelete calls). This does not have to do anything beyond returning index statistics, but it might perform bulk cleanup such as reclaiming empty index pages. stats is whatever the last ambulkdelete call returned, or NULL if ambulkdelete was not called because no tuples needed to be deleted. If the result is not NULL it must be a palloc'd struct. The statistics it contains will be used to update pg_class, and will be reported by VACUUM if VERBOSE is given. It is OK to return NULL if the index was not changed at all during the VACUUM operation, but otherwise correct stats should be returned.

amvacuumcleanup will also be called at completion of an ANALYZE operation. In this case stats is always NULL and any return value will be ignored. This case can be distinguished by checking info->analyze_only. It is recommended that the access method do nothing except post-insert cleanup in such a call, and that only in an autovacuum worker process.

bool
amcanreturn (Relation indexRelation, int attno);

Check whether the index can support index-only scans on the given column, by returning the column's original indexed value. The attribute number is 1-based, i.e., the first column's attno is 1. Returns true if supported, else false. This function should always return true for included columns (if those are supported), since there's little point in an included column that can't be retrieved. If the access method does not support index-only scans at all, the amcanreturn field in its IndexAmRoutine struct can be set to NULL.

void
amcostestimate (PlannerInfo *root,
                IndexPath *path,
                double loop_count,
                Cost *indexStartupCost,
                Cost *indexTotalCost,
                Selectivity *indexSelectivity,
                double *indexCorrelation,
                double *indexPages);

Estimate the costs of an index scan. This function is described fully in Section 62.6, below.

bytea *
amoptions (ArrayType *reloptions,
           bool validate);

Parse and validate the reloptions array for an index. This is called only when a non-null reloptions array exists for the index. reloptions is a text array containing entries of the form name=value. The function should construct a bytea value, which will be copied into the rd_options field of the index's relcache entry. The data contents of the bytea value are open for the access method to define; most of the standard access methods use struct StdRdOptions. When validate is true, the function should report a suitable error message if any of the options are unrecognized or have invalid values; when validate is false, invalid entries should be silently ignored. (validate is false when loading options already stored in pg_catalog; an invalid entry could only be found if the access method has changed its rules for options, and in that case ignoring obsolete entries is appropriate.) It is OK to return NULL if default behavior is wanted.

bool
amproperty (Oid index_oid, int attno,
            IndexAMProperty prop, const char *propname,
            bool *res, bool *isnull);

The amproperty method allows index access methods to override the default behavior of pg_index_column_has_property and related functions. If the access method does not have any special behavior for index property inquiries, the amproperty field in its IndexAmRoutine struct can be set to NULL. Otherwise, the amproperty method will be called with index_oid and attno both zero for pg_indexam_has_property calls, or with index_oid valid and attno zero for pg_index_has_property calls, or with index_oid valid and attno greater than zero for pg_index_column_has_property calls. prop is an enum value identifying the property being tested, while propname is the original property name string. If the core code does not recognize the property name then prop is AMPROP_UNKNOWN. Access methods can define custom property names by checking propname for a match (use pg_strcasecmp to match, for consistency with the core code); for names known to the core code, it's better to inspect prop. If the amproperty method returns true then it has determined the property test result: it must set *res to the Boolean value to return, or set *isnull to true to return a NULL. (Both of the referenced variables are initialized to false before the call.) If the amproperty method returns false then the core code will proceed with its normal logic for determining the property test result.

Access methods that support ordering operators should implement AMPROP_DISTANCE_ORDERABLE property testing, as the core code does not know how to do that and will return NULL. It may also be advantageous to implement AMPROP_RETURNABLE testing, if that can be done more cheaply than by opening the index and calling amcanreturn, which is the core code's default behavior. The default behavior should be satisfactory for all other standard properties.

char *
ambuildphasename (int64 phasenum);

Return the textual name of the given build phase number. The phase numbers are those reported during an index build via the pgstat_progress_update_param interface. The phase names are then exposed in the pg_stat_progress_create_index view.

bool
amvalidate (Oid opclassoid);

Validate the catalog entries for the specified operator class, so far as the access method can reasonably do that. For example, this might include testing that all required support functions are provided. The amvalidate function must return false if the opclass is invalid. Problems should be reported with ereport messages, typically at INFO level.

void
amadjustmembers (Oid opfamilyoid,
                 Oid opclassoid,
                 List *operators,
                 List *functions);

Validate proposed new operator and function members of an operator family, so far as the access method can reasonably do that, and set their dependency types if the default is not satisfactory. This is called during CREATE OPERATOR CLASS and during ALTER OPERATOR FAMILY ADD; in the latter case opclassoid is InvalidOid. The List arguments are lists of OpFamilyMember structs, as defined in amapi.h. Tests done by this function will typically be a subset of those performed by amvalidate, since amadjustmembers cannot assume that it is seeing a complete set of members. For example, it would be reasonable to check the signature of a support function, but not to check whether all required support functions are provided. Any problems can be reported by throwing an error. The dependency-related fields of the OpFamilyMember structs are initialized by the core code to create hard dependencies on the opclass if this is CREATE OPERATOR CLASS, or soft dependencies on the opfamily if this is ALTER OPERATOR FAMILY ADD. amadjustmembers can adjust these fields if some other behavior is more appropriate. For example, GIN, GiST, and SP-GiST always set operator members to have soft dependencies on the opfamily, since the connection between an operator and an opclass is relatively weak in these index types; so it is reasonable to allow operator members to be added and removed freely. Optional support functions are typically also given soft dependencies, so that they can be removed if necessary.

The purpose of an index, of course, is to support scans for tuples matching an indexable WHERE condition, often called a qualifier or scan key. The semantics of index scanning are described more fully in Section 62.3, below. An index access method can support “plain” index scans, “bitmap” index scans, or both. The scan-related functions that an index access method must or may provide are:

IndexScanDesc
ambeginscan (Relation indexRelation,
             int nkeys,
             int norderbys);

Prepare for an index scan. The nkeys and norderbys parameters indicate the number of quals and ordering operators that will be used in the scan; these may be useful for space allocation purposes. Note that the actual values of the scan keys aren't provided yet. The result must be a palloc'd struct. For implementation reasons the index access method must create this struct by calling RelationGetIndexScan(). In most cases ambeginscan does little beyond making that call and perhaps acquiring locks; the interesting parts of index-scan startup are in amrescan.

void
amrescan (IndexScanDesc scan,
          ScanKey keys,
          int nkeys,
          ScanKey orderbys,
          int norderbys);

Start or restart an index scan, possibly with new scan keys. (To restart using previously-passed keys, NULL is passed for keys and/or orderbys.) Note that it is not allowed for the number of keys or order-by operators to be larger than what was passed to ambeginscan. In practice the restart feature is used when a new outer tuple is selected by a nested-loop join and so a new key comparison value is needed, but the scan key structure remains the same.

bool
amgettuple (IndexScanDesc scan,
            ScanDirection direction);

Fetch the next tuple in the given scan, moving in the given direction (forward or backward in the index). Returns true if a tuple was obtained, false if no matching tuples remain. In the true case the tuple TID is stored into the scan structure. Note that “success” means only that the index contains an entry that matches the scan keys, not that the tuple necessarily still exists in the heap or will pass the caller's snapshot test. On success, amgettuple must also set scan->xs_recheck to true or false. False means it is certain that the index entry matches the scan keys. True means this is not certain, and the conditions represented by the scan keys must be rechecked against the heap tuple after fetching it. This provision supports “lossy” index operators. Note that rechecking will extend only to the scan conditions; a partial index predicate (if any) is never rechecked by amgettuple callers.

If the index supports index-only scans (i.e., amcanreturn returns true for any of its columns), then on success the AM must also check scan->xs_want_itup, and if that is true it must return the originally indexed data for the index entry. Columns for which amcanreturn returns false can be returned as nulls. The data can be returned in the form of an IndexTuple pointer stored at scan->xs_itup, with tuple descriptor scan->xs_itupdesc; or in the form of a HeapTuple pointer stored at scan->xs_hitup, with tuple descriptor scan->xs_hitupdesc. (The latter format should be used when reconstructing data that might possibly not fit into an IndexTuple.) In either case, management of the data referenced by the pointer is the access method's responsibility. The data must remain good at least until the next amgettuple, amrescan, or amendscan call for the scan.

The amgettuple function need only be provided if the access method supports “plain” index scans. If it doesn't, the amgettuple field in its IndexAmRoutine struct must be set to NULL.

int64
amgetbitmap (IndexScanDesc scan,
             TIDBitmap *tbm);

Fetch all tuples in the given scan and add them to the caller-supplied TIDBitmap (that is, OR the set of tuple IDs into whatever set is already in the bitmap). The number of tuples fetched is returned (this might be just an approximate count, for instance some AMs do not detect duplicates). While inserting tuple IDs into the bitmap, amgetbitmap can indicate that rechecking of the scan conditions is required for specific tuple IDs. This is analogous to the xs_recheck output parameter of amgettuple. Note: in the current implementation, support for this feature is conflated with support for lossy storage of the bitmap itself, and therefore callers recheck both the scan conditions and the partial index predicate (if any) for recheckable tuples. That might not always be true, however. amgetbitmap and amgettuple cannot be used in the same index scan; there are other restrictions too when using amgetbitmap, as explained in Section 62.3.

The amgetbitmap function need only be provided if the access method supports “bitmap” index scans. If it doesn't, the amgetbitmap field in its IndexAmRoutine struct must be set to NULL.

void
amendscan (IndexScanDesc scan);

End a scan and release resources. The scan struct itself should not be freed, but any locks or pins taken internally by the access method must be released, as well as any other memory allocated by ambeginscan and other scan-related functions.

void
ammarkpos (IndexScanDesc scan);

Mark current scan position. The access method need only support one remembered scan position per scan.

The ammarkpos function need only be provided if the access method supports ordered scans. If it doesn't, the ammarkpos field in its IndexAmRoutine struct may be set to NULL.

void
amrestrpos (IndexScanDesc scan);

Restore the scan to the most recently marked position.

The amrestrpos function need only be provided if the access method supports ordered scans. If it doesn't, the amrestrpos field in its IndexAmRoutine struct may be set to NULL.

In addition to supporting ordinary index scans, some types of index may wish to support parallel index scans, which allow multiple backends to cooperate in performing an index scan. The index access method should arrange things so that each cooperating process returns a subset of the tuples that would be performed by an ordinary, non-parallel index scan, but in such a way that the union of those subsets is equal to the set of tuples that would be returned by an ordinary, non-parallel index scan. Furthermore, while there need not be any global ordering of tuples returned by a parallel scan, the ordering of that subset of tuples returned within each cooperating backend must match the requested ordering. The following functions may be implemented to support parallel index scans:

Size
amestimateparallelscan (void);

Estimate and return the number of bytes of dynamic shared memory which the access method will be needed to perform a parallel scan. (This number is in addition to, not in lieu of, the amount of space needed for AM-independent data in ParallelIndexScanDescData.)

It is not necessary to implement this function for access methods which do not support parallel scans or for which the number of additional bytes of storage required is zero.

void
aminitparallelscan (void *target);

This function will be called to initialize dynamic shared memory at the beginning of a parallel scan. target will point to at least the number of bytes previously returned by amestimateparallelscan, and this function may use that amount of space to store whatever data it wishes.

It is not necessary to implement this function for access methods which do not support parallel scans or in cases where the shared memory space required needs no initialization.

void
amparallelrescan (IndexScanDesc scan);

This function, if implemented, will be called when a parallel index scan must be restarted. It should reset any shared state set up by aminitparallelscan such that the scan will be restarted from the beginning.

Prev | Up | Next
62.1. Basic API Structure for Indexes | Home | 62.3. Index Scanning


62.3. Index Scanning
Prev Up Chapter 62. Index Access Method Interface Definition Home Next

62.3. Index Scanning

In an index scan, the index access method is responsible for regurgitating the TIDs of all the tuples it has been told about that match the scan keys. The access method is not involved in actually fetching those tuples from the index's parent table, nor in determining whether they pass the scan's visibility test or other conditions.

A scan key is the internal representation of a WHERE clause of the form index_key operator constant, where the index key is one of the columns of the index and the operator is one of the members of the operator family associated with that index column. An index scan has zero or more scan keys, which are implicitly ANDed — the returned tuples are expected to satisfy all the indicated conditions.

The access method can report that the index is lossy, or requires rechecks, for a particular query. This implies that the index scan will return all the entries that pass the scan key, plus possibly additional entries that do not. The core system's index-scan machinery will then apply the index conditions again to the heap tuple to verify whether or not it really should be selected. If the recheck option is not specified, the index scan must return exactly the set of matching entries.

Note that it is entirely up to the access method to ensure that it correctly finds all and only the entries passing all the given scan keys. Also, the core system will simply hand off all the WHERE clauses that match the index keys and operator families, without any semantic analysis to determine whether they are redundant or contradictory. As an example, given WHERE x > 4 AND x > 14 where x is a b-tree indexed column, it is left to the b-tree amrescan function to realize that the first scan key is redundant and can be discarded. The extent of preprocessing needed during amrescan will depend on the extent to which the index access method needs to reduce the scan keys to a “normalized” form.

Some access methods return index entries in a well-defined order, others do not. There are actually two different ways that an access method can support sorted output:

  • Access methods that always return entries in the natural ordering of their data (such as btree) should set amcanorder to true. Currently, such access methods must use btree-compatible strategy numbers for their equality and ordering operators.

  • Access methods that support ordering operators should set amcanorderbyop to true. This indicates that the index is capable of returning entries in an order satisfying ORDER BY index_key operator constant. Scan modifiers of that form can be passed to amrescan as described previously.

The amgettuple function has a direction argument, which can be either ForwardScanDirection (the normal case) or BackwardScanDirection. If the first call after amrescan specifies BackwardScanDirection, then the set of matching index entries is to be scanned back-to-front rather than in the normal front-to-back direction, so amgettuple must return the last matching tuple in the index, rather than the first one as it normally would. (This will only occur for access methods that set amcanorder to true.) After the first call, amgettuple must be prepared to advance the scan in either direction from the most recently returned entry. (But if amcanbackward is false, all subsequent calls will have the same direction as the first one.)

Access methods that support ordered scans must support “marking” a position in a scan and later returning to the marked position. The same position might be restored multiple times. However, only one position need be remembered per scan; a new ammarkpos call overrides the previously marked position. An access method that does not support ordered scans need not provide ammarkpos and amrestrpos functions in IndexAmRoutine; set those pointers to NULL instead.

Both the scan position and the mark position (if any) must be maintained consistently in the face of concurrent insertions or deletions in the index. It is OK if a freshly-inserted entry is not returned by a scan that would have found the entry if it had existed when the scan started, or for the scan to return such an entry upon rescanning or backing up even though it had not been returned the first time through. Similarly, a concurrent delete might or might not be reflected in the results of a scan. What is important is that insertions or deletions not cause the scan to miss or multiply return entries that were not themselves being inserted or deleted.

If the index stores the original indexed data values (and not some lossy representation of them), it is useful to support index-only scans, in which the index returns the actual data not just the TID of the heap tuple. This will only avoid I/O if the visibility map shows that the TID is on an all-visible page; else the heap tuple must be visited anyway to check MVCC visibility. But that is no concern of the access method's.

Instead of using amgettuple, an index scan can be done with amgetbitmap to fetch all tuples in one call. This can be noticeably more efficient than amgettuple because it allows avoiding lock/unlock cycles within the access method. In principle amgetbitmap should have the same effects as repeated amgettuple calls, but we impose several restrictions to simplify matters. First of all, amgetbitmap returns all tuples at once and marking or restoring scan positions isn't supported. Secondly, the tuples are returned in a bitmap which doesn't have any specific ordering, which is why amgetbitmap doesn't take a direction argument. (Ordering operators will never be supplied for such a scan, either.) Also, there is no provision for index-only scans with amgetbitmap, since there is no way to return the contents of index tuples. Finally, amgetbitmap does not guarantee any locking of the returned tuples, with implications spelled out in Section 62.4.

Note that it is permitted for an access method to implement only amgetbitmap and not amgettuple, or vice versa, if its internal implementation is unsuited to one API or the other.

Prev | Up | Next
62.2. Index Access Method Functions | Home | 62.4. Index Locking Considerations


62.4. Index Locking Considerations
Prev Up Chapter 62. Index Access Method Interface Definition Home Next

62.4. Index Locking Considerations

Index access methods must handle concurrent updates of the index by multiple processes. The core PostgreSQL system obtains AccessShareLock on the index during an index scan, and RowExclusiveLock when updating the index (including plain VACUUM). Since these lock types do not conflict, the access method is responsible for handling any fine-grained locking it might need. An ACCESS EXCLUSIVE lock on the index as a whole will be taken only during index creation, destruction, or REINDEX (SHARE UPDATE EXCLUSIVE is taken instead with CONCURRENTLY).

Building an index type that supports concurrent updates usually requires extensive and subtle analysis of the required behavior. For the b-tree and hash index types, you can read about the design decisions involved in src/backend/access/nbtree/README and src/backend/access/hash/README.

Aside from the index's own internal consistency requirements, concurrent updates create issues about consistency between the parent table (the heap) and the index. Because PostgreSQL separates accesses and updates of the heap from those of the index, there are windows in which the index might be inconsistent with the heap. We handle this problem with the following rules:

  • A new heap entry is made before making its index entries. (Therefore a concurrent index scan is likely to fail to see the heap entry. This is okay because the index reader would be uninterested in an uncommitted row anyway. But see Section 62.5.)

  • When a heap entry is to be deleted (by VACUUM), all its index entries must be removed first.

  • An index scan must maintain a pin on the index page holding the item last returned by amgettuple, and ambulkdelete cannot delete entries from pages that are pinned by other backends. The need for this rule is explained below.

Without the third rule, it is possible for an index reader to see an index entry just before it is removed by VACUUM, and then to arrive at the corresponding heap entry after that was removed by VACUUM. This creates no serious problems if that item number is still unused when the reader reaches it, since an empty item slot will be ignored by heap_fetch(). But what if a third backend has already re-used the item slot for something else? When using an MVCC-compliant snapshot, there is no problem because the new occupant of the slot is certain to be too new to pass the snapshot test. However, with a non-MVCC-compliant snapshot (such as SnapshotAny), it would be possible to accept and return a row that does not in fact match the scan keys. We could defend against this scenario by requiring the scan keys to be rechecked against the heap row in all cases, but that is too expensive. Instead, we use a pin on an index page as a proxy to indicate that the reader might still be “in flight” from the index entry to the matching heap entry. Making ambulkdelete block on such a pin ensures that VACUUM cannot delete the heap entry before the reader is done with it. This solution costs little in run time, and adds blocking overhead only in the rare cases where there actually is a conflict.

This solution requires that index scans be “synchronous”: we have to fetch each heap tuple immediately after scanning the corresponding index entry. This is expensive for a number of reasons. An “asynchronous” scan in which we collect many TIDs from the index, and only visit the heap tuples sometime later, requires much less index locking overhead and can allow a more efficient heap access pattern. Per the above analysis, we must use the synchronous approach for non-MVCC-compliant snapshots, but an asynchronous scan is workable for a query using an MVCC snapshot.

In an amgetbitmap index scan, the access method does not keep an index pin on any of the returned tuples. Therefore it is only safe to use such scans with MVCC-compliant snapshots.

When the ampredlocks flag is not set, any scan using that index access method within a serializable transaction will acquire a nonblocking predicate lock on the full index. This will generate a read-write conflict with the insert of any tuple into that index by a concurrent serializable transaction. If certain patterns of read-write conflicts are detected among a set of concurrent serializable transactions, one of those transactions may be canceled to protect data integrity. When the flag is set, it indicates that the index access method implements finer-grained predicate locking, which will tend to reduce the frequency of such transaction cancellations.

Prev | Up | Next
62.3. Index Scanning | Home | 62.5. Index Uniqueness Checks


62.5. Index Uniqueness Checks
Prev Up Chapter 62. Index Access Method Interface Definition Home Next

62.5. Index Uniqueness Checks

PostgreSQL enforces SQL uniqueness constraints using unique indexes, which are indexes that disallow multiple entries with identical keys. An access method that supports this feature sets amcanunique true. (At present, only b-tree supports it.) Columns listed in the INCLUDE clause are not considered when enforcing uniqueness.

Because of MVCC, it is always necessary to allow duplicate entries to exist physically in an index: the entries might refer to successive versions of a single logical row. The behavior we actually want to enforce is that no MVCC snapshot could include two rows with equal index keys. This breaks down into the following cases that must be checked when inserting a new row into a unique index:

  • If a conflicting valid row has been deleted by the current transaction, it's okay. (In particular, since an UPDATE always deletes the old row version before inserting the new version, this will allow an UPDATE on a row without changing the key.)

  • If a conflicting row has been inserted by an as-yet-uncommitted transaction, the would-be inserter must wait to see if that transaction commits. If it rolls back then there is no conflict. If it commits without deleting the conflicting row again, there is a uniqueness violation. (In practice we just wait for the other transaction to end and then redo the visibility check in toto.)

  • Similarly, if a conflicting valid row has been deleted by an as-yet-uncommitted transaction, the would-be inserter must wait for that transaction to commit or abort, and then repeat the test.

Furthermore, immediately before reporting a uniqueness violation according to the above rules, the access method must recheck the liveness of the row being inserted. If it is committed dead then no violation should be reported. (This case cannot occur during the ordinary scenario of inserting a row that's just been created by the current transaction. It can happen during CREATE UNIQUE INDEX CONCURRENTLY, however.)

We require the index access method to apply these tests itself, which means that it must reach into the heap to check the commit status of any row that is shown to have a duplicate key according to the index contents. This is without a doubt ugly and non-modular, but it saves redundant work: if we did a separate probe then the index lookup for a conflicting row would be essentially repeated while finding the place to insert the new row's index entry. What's more, there is no obvious way to avoid race conditions unless the conflict check is an integral part of insertion of the new index entry.

If the unique constraint is deferrable, there is additional complexity: we need to be able to insert an index entry for a new row, but defer any uniqueness-violation error until end of statement or even later. To avoid unnecessary repeat searches of the index, the index access method should do a preliminary uniqueness check during the initial insertion. If this shows that there is definitely no conflicting live tuple, we are done. Otherwise, we schedule a recheck to occur when it is time to enforce the constraint. If, at the time of the recheck, both the inserted tuple and some other tuple with the same key are live, then the error must be reported. (Note that for this purpose, “live” actually means “any tuple in the index entry's HOT chain is live”.) To implement this, the aminsert function is passed a checkUnique parameter having one of the following values:

  • UNIQUE_CHECK_NO indicates that no uniqueness checking should be done (this is not a unique index).

  • UNIQUE_CHECK_YES indicates that this is a non-deferrable unique index, and the uniqueness check must be done immediately, as described above.

  • UNIQUE_CHECK_PARTIAL indicates that the unique constraint is deferrable. PostgreSQL will use this mode to insert each row's index entry. The access method must allow duplicate entries into the index, and report any potential duplicates by returning false from aminsert. For each row for which false is returned, a deferred recheck will be scheduled.

    The access method must identify any rows which might violate the unique constraint, but it is not an error for it to report false positives. This allows the check to be done without waiting for other transactions to finish; conflicts reported here are not treated as errors and will be rechecked later, by which time they may no longer be conflicts.

  • UNIQUE_CHECK_EXISTING indicates that this is a deferred recheck of a row that was reported as a potential uniqueness violation. Although this is implemented by calling aminsert, the access method must not insert a new index entry in this case. The index entry is already present. Rather, the access method must check to see if there is another live index entry. If so, and if the target row is also still live, report error.

    It is recommended that in a UNIQUE_CHECK_EXISTING call, the access method further verify that the target row actually does have an existing entry in the index, and report error if not. This is a good idea because the index tuple values passed to aminsert will have been recomputed. If the index definition involves functions that are not really immutable, we might be checking the wrong area of the index. Checking that the target row is found in the recheck verifies that we are scanning for the same tuple values as were used in the original insertion.

Prev | Up | Next
62.4. Index Locking Considerations | Home | 62.6. Index Cost Estimation Functions


62.6. Index Cost Estimation Functions
Prev Up Chapter 62. Index Access Method Interface Definition Home Next

62.6. Index Cost Estimation Functions

The amcostestimate function is given information describing a possible index scan, including lists of WHERE and ORDER BY clauses that have been determined to be usable with the index. It must return estimates of the cost of accessing the index and the selectivity of the WHERE clauses (that is, the fraction of parent-table rows that will be retrieved during the index scan). For simple cases, nearly all the work of the cost estimator can be done by calling standard routines in the optimizer; the point of having an amcostestimate function is to allow index access methods to provide index-type-specific knowledge, in case it is possible to improve on the standard estimates.

Each amcostestimate function must have the signature:

void
amcostestimate (PlannerInfo *root,
                IndexPath *path,
                double loop_count,
                Cost *indexStartupCost,
                Cost *indexTotalCost,
                Selectivity *indexSelectivity,
                double *indexCorrelation,
                double *indexPages);

The first three parameters are inputs:

root

The planner's information about the query being processed.

path

The index access path being considered. All fields except cost and selectivity values are valid.

loop_count

The number of repetitions of the index scan that should be factored into the cost estimates. This will typically be greater than one when considering a parameterized scan for use in the inside of a nestloop join. Note that the cost estimates should still be for just one scan; a larger loop_count means that it may be appropriate to allow for some caching effects across multiple scans.

The last five parameters are pass-by-reference outputs:

*indexStartupCost

Set to cost of index start-up processing

*indexTotalCost

Set to total cost of index processing

*indexSelectivity

Set to index selectivity

*indexCorrelation

Set to correlation coefficient between index scan order and underlying table's order

*indexPages

Set to number of index leaf pages

Note that cost estimate functions must be written in C, not in SQL or any available procedural language, because they must access internal data structures of the planner/optimizer.

The index access costs should be computed using the parameters used by src/backend/optimizer/path/costsize.c: a sequential disk block fetch has cost seq_page_cost, a nonsequential fetch has cost random_page_cost, and the cost of processing one index row should usually be taken as cpu_index_tuple_cost. In addition, an appropriate multiple of cpu_operator_cost should be charged for any comparison operators invoked during index processing (especially evaluation of the indexquals themselves).

The access costs should include all disk and CPU costs associated with scanning the index itself, but not the costs of retrieving or processing the parent-table rows that are identified by the index.

The “start-up cost” is the part of the total scan cost that must be expended before we can begin to fetch the first row. For most indexes this can be taken as zero, but an index type with a high start-up cost might want to set it nonzero.

The indexSelectivity should be set to the estimated fraction of the parent table rows that will be retrieved during the index scan. In the case of a lossy query, this will typically be higher than the fraction of rows that actually pass the given qual conditions.

The indexCorrelation should be set to the correlation (ranging between -1.0 and 1.0) between the index order and the table order. This is used to adjust the estimate for the cost of fetching rows from the parent table.

The indexPages should be set to the number of leaf pages. This is used to estimate the number of workers for parallel index scan.

When loop_count is greater than one, the returned numbers should be averages expected for any one scan of the index.

Cost Estimation

A typical cost estimator will proceed as follows:

  1. Estimate and return the fraction of parent-table rows that will be visited based on the given qual conditions. In the absence of any index-type-specific knowledge, use the standard optimizer function clauselist_selectivity():

    *indexSelectivity = clauselist_selectivity(root, path->indexquals,
                                           path->indexinfo->rel->relid,
                                           JOIN_INNER, NULL);

  2. Estimate the number of index rows that will be visited during the scan. For many index types this is the same as indexSelectivity times the number of rows in the index, but it might be more. (Note that the index's size in pages and rows is available from the path->indexinfo struct.)

  3. Estimate the number of index pages that will be retrieved during the scan. This might be just indexSelectivity times the index's size in pages.

  4. Compute the index access cost. A generic estimator might do this:

    /*
 * Our generic assumption is that the index pages will be read
 * sequentially, so they cost seq_page_cost each, not random_page_cost.
 * Also, we charge for evaluation of the indexquals at each index row.
 * All the costs are assumed to be paid incrementally during the scan.
 */
cost_qual_eval(&index_qual_cost, path->indexquals, root);
*indexStartupCost = index_qual_cost.startup;
*indexTotalCost = seq_page_cost * numIndexPages +
    (cpu_index_tuple_cost + index_qual_cost.per_tuple) * numIndexTuples;

    However, the above does not account for amortization of index reads across repeated index scans.

  5. Estimate the index correlation. For a simple ordered index on a single field, this can be retrieved from pg_statistic. If the correlation is not known, the conservative estimate is zero (no correlation).

Examples of cost estimator functions can be found in src/backend/utils/adt/selfuncs.c.

Prev | Up | Next
62.5. Index Uniqueness Checks | Home | Chapter 63. Generic WAL Records


Chapter 63. Generic WAL Records
Prev Up Part VII. Internals Home Next

Chapter 63. Generic WAL Records

Although all built-in WAL-logged modules have their own types of WAL records, there is also a generic WAL record type, which describes changes to pages in a generic way. This is useful for extensions that provide custom access methods, because they cannot register their own WAL redo routines.

The API for constructing generic WAL records is defined in access/generic_xlog.h and implemented in access/transam/generic_xlog.c.

To perform a WAL-logged data update using the generic WAL record facility, follow these steps:

  1. state = GenericXLogStart(relation) — start construction of a generic WAL record for the given relation.

  2. page = GenericXLogRegisterBuffer(state, buffer, flags) — register a buffer to be modified within the current generic WAL record. This function returns a pointer to a temporary copy of the buffer's page, where modifications should be made. (Do not modify the buffer's contents directly.) The third argument is a bit mask of flags applicable to the operation. Currently the only such flag is GENERIC_XLOG_FULL_IMAGE, which indicates that a full-page image rather than a delta update should be included in the WAL record. Typically this flag would be set if the page is new or has been rewritten completely. GenericXLogRegisterBuffer can be repeated if the WAL-logged action needs to modify multiple pages.

  3. Apply modifications to the page images obtained in the previous step.

  4. GenericXLogFinish(state) — apply the changes to the buffers and emit the generic WAL record.

WAL record construction can be canceled between any of the above steps by calling GenericXLogAbort(state). This will discard all changes to the page image copies.

Please note the following points when using the generic WAL record facility:

  • No direct modifications of buffers are allowed! All modifications must be done in copies acquired from GenericXLogRegisterBuffer(). In other words, code that makes generic WAL records should never call BufferGetPage() for itself. However, it remains the caller's responsibility to pin/unpin and lock/unlock the buffers at appropriate times. Exclusive lock must be held on each target buffer from before GenericXLogRegisterBuffer() until after GenericXLogFinish().

  • Registrations of buffers (step 2) and modifications of page images (step 3) can be mixed freely, i.e., both steps may be repeated in any sequence. Keep in mind that buffers should be registered in the same order in which locks are to be obtained on them during replay.

  • The maximum number of buffers that can be registered for a generic WAL record is MAX_GENERIC_XLOG_PAGES. An error will be thrown if this limit is exceeded.

  • Generic WAL assumes that the pages to be modified have standard layout, and in particular that there is no useful data between pd_lower and pd_upper.

  • Since you are modifying copies of buffer pages, GenericXLogStart() does not start a critical section. Thus, you can safely do memory allocation, error throwing, etc. between GenericXLogStart() and GenericXLogFinish(). The only actual critical section is present inside GenericXLogFinish(). There is no need to worry about calling GenericXLogAbort() during an error exit, either.

  • GenericXLogFinish() takes care of marking buffers dirty and setting their LSNs. You do not need to do this explicitly.

  • For unlogged relations, everything works the same except that no actual WAL record is emitted. Thus, you typically do not need to do any explicit checks for unlogged relations.

  • The generic WAL redo function will acquire exclusive locks to buffers in the same order as they were registered. After redoing all changes, the locks will be released in the same order.

  • If GENERIC_XLOG_FULL_IMAGE is not specified for a registered buffer, the generic WAL record contains a delta between the old and the new page images. This delta is based on byte-by-byte comparison. This is not very compact for the case of moving data within a page, and might be improved in the future.

Prev | Up | Next
62.6. Index Cost Estimation Functions | Home | Chapter 64. B-Tree Indexes


Chapter 64. B-Tree Indexes
Prev Up Part VII. Internals Home Next

Chapter 64. B-Tree Indexes

Table of Contents

64.1. Introduction
64.2. Behavior of B-Tree Operator Classes
64.3. B-Tree Support Functions
64.4. Implementation
64.4.1. B-Tree Structure
64.4.2. Bottom-up Index Deletion
64.4.3. Deduplication
Prev | Up | Next
Chapter 63. Generic WAL Records | Home | 64.1. Introduction


64.1. Introduction
Prev Up Chapter 64. B-Tree Indexes Home Next

64.1. Introduction

PostgreSQL includes an implementation of the standard btree (multi-way balanced tree) index data structure. Any data type that can be sorted into a well-defined linear order can be indexed by a btree index. The only limitation is that an index entry cannot exceed approximately one-third of a page (after TOAST compression, if applicable).

Because each btree operator class imposes a sort order on its data type, btree operator classes (or, really, operator families) have come to be used as PostgreSQL's general representation and understanding of sorting semantics. Therefore, they've acquired some features that go beyond what would be needed just to support btree indexes, and parts of the system that are quite distant from the btree AM make use of them.

Prev | Up | Next
Chapter 64. B-Tree Indexes | Home | 64.2. Behavior of B-Tree Operator Classes


64.2. Behavior of B-Tree Operator Classes
Prev Up Chapter 64. B-Tree Indexes Home Next

64.2. Behavior of B-Tree Operator Classes

As shown in Table 38.3, a btree operator class must provide five comparison operators, <, <=, =, >= and >. One might expect that <> should also be part of the operator class, but it is not, because it would almost never be useful to use a <> WHERE clause in an index search. (For some purposes, the planner treats <> as associated with a btree operator class; but it finds that operator via the = operator's negator link, rather than from pg_amop.)

When several data types share near-identical sorting semantics, their operator classes can be grouped into an operator family. Doing so is advantageous because it allows the planner to make deductions about cross-type comparisons. Each operator class within the family should contain the single-type operators (and associated support functions) for its input data type, while cross-type comparison operators and support functions are “loose” in the family. It is recommendable that a complete set of cross-type operators be included in the family, thus ensuring that the planner can represent any comparison conditions that it deduces from transitivity.

There are some basic assumptions that a btree operator family must satisfy:

  • An = operator must be an equivalence relation; that is, for all non-null values A, B, C of the data type:

    • A = A is true (reflexive law)

    • if A = B, then B = A (symmetric law)

    • if A = B and B = C, then A = C (transitive law)

  • A < operator must be a strong ordering relation; that is, for all non-null values A, B, C:

    • A < A is false (irreflexive law)

    • if A < B and B < C, then A < C (transitive law)

  • Furthermore, the ordering is total; that is, for all non-null values A, B:

    • exactly one of A < B, A = B, and B < A is true (trichotomy law)

    (The trichotomy law justifies the definition of the comparison support function, of course.)

The other three operators are defined in terms of = and < in the obvious way, and must act consistently with them.

For an operator family supporting multiple data types, the above laws must hold when A, B, C are taken from any data types in the family. The transitive laws are the trickiest to ensure, as in cross-type situations they represent statements that the behaviors of two or three different operators are consistent. As an example, it would not work to put float8 and numeric into the same operator family, at least not with the current semantics that numeric values are converted to float8 for comparison to a float8. Because of the limited accuracy of float8, this means there are distinct numeric values that will compare equal to the same float8 value, and thus the transitive law would fail.

Another requirement for a multiple-data-type family is that any implicit or binary-coercion casts that are defined between data types included in the operator family must not change the associated sort ordering.

It should be fairly clear why a btree index requires these laws to hold within a single data type: without them there is no ordering to arrange the keys with. Also, index searches using a comparison key of a different data type require comparisons to behave sanely across two data types. The extensions to three or more data types within a family are not strictly required by the btree index mechanism itself, but the planner relies on them for optimization purposes.

Prev | Up | Next
64.1. Introduction | Home | 64.3. B-Tree Support Functions


64.3. B-Tree Support Functions
Prev Up Chapter 64. B-Tree Indexes Home Next

64.3. B-Tree Support Functions

As shown in Table 38.9, btree defines one required and four optional support functions. The five user-defined methods are:

order

For each combination of data types that a btree operator family provides comparison operators for, it must provide a comparison support function, registered in pg_amproc with support function number 1 and amproclefttype/amprocrighttype equal to the left and right data types for the comparison (i.e., the same data types that the matching operators are registered with in pg_amop). The comparison function must take two non-null values A and B and return an int32 value that is < 0, 0, or > 0 when A < B, A = B, or A > B, respectively. A null result is disallowed: all values of the data type must be comparable. See src/backend/access/nbtree/nbtcompare.c for examples.

If the compared values are of a collatable data type, the appropriate collation OID will be passed to the comparison support function, using the standard PG_GET_COLLATION() mechanism.

sortsupport

Optionally, a btree operator family may provide sort support function(s), registered under support function number 2. These functions allow implementing comparisons for sorting purposes in a more efficient way than naively calling the comparison support function. The APIs involved in this are defined in src/include/utils/sortsupport.h.

in_range

Optionally, a btree operator family may provide in_range support function(s), registered under support function number 3. These are not used during btree index operations; rather, they extend the semantics of the operator family so that it can support window clauses containing the RANGE offset PRECEDING and RANGE offset FOLLOWING frame bound types (see Section 4.2.8). Fundamentally, the extra information provided is how to add or subtract an offset value in a way that is compatible with the family's data ordering.

An in_range function must have the signature

in_range(val type1, base type1, offset type2, sub bool, less bool)
returns bool

val and base must be of the same type, which is one of the types supported by the operator family (i.e., a type for which it provides an ordering). However, offset could be of a different type, which might be one otherwise unsupported by the family. An example is that the built-in time_ops family provides an in_range function that has offset of type interval. A family can provide in_range functions for any of its supported types and one or more offset types. Each in_range function should be entered in pg_amproc with amproclefttype equal to type1 and amprocrighttype equal to type2.

The essential semantics of an in_range function depend on the two Boolean flag parameters. It should add or subtract base and offset, then compare val to the result, as follows:

  • if !sub and !less, return val >= (base + offset)

  • if !sub and less, return val <= (base + offset)

  • if sub and !less, return val >= (base - offset)

  • if sub and less, return val <= (base - offset)

Before doing so, the function should check the sign of offset: if it is less than zero, raise error ERRCODE_INVALID_PRECEDING_OR_FOLLOWING_SIZE (22013) with error text like “invalid preceding or following size in window function”. (This is required by the SQL standard, although nonstandard operator families might perhaps choose to ignore this restriction, since there seems to be little semantic necessity for it.) This requirement is delegated to the in_range function so that the core code needn't understand what “less than zero” means for a particular data type.

An additional expectation is that in_range functions should, if practical, avoid throwing an error if base + offset or base - offset would overflow. The correct comparison result can be determined even if that value would be out of the data type's range. Note that if the data type includes concepts such as “infinity” or “NaN”, extra care may be needed to ensure that in_range's results agree with the normal sort order of the operator family.

The results of the in_range function must be consistent with the sort ordering imposed by the operator family. To be precise, given any fixed values of offset and sub, then:

  • If in_range with less = true is true for some val1 and base, it must be true for every val2 <= val1 with the same base.

  • If in_range with less = true is false for some val1 and base, it must be false for every val2 >= val1 with the same base.

  • If in_range with less = true is true for some val and base1, it must be true for every base2 >= base1 with the same val.

  • If in_range with less = true is false for some val and base1, it must be false for every base2 <= base1 with the same val.

Analogous statements with inverted conditions hold when less = false.

If the type being ordered (type1) is collatable, the appropriate collation OID will be passed to the in_range function, using the standard PG_GET_COLLATION() mechanism.

in_range functions need not handle NULL inputs, and typically will be marked strict.

equalimage

Optionally, a btree operator family may provide equalimage (“equality implies image equality”) support functions, registered under support function number 4. These functions allow the core code to determine when it is safe to apply the btree deduplication optimization. Currently, equalimage functions are only called when building or rebuilding an index.

An equalimage function must have the signature

equalimage(opcintype oid) returns bool

The return value is static information about an operator class and collation. Returning true indicates that the order function for the operator class is guaranteed to only return 0 (“arguments are equal”) when its A and B arguments are also interchangeable without any loss of semantic information. Not registering an equalimage function or returning false indicates that this condition cannot be assumed to hold.

The opcintype argument is the pg_type.oid of the data type that the operator class indexes. This is a convenience that allows reuse of the same underlying equalimage function across operator classes. If opcintype is a collatable data type, the appropriate collation OID will be passed to the equalimage function, using the standard PG_GET_COLLATION() mechanism.

As far as the operator class is concerned, returning true indicates that deduplication is safe (or safe for the collation whose OID was passed to its equalimage function). However, the core code will only deem deduplication safe for an index when every indexed column uses an operator class that registers an equalimage function, and each function actually returns true when called.

Image equality is almost the same condition as simple bitwise equality. There is one subtle difference: When indexing a varlena data type, the on-disk representation of two image equal datums may not be bitwise equal due to inconsistent application of TOAST compression on input. Formally, when an operator class's equalimage function returns true, it is safe to assume that the datum_image_eq() C function will always agree with the operator class's order function (provided that the same collation OID is passed to both the equalimage and order functions).

The core code is fundamentally unable to deduce anything about the “equality implies image equality” status of an operator class within a multiple-data-type family based on details from other operator classes in the same family. Also, it is not sensible for an operator family to register a cross-type equalimage function, and attempting to do so will result in an error. This is because “equality implies image equality” status does not just depend on sorting/equality semantics, which are more or less defined at the operator family level. In general, the semantics that one particular data type implements must be considered separately.

The convention followed by the operator classes included with the core PostgreSQL distribution is to register a stock, generic equalimage function. Most operator classes register btequalimage(), which indicates that deduplication is safe unconditionally. Operator classes for collatable data types such as text register btvarstrequalimage(), which indicates that deduplication is safe with deterministic collations. Best practice for third-party extensions is to register their own custom function to retain control.

options

Optionally, a B-tree operator family may provide options (“operator class specific options”) support functions, registered under support function number 5. These functions define a set of user-visible parameters that control operator class behavior.

An options support function must have the signature

options(relopts local_relopts *) returns void

The function is passed a pointer to a local_relopts struct, which needs to be filled with a set of operator class specific options. The options can be accessed from other support functions using the PG_HAS_OPCLASS_OPTIONS() and PG_GET_OPCLASS_OPTIONS() macros.

Currently, no B-Tree operator class has an options support function. B-tree doesn't allow flexible representation of keys like GiST, SP-GiST, GIN and BRIN do. So, options probably doesn't have much application in the current B-tree index access method. Nevertheless, this support function was added to B-tree for uniformity, and will probably find uses during further evolution of B-tree in PostgreSQL.

Prev | Up | Next
64.2. Behavior of B-Tree Operator Classes | Home | 64.4. Implementation


64.4. Implementation
Prev Up Chapter 64. B-Tree Indexes Home Next

64.4. Implementation

64.4.1. B-Tree Structure
64.4.2. Bottom-up Index Deletion
64.4.3. Deduplication

This section covers B-Tree index implementation details that may be of use to advanced users. See src/backend/access/nbtree/README in the source distribution for a much more detailed, internals-focused description of the B-Tree implementation.

64.4.1. B-Tree Structure

PostgreSQL B-Tree indexes are multi-level tree structures, where each level of the tree can be used as a doubly-linked list of pages. A single metapage is stored in a fixed position at the start of the first segment file of the index. All other pages are either leaf pages or internal pages. Leaf pages are the pages on the lowest level of the tree. All other levels consist of internal pages. Each leaf page contains tuples that point to table rows. Each internal page contains tuples that point to the next level down in the tree. Typically, over 99% of all pages are leaf pages. Both internal pages and leaf pages use the standard page format described in Section 70.6.

New leaf pages are added to a B-Tree index when an existing leaf page cannot fit an incoming tuple. A page split operation makes room for items that originally belonged on the overflowing page by moving a portion of the items to a new page. Page splits must also insert a new downlink to the new page in the parent page, which may cause the parent to split in turn. Page splits “cascade upwards” in a recursive fashion. When the root page finally cannot fit a new downlink, a root page split operation takes place. This adds a new level to the tree structure by creating a new root page that is one level above the original root page.

64.4.2. Bottom-up Index Deletion

B-Tree indexes are not directly aware that under MVCC, there might be multiple extant versions of the same logical table row; to an index, each tuple is an independent object that needs its own index entry. “Version churn” tuples may sometimes accumulate and adversely affect query latency and throughput. This typically occurs with UPDATE-heavy workloads where most individual updates cannot apply the HOT optimization. Changing the value of only one column covered by one index during an UPDATE always necessitates a new set of index tuples — one for each and every index on the table. Note in particular that this includes indexes that were not “logically modified” by the UPDATE. All indexes will need a successor physical index tuple that points to the latest version in the table. Each new tuple within each index will generally need to coexist with the original “updated” tuple for a short period of time (typically until shortly after the UPDATE transaction commits).

B-Tree indexes incrementally delete version churn index tuples by performing bottom-up index deletion passes. Each deletion pass is triggered in reaction to an anticipated “version churn page split”. This only happens with indexes that are not logically modified by UPDATE statements, where concentrated build up of obsolete versions in particular pages would occur otherwise. A page split will usually be avoided, though it's possible that certain implementation-level heuristics will fail to identify and delete even one garbage index tuple (in which case a page split or deduplication pass resolves the issue of an incoming new tuple not fitting on a leaf page). The worst-case number of versions that any index scan must traverse (for any single logical row) is an important contributor to overall system responsiveness and throughput. A bottom-up index deletion pass targets suspected garbage tuples in a single leaf page based on qualitative distinctions involving logical rows and versions. This contrasts with the “top-down” index cleanup performed by autovacuum workers, which is triggered when certain quantitative table-level thresholds are exceeded (see Section 25.1.6).

Note

Not all deletion operations that are performed within B-Tree indexes are bottom-up deletion operations. There is a distinct category of index tuple deletion: simple index tuple deletion. This is a deferred maintenance operation that deletes index tuples that are known to be safe to delete (those whose item identifier's LP_DEAD bit is already set). Like bottom-up index deletion, simple index deletion takes place at the point that a page split is anticipated as a way of avoiding the split.

Simple deletion is opportunistic in the sense that it can only take place when recent index scans set the LP_DEAD bits of affected items in passing. Prior to PostgreSQL 14, the only category of B-Tree deletion was simple deletion. The main differences between it and bottom-up deletion are that only the former is opportunistically driven by the activity of passing index scans, while only the latter specifically targets version churn from UPDATEs that do not logically modify indexed columns.

Bottom-up index deletion performs the vast majority of all garbage index tuple cleanup for particular indexes with certain workloads. This is expected with any B-Tree index that is subject to significant version churn from UPDATEs that rarely or never logically modify the columns that the index covers. The average and worst-case number of versions per logical row can be kept low purely through targeted incremental deletion passes. It's quite possible that the on-disk size of certain indexes will never increase by even one single page/block despite constant version churn from UPDATEs. Even then, an exhaustive “clean sweep” by a VACUUM operation (typically run in an autovacuum worker process) will eventually be required as a part of collective cleanup of the table and each of its indexes.

Unlike VACUUM, bottom-up index deletion does not provide any strong guarantees about how old the oldest garbage index tuple may be. No index can be permitted to retain “floating garbage” index tuples that became dead prior to a conservative cutoff point shared by the table and all of its indexes collectively. This fundamental table-level invariant makes it safe to recycle table TIDs. This is how it is possible for distinct logical rows to reuse the same table TID over time (though this can never happen with two logical rows whose lifetimes span the same VACUUM cycle).

64.4.3. Deduplication

A duplicate is a leaf page tuple (a tuple that points to a table row) where all indexed key columns have values that match corresponding column values from at least one other leaf page tuple in the same index. Duplicate tuples are quite common in practice. B-Tree indexes can use a special, space-efficient representation for duplicates when an optional technique is enabled: deduplication.

Deduplication works by periodically merging groups of duplicate tuples together, forming a single posting list tuple for each group. The column key value(s) only appear once in this representation. This is followed by a sorted array of TIDs that point to rows in the table. This significantly reduces the storage size of indexes where each value (or each distinct combination of column values) appears several times on average. The latency of queries can be reduced significantly. Overall query throughput may increase significantly. The overhead of routine index vacuuming may also be reduced significantly.

Note

B-Tree deduplication is just as effective with “duplicates” that contain a NULL value, even though NULL values are never equal to each other according to the = member of any B-Tree operator class. As far as any part of the implementation that understands the on-disk B-Tree structure is concerned, NULL is just another value from the domain of indexed values.

The deduplication process occurs lazily, when a new item is inserted that cannot fit on an existing leaf page, though only when index tuple deletion could not free sufficient space for the new item (typically deletion is briefly considered and then skipped over). Unlike GIN posting list tuples, B-Tree posting list tuples do not need to expand every time a new duplicate is inserted; they are merely an alternative physical representation of the original logical contents of the leaf page. This design prioritizes consistent performance with mixed read-write workloads. Most client applications will at least see a moderate performance benefit from using deduplication. Deduplication is enabled by default.

CREATE INDEX and REINDEX apply deduplication to create posting list tuples, though the strategy they use is slightly different. Each group of duplicate ordinary tuples encountered in the sorted input taken from the table is merged into a posting list tuple before being added to the current pending leaf page. Individual posting list tuples are packed with as many TIDs as possible. Leaf pages are written out in the usual way, without any separate deduplication pass. This strategy is well-suited to CREATE INDEX and REINDEX because they are once-off batch operations.

Write-heavy workloads that don't benefit from deduplication due to having few or no duplicate values in indexes will incur a small, fixed performance penalty (unless deduplication is explicitly disabled). The deduplicate_items storage parameter can be used to disable deduplication within individual indexes. There is never any performance penalty with read-only workloads, since reading posting list tuples is at least as efficient as reading the standard tuple representation. Disabling deduplication isn't usually helpful.

It is sometimes possible for unique indexes (as well as unique constraints) to use deduplication. This allows leaf pages to temporarily “absorb” extra version churn duplicates. Deduplication in unique indexes augments bottom-up index deletion, especially in cases where a long-running transaction holds a snapshot that blocks garbage collection. The goal is to buy time for the bottom-up index deletion strategy to become effective again. Delaying page splits until a single long-running transaction naturally goes away can allow a bottom-up deletion pass to succeed where an earlier deletion pass failed.

Tip

A special heuristic is applied to determine whether a deduplication pass in a unique index should take place. It can often skip straight to splitting a leaf page, avoiding a performance penalty from wasting cycles on unhelpful deduplication passes. If you're concerned about the overhead of deduplication, consider setting deduplicate_items = off selectively. Leaving deduplication enabled in unique indexes has little downside.

Deduplication cannot be used in all cases due to implementation-level restrictions. Deduplication safety is determined when CREATE INDEX or REINDEX is run.

Note that deduplication is deemed unsafe and cannot be used in the following cases involving semantically significant differences among equal datums:

  • text, varchar, and char cannot use deduplication when a nondeterministic collation is used. Case and accent differences must be preserved among equal datums.

  • numeric cannot use deduplication. Numeric display scale must be preserved among equal datums.

  • jsonb cannot use deduplication, since the jsonb B-Tree operator class uses numeric internally.

  • float4 and float8 cannot use deduplication. These types have distinct representations for -0 and 0, which are nevertheless considered equal. This difference must be preserved.

There is one further implementation-level restriction that may be lifted in a future version of PostgreSQL:

  • Container types (such as composite types, arrays, or range types) cannot use deduplication.

There is one further implementation-level restriction that applies regardless of the operator class or collation used:

  • INCLUDE indexes can never use deduplication.

Prev | Up | Next
64.3. B-Tree Support Functions | Home | Chapter 65. GiST Indexes


Chapter 65. GiST Indexes
Prev Up Part VII. Internals Home Next

Chapter 65. GiST Indexes

Table of Contents

65.1. Introduction
65.2. Built-in Operator Classes
65.3. Extensibility
65.4. Implementation
65.4.1. GiST Index Build Methods
65.5. Examples
Prev | Up | Next
64.4. Implementation | Home | 65.1. Introduction


65.1. Introduction
Prev Up Chapter 65. GiST Indexes Home Next

65.1. Introduction

GiST stands for Generalized Search Tree. It is a balanced, tree-structured access method, that acts as a base template in which to implement arbitrary indexing schemes. B-trees, R-trees and many other indexing schemes can be implemented in GiST.

One advantage of GiST is that it allows the development of custom data types with the appropriate access methods, by an expert in the domain of the data type, rather than a database expert.

Some of the information here is derived from the University of California at Berkeley's GiST Indexing Project web site and Marcel Kornacker's thesis, Access Methods for Next-Generation Database Systems. The GiST implementation in PostgreSQL is primarily maintained by Teodor Sigaev and Oleg Bartunov, and there is more information on their web site.

Prev | Up | Next
Chapter 65. GiST Indexes | Home | 65.2. Built-in Operator Classes


65.2. Built-in Operator Classes
Prev Up Chapter 65. GiST Indexes Home Next

65.2. Built-in Operator Classes

The core PostgreSQL distribution includes the GiST operator classes shown in Table 65.1. (Some of the optional modules described in Appendix F provide additional GiST operator classes.)

Table 65.1. Built-in GiST Operator Classes

Name | Indexable Operators | Ordering Operators
box_ops | << (box, box) | <-> (box, point)
 | &< (box, box)
 | && (box, box)
 | &> (box, box)
 | >> (box, box)
 | ~= (box, box)
 | @> (box, box)
 | <@ (box, box)
 | &<| (box, box)
 | <<| (box, box)
 | |>> (box, box)
 | |&> (box, box)
 | ~ (box, box)
 | @ (box, box)
circle_ops | << (circle, circle) | <-> (circle, point)
 | &< (circle, circle)
 | &> (circle, circle)
 | >> (circle, circle)
 | <@ (circle, circle)
 | @> (circle, circle)
 | ~= (circle, circle)
 | && (circle, circle)
 | |>> (circle, circle)
 | <<| (circle, circle)
 | &<| (circle, circle)
 | |&> (circle, circle)
 | @ (circle, circle)
 | ~ (circle, circle)
inet_ops | << (inet, inet) |
 | <<= (inet, inet)
 | >> (inet, inet)
 | >>= (inet, inet)
 | = (inet, inet)
 | <> (inet, inet)
 | < (inet, inet)
 | <= (inet, inet)
 | > (inet, inet)
 | >= (inet, inet)
 | && (inet, inet)
multirange_ops | = (anymultirange, anymultirange) |
 | && (anymultirange, anymultirange)
 | && (anymultirange, anyrange)
 | @> (anymultirange, anyelement)
 | @> (anymultirange, anymultirange)
 | @> (anymultirange, anyrange)
 | <@ (anymultirange, anymultirange)
 | <@ (anymultirange, anyrange)
 | << (anymultirange, anymultirange)
 | << (anymultirange, anyrange)
 | >> (anymultirange, anymultirange)
 | >> (anymultirange, anyrange)
 | &< (anymultirange, anymultirange)
 | &< (anymultirange, anyrange)
 | &> (anymultirange, anymultirange)
 | &> (anymultirange, anyrange)
 | -|- (anymultirange, anymultirange)
 | -|- (anymultirange, anyrange)
point_ops | |>> (point, point) | <-> (point, point)
 | << (point, point)
 | >> (point, point)
 | <<| (point, point)
 | ~= (point, point)
 | <@ (point, box)
 | <@ (point, polygon)
 | <@ (point, circle)
poly_ops | << (polygon, polygon) | <-> (polygon, point)
 | &< (polygon, polygon)
 | &> (polygon, polygon)
 | >> (polygon, polygon)
 | <@ (polygon, polygon)
 | @> (polygon, polygon)
 | ~= (polygon, polygon)
 | && (polygon, polygon)
 | <<| (polygon, polygon)
 | &<| (polygon, polygon)
 | |&> (polygon, polygon)
 | |>> (polygon, polygon)
 | @ (polygon, polygon)
 | ~ (polygon, polygon)
range_ops | = (anyrange, anyrange) |
 | && (anyrange, anyrange)
 | && (anyrange, anymultirange)
 | @> (anyrange, anyelement)
 | @> (anyrange, anyrange)
 | @> (anyrange, anymultirange)
 | <@ (anyrange, anyrange)
 | <@ (anyrange, anymultirange)
 | << (anyrange, anyrange)
 | << (anyrange, anymultirange)
 | >> (anyrange, anyrange)
 | >> (anyrange, anymultirange)
 | &< (anyrange, anyrange)
 | &< (anyrange, anymultirange)
 | &> (anyrange, anyrange)
 | &> (anyrange, anymultirange)
 | -|- (anyrange, anyrange)
 | -|- (anyrange, anymultirange)
tsquery_ops | <@ (tsquery, tsquery) |
 | @> (tsquery, tsquery)
tsvector_ops | @@ (tsvector, tsquery) |

For historical reasons, the inet_ops operator class is not the default class for types inet and cidr. To use it, mention the class name in CREATE INDEX, for example

CREATE INDEX ON my_table USING GIST (my_inet_column inet_ops);

Prev | Up | Next
65.1. Introduction | Home | 65.3. Extensibility


65.3. Extensibility
Prev Up Chapter 65. GiST Indexes Home Next

65.3. Extensibility

Traditionally, implementing a new index access method meant a lot of difficult work. It was necessary to understand the inner workings of the database, such as the lock manager and Write-Ahead Log. The GiST interface has a high level of abstraction, requiring the access method implementer only to implement the semantics of the data type being accessed. The GiST layer itself takes care of concurrency, logging and searching the tree structure.

This extensibility should not be confused with the extensibility of the other standard search trees in terms of the data they can handle. For example, PostgreSQL supports extensible B-trees and hash indexes. That means that you can use PostgreSQL to build a B-tree or hash over any data type you want. But B-trees only support range predicates (<, =, >), and hash indexes only support equality queries.

So if you index, say, an image collection with a PostgreSQL B-tree, you can only issue queries such as “is imagex equal to imagey”, “is imagex less than imagey” and “is imagex greater than imagey”. Depending on how you define “equals”, “less than” and “greater than” in this context, this could be useful. However, by using a GiST based index, you could create ways to ask domain-specific questions, perhaps “find all images of horses” or “find all over-exposed images”.

All it takes to get a GiST access method up and running is to implement several user-defined methods, which define the behavior of keys in the tree. Of course these methods have to be pretty fancy to support fancy queries, but for all the standard queries (B-trees, R-trees, etc.) they're relatively straightforward. In short, GiST combines extensibility along with generality, code reuse, and a clean interface.

There are five methods that an index operator class for GiST must provide, and six that are optional. Correctness of the index is ensured by proper implementation of the same, consistent and union methods, while efficiency (size and speed) of the index will depend on the penalty and picksplit methods. Two optional methods are compress and decompress, which allow an index to have internal tree data of a different type than the data it indexes. The leaves are to be of the indexed data type, while the other tree nodes can be of any C struct (but you still have to follow PostgreSQL data type rules here, see about varlena for variable sized data). If the tree's internal data type exists at the SQL level, the STORAGE option of the CREATE OPERATOR CLASS command can be used. The optional eighth method is distance, which is needed if the operator class wishes to support ordered scans (nearest-neighbor searches). The optional ninth method fetch is needed if the operator class wishes to support index-only scans, except when the compress method is omitted. The optional tenth method options is needed if the operator class has user-specified parameters. The optional eleventh method sortsupport is used to speed up building a GiST index.

consistent

Given an index entry p and a query value q, this function determines whether the index entry is “consistent” with the query; that is, could the predicate “indexed_column indexable_operator q” be true for any row represented by the index entry? For a leaf index entry this is equivalent to testing the indexable condition, while for an internal tree node this determines whether it is necessary to scan the subtree of the index represented by the tree node. When the result is true, a recheck flag must also be returned. This indicates whether the predicate is certainly true or only possibly true. If recheck = false then the index has tested the predicate condition exactly, whereas if recheck = true the row is only a candidate match. In that case the system will automatically evaluate the indexable_operator against the actual row value to see if it is really a match. This convention allows GiST to support both lossless and lossy index structures.

The SQL declaration of the function must look like this:

CREATE OR REPLACE FUNCTION my_consistent(internal, data_type, smallint, oid, internal)
RETURNS bool
AS 'MODULE_PATHNAME'
LANGUAGE C STRICT;

And the matching code in the C module could then follow this skeleton:

PG_FUNCTION_INFO_V1(my_consistent);

Datum
my_consistent(PG_FUNCTION_ARGS)
{
    GISTENTRY  *entry = (GISTENTRY *) PG_GETARG_POINTER(0);
    data_type  *query = PG_GETARG_DATA_TYPE_P(1);
    StrategyNumber strategy = (StrategyNumber) PG_GETARG_UINT16(2);
    /* Oid subtype = PG_GETARG_OID(3); */
    bool       *recheck = (bool *) PG_GETARG_POINTER(4);
    data_type  *key = DatumGetDataType(entry->key);
    bool        retval;

    /*
     * determine return value as a function of strategy, key and query.
     *
     * Use GIST_LEAF(entry) to know where you're called in the index tree,
     * which comes handy when supporting the = operator for example (you could
     * check for non empty union() in non-leaf nodes and equality in leaf
     * nodes).
     */

    *recheck = true;        /* or false if check is exact */

    PG_RETURN_BOOL(retval);
}

Here, key is an element in the index and query the value being looked up in the index. The StrategyNumber parameter indicates which operator of your operator class is being applied — it matches one of the operator numbers in the CREATE OPERATOR CLASS command.

Depending on which operators you have included in the class, the data type of query could vary with the operator, since it will be whatever type is on the right-hand side of the operator, which might be different from the indexed data type appearing on the left-hand side. (The above code skeleton assumes that only one type is possible; if not, fetching the query argument value would have to depend on the operator.) It is recommended that the SQL declaration of the consistent function use the opclass's indexed data type for the query argument, even though the actual type might be something else depending on the operator.

union

This method consolidates information in the tree. Given a set of entries, this function generates a new index entry that represents all the given entries.

The SQL declaration of the function must look like this:

CREATE OR REPLACE FUNCTION my_union(internal, internal)
RETURNS storage_type
AS 'MODULE_PATHNAME'
LANGUAGE C STRICT;

And the matching code in the C module could then follow this skeleton:

PG_FUNCTION_INFO_V1(my_union);

Datum
my_union(PG_FUNCTION_ARGS)
{
    GistEntryVector *entryvec = (GistEntryVector *) PG_GETARG_POINTER(0);
    GISTENTRY  *ent = entryvec->vector;
    data_type  *out,
               *tmp,
               *old;
    int         numranges,
                i = 0;

    numranges = entryvec->n;
    tmp = DatumGetDataType(ent[0].key);
    out = tmp;

    if (numranges == 1)
    {
        out = data_type_deep_copy(tmp);

        PG_RETURN_DATA_TYPE_P(out);
    }

    for (i = 1; i < numranges; i++)
    {
        old = out;
        tmp = DatumGetDataType(ent[i].key);
        out = my_union_implementation(out, tmp);
    }

    PG_RETURN_DATA_TYPE_P(out);
}

As you can see, in this skeleton we're dealing with a data type where union(X, Y, Z) = union(union(X, Y), Z). It's easy enough to support data types where this is not the case, by implementing the proper union algorithm in this GiST support method.

The result of the union function must be a value of the index's storage type, whatever that is (it might or might not be different from the indexed column's type). The union function should return a pointer to newly palloc()ed memory. You can't just return the input value as-is, even if there is no type change.

As shown above, the union function's first internal argument is actually a GistEntryVector pointer. The second argument is a pointer to an integer variable, which can be ignored. (It used to be required that the union function store the size of its result value into that variable, but this is no longer necessary.)

compress

Converts a data item into a format suitable for physical storage in an index page. If the compress method is omitted, data items are stored in the index without modification.

The SQL declaration of the function must look like this:

CREATE OR REPLACE FUNCTION my_compress(internal)
RETURNS internal
AS 'MODULE_PATHNAME'
LANGUAGE C STRICT;

And the matching code in the C module could then follow this skeleton:

PG_FUNCTION_INFO_V1(my_compress);

Datum
my_compress(PG_FUNCTION_ARGS)
{
    GISTENTRY  *entry = (GISTENTRY *) PG_GETARG_POINTER(0);
    GISTENTRY  *retval;

    if (entry->leafkey)
    {
        /* replace entry->key with a compressed version */
        compressed_data_type *compressed_data = palloc(sizeof(compressed_data_type));

        /* fill *compressed_data from entry->key ... */

        retval = palloc(sizeof(GISTENTRY));
        gistentryinit(*retval, PointerGetDatum(compressed_data),
                      entry->rel, entry->page, entry->offset, FALSE);
    }
    else
    {
        /* typically we needn't do anything with non-leaf entries */
        retval = entry;
    }

    PG_RETURN_POINTER(retval);
}

You have to adapt compressed_data_type to the specific type you're converting to in order to compress your leaf nodes, of course.

decompress

Converts the stored representation of a data item into a format that can be manipulated by the other GiST methods in the operator class. If the decompress method is omitted, it is assumed that the other GiST methods can work directly on the stored data format. (decompress is not necessarily the reverse of the compress method; in particular, if compress is lossy then it's impossible for decompress to exactly reconstruct the original data. decompress is not necessarily equivalent to fetch, either, since the other GiST methods might not require full reconstruction of the data.)

The SQL declaration of the function must look like this:

CREATE OR REPLACE FUNCTION my_decompress(internal)
RETURNS internal
AS 'MODULE_PATHNAME'
LANGUAGE C STRICT;

And the matching code in the C module could then follow this skeleton:

PG_FUNCTION_INFO_V1(my_decompress);

Datum
my_decompress(PG_FUNCTION_ARGS)
{
    PG_RETURN_POINTER(PG_GETARG_POINTER(0));
}

The above skeleton is suitable for the case where no decompression is needed. (But, of course, omitting the method altogether is even easier, and is recommended in such cases.)

penalty

Returns a value indicating the “cost” of inserting the new entry into a particular branch of the tree. Items will be inserted down the path of least penalty in the tree. Values returned by penalty should be non-negative. If a negative value is returned, it will be treated as zero.

The SQL declaration of the function must look like this:

CREATE OR REPLACE FUNCTION my_penalty(internal, internal, internal)
RETURNS internal
AS 'MODULE_PATHNAME'
LANGUAGE C STRICT;  -- in some cases penalty functions need not be strict

And the matching code in the C module could then follow this skeleton:

PG_FUNCTION_INFO_V1(my_penalty);

Datum
my_penalty(PG_FUNCTION_ARGS)
{
    GISTENTRY  *origentry = (GISTENTRY *) PG_GETARG_POINTER(0);
    GISTENTRY  *newentry = (GISTENTRY *) PG_GETARG_POINTER(1);
    float      *penalty = (float *) PG_GETARG_POINTER(2);
    data_type  *orig = DatumGetDataType(origentry->key);
    data_type  *new = DatumGetDataType(newentry->key);

    *penalty = my_penalty_implementation(orig, new);
    PG_RETURN_POINTER(penalty);
}

For historical reasons, the penalty function doesn't just return a float result; instead it has to store the value at the location indicated by the third argument. The return value per se is ignored, though it's conventional to pass back the address of that argument.

The penalty function is crucial to good performance of the index. It'll get used at insertion time to determine which branch to follow when choosing where to add the new entry in the tree. At query time, the more balanced the index, the quicker the lookup.

picksplit

When an index page split is necessary, this function decides which entries on the page are to stay on the old page, and which are to move to the new page.

The SQL declaration of the function must look like this:

CREATE OR REPLACE FUNCTION my_picksplit(internal, internal)
RETURNS internal
AS 'MODULE_PATHNAME'
LANGUAGE C STRICT;

And the matching code in the C module could then follow this skeleton:

PG_FUNCTION_INFO_V1(my_picksplit);

Datum
my_picksplit(PG_FUNCTION_ARGS)
{
    GistEntryVector *entryvec = (GistEntryVector *) PG_GETARG_POINTER(0);
    GIST_SPLITVEC *v = (GIST_SPLITVEC *) PG_GETARG_POINTER(1);
    OffsetNumber maxoff = entryvec->n - 1;
    GISTENTRY  *ent = entryvec->vector;
    int         i,
                nbytes;
    OffsetNumber *left,
               *right;
    data_type  *tmp_union;
    data_type  *unionL;
    data_type  *unionR;
    GISTENTRY **raw_entryvec;

    maxoff = entryvec->n - 1;
    nbytes = (maxoff + 1) * sizeof(OffsetNumber);

    v->spl_left = (OffsetNumber *) palloc(nbytes);
    left = v->spl_left;
    v->spl_nleft = 0;

    v->spl_right = (OffsetNumber *) palloc(nbytes);
    right = v->spl_right;
    v->spl_nright = 0;

    unionL = NULL;
    unionR = NULL;

    /* Initialize the raw entry vector. */
    raw_entryvec = (GISTENTRY **) malloc(entryvec->n * sizeof(void *));
    for (i = FirstOffsetNumber; i <= maxoff; i = OffsetNumberNext(i))
        raw_entryvec[i] = &(entryvec->vector[i]);

    for (i = FirstOffsetNumber; i <= maxoff; i = OffsetNumberNext(i))
    {
        int         real_index = raw_entryvec[i] - entryvec->vector;

        tmp_union = DatumGetDataType(entryvec->vector[real_index].key);
        Assert(tmp_union != NULL);

        /*
         * Choose where to put the index entries and update unionL and unionR
         * accordingly. Append the entries to either v->spl_left or
         * v->spl_right, and care about the counters.
         */

        if (my_choice_is_left(unionL, curl, unionR, curr))
        {
            if (unionL == NULL)
                unionL = tmp_union;
            else
                unionL = my_union_implementation(unionL, tmp_union);

            *left = real_index;
            ++left;
            ++(v->spl_nleft);
        }
        else
        {
            /*
             * Same on the right
             */
        }
    }

    v->spl_ldatum = DataTypeGetDatum(unionL);
    v->spl_rdatum = DataTypeGetDatum(unionR);
    PG_RETURN_POINTER(v);
}

Notice that the picksplit function's result is delivered by modifying the passed-in v structure. The return value per se is ignored, though it's conventional to pass back the address of v.

Like penalty, the picksplit function is crucial to good performance of the index. Designing suitable penalty and picksplit implementations is where the challenge of implementing well-performing GiST indexes lies.

same

Returns true if two index entries are identical, false otherwise. (An “index entry” is a value of the index's storage type, not necessarily the original indexed column's type.)

The SQL declaration of the function must look like this:

CREATE OR REPLACE FUNCTION my_same(storage_type, storage_type, internal)
RETURNS internal
AS 'MODULE_PATHNAME'
LANGUAGE C STRICT;

And the matching code in the C module could then follow this skeleton:

PG_FUNCTION_INFO_V1(my_same);

Datum
my_same(PG_FUNCTION_ARGS)
{
    prefix_range *v1 = PG_GETARG_PREFIX_RANGE_P(0);
    prefix_range *v2 = PG_GETARG_PREFIX_RANGE_P(1);
    bool       *result = (bool *) PG_GETARG_POINTER(2);

    *result = my_eq(v1, v2);
    PG_RETURN_POINTER(result);
}

For historical reasons, the same function doesn't just return a Boolean result; instead it has to store the flag at the location indicated by the third argument. The return value per se is ignored, though it's conventional to pass back the address of that argument.

distance

Given an index entry p and a query value q, this function determines the index entry's “distance” from the query value. This function must be supplied if the operator class contains any ordering operators. A query using the ordering operator will be implemented by returning index entries with the smallest “distance” values first, so the results must be consistent with the operator's semantics. For a leaf index entry the result just represents the distance to the index entry; for an internal tree node, the result must be the smallest distance that any child entry could have.

The SQL declaration of the function must look like this:

CREATE OR REPLACE FUNCTION my_distance(internal, data_type, smallint, oid, internal)
RETURNS float8
AS 'MODULE_PATHNAME'
LANGUAGE C STRICT;

And the matching code in the C module could then follow this skeleton:

PG_FUNCTION_INFO_V1(my_distance);

Datum
my_distance(PG_FUNCTION_ARGS)
{
    GISTENTRY  *entry = (GISTENTRY *) PG_GETARG_POINTER(0);
    data_type  *query = PG_GETARG_DATA_TYPE_P(1);
    StrategyNumber strategy = (StrategyNumber) PG_GETARG_UINT16(2);
    /* Oid subtype = PG_GETARG_OID(3); */
    /* bool *recheck = (bool *) PG_GETARG_POINTER(4); */
    data_type  *key = DatumGetDataType(entry->key);
    double      retval;

    /*
     * determine return value as a function of strategy, key and query.
     */

    PG_RETURN_FLOAT8(retval);
}

The arguments to the distance function are identical to the arguments of the consistent function.

Some approximation is allowed when determining the distance, so long as the result is never greater than the entry's actual distance. Thus, for example, distance to a bounding box is usually sufficient in geometric applications. For an internal tree node, the distance returned must not be greater than the distance to any of the child nodes. If the returned distance is not exact, the function must set *recheck to true. (This is not necessary for internal tree nodes; for them, the calculation is always assumed to be inexact.) In this case the executor will calculate the accurate distance after fetching the tuple from the heap, and reorder the tuples if necessary.

If the distance function returns *recheck = true for any leaf node, the original ordering operator's return type must be float8 or float4, and the distance function's result values must be comparable to those of the original ordering operator, since the executor will sort using both distance function results and recalculated ordering-operator results. Otherwise, the distance function's result values can be any finite float8 values, so long as the relative order of the result values matches the order returned by the ordering operator. (Infinity and minus infinity are used internally to handle cases such as nulls, so it is not recommended that distance functions return these values.)

fetch

Converts the compressed index representation of a data item into the original data type, for index-only scans. The returned data must be an exact, non-lossy copy of the originally indexed value.

The SQL declaration of the function must look like this:

CREATE OR REPLACE FUNCTION my_fetch(internal)
RETURNS internal
AS 'MODULE_PATHNAME'
LANGUAGE C STRICT;

The argument is a pointer to a GISTENTRY struct. On entry, its key field contains a non-NULL leaf datum in compressed form. The return value is another GISTENTRY struct, whose key field contains the same datum in its original, uncompressed form. If the opclass's compress function does nothing for leaf entries, the fetch method can return the argument as-is. Or, if the opclass does not have a compress function, the fetch method can be omitted as well, since it would necessarily be a no-op.

The matching code in the C module could then follow this skeleton:

PG_FUNCTION_INFO_V1(my_fetch);

Datum
my_fetch(PG_FUNCTION_ARGS)
{
    GISTENTRY  *entry = (GISTENTRY *) PG_GETARG_POINTER(0);
    input_data_type *in = DatumGetPointer(entry->key);
    fetched_data_type *fetched_data;
    GISTENTRY  *retval;

    retval = palloc(sizeof(GISTENTRY));
    fetched_data = palloc(sizeof(fetched_data_type));

    /*
     * Convert 'fetched_data' into the a Datum of the original datatype.
     */

    /* fill *retval from fetched_data. */
    gistentryinit(*retval, PointerGetDatum(converted_datum),
                  entry->rel, entry->page, entry->offset, FALSE);

    PG_RETURN_POINTER(retval);
}

If the compress method is lossy for leaf entries, the operator class cannot support index-only scans, and must not define a fetch function.

options

Allows definition of user-visible parameters that control operator class behavior.

The SQL declaration of the function must look like this:

CREATE OR REPLACE FUNCTION my_options(internal)
RETURNS void
AS 'MODULE_PATHNAME'
LANGUAGE C STRICT;

The function is passed a pointer to a local_relopts struct, which needs to be filled with a set of operator class specific options. The options can be accessed from other support functions using the PG_HAS_OPCLASS_OPTIONS() and PG_GET_OPCLASS_OPTIONS() macros.

An example implementation of my_options() and parameters use from other support functions are given below:

typedef enum MyEnumType
{
    MY_ENUM_ON,
    MY_ENUM_OFF,
    MY_ENUM_AUTO
} MyEnumType;

typedef struct
{
    int32   vl_len_;    /* varlena header (do not touch directly!) */
    int     int_param;  /* integer parameter */
    double  real_param; /* real parameter */
    MyEnumType enum_param; /* enum parameter */
    int     str_param;  /* string parameter */
} MyOptionsStruct;

/* String representation of enum values */
static relopt_enum_elt_def myEnumValues[] =
{
    {"on", MY_ENUM_ON},
    {"off", MY_ENUM_OFF},
    {"auto", MY_ENUM_AUTO},
    {(const char *) NULL}   /* list terminator */
};

static char *str_param_default = "default";

/*
 * Sample validator: checks that string is not longer than 8 bytes.
 */
static void
validate_my_string_relopt(const char *value)
{
    if (strlen(value) > 8)
        ereport(ERROR,
                (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
                 errmsg("str_param must be at most 8 bytes")));
}

/*
 * Sample filler: switches characters to lower case.
 */
static Size
fill_my_string_relopt(const char *value, void *ptr)
{
    char   *tmp = str_tolower(value, strlen(value), DEFAULT_COLLATION_OID);
    int     len = strlen(tmp);

    if (ptr)
        strcpy((char *) ptr, tmp);

    pfree(tmp);
    return len + 1;
}

PG_FUNCTION_INFO_V1(my_options);

Datum
my_options(PG_FUNCTION_ARGS)
{
    local_relopts *relopts = (local_relopts *) PG_GETARG_POINTER(0);

    init_local_reloptions(relopts, sizeof(MyOptionsStruct));
    add_local_int_reloption(relopts, "int_param", "integer parameter",
                            100, 0, 1000000,
                            offsetof(MyOptionsStruct, int_param));
    add_local_real_reloption(relopts, "real_param", "real parameter",
                             1.0, 0.0, 1000000.0,
                             offsetof(MyOptionsStruct, real_param));
    add_local_enum_reloption(relopts, "enum_param", "enum parameter",
                             myEnumValues, MY_ENUM_ON,
                             "Valid values are: \"on\", \"off\" and \"auto\".",
                             offsetof(MyOptionsStruct, enum_param));
    add_local_string_reloption(relopts, "str_param", "string parameter",
                               str_param_default,
                               &validate_my_string_relopt,
                               &fill_my_string_relopt,
                               offsetof(MyOptionsStruct, str_param));

    PG_RETURN_VOID();
}

PG_FUNCTION_INFO_V1(my_compress);

Datum
my_compress(PG_FUNCTION_ARGS)
{
    int     int_param = 100;
    double  real_param = 1.0;
    MyEnumType enum_param = MY_ENUM_ON;
    char   *str_param = str_param_default;

    /*
     * Normally, when opclass contains 'options' method, then options are always
     * passed to support functions.  However, if you add 'options' method to
     * existing opclass, previously defined indexes have no options, so the
     * check is required.
     */
    if (PG_HAS_OPCLASS_OPTIONS())
    {
        MyOptionsStruct *options = (MyOptionsStruct *) PG_GET_OPCLASS_OPTIONS();

        int_param = options->int_param;
        real_param = options->real_param;
        enum_param = options->enum_param;
        str_param = GET_STRING_RELOPTION(options, str_param);
    }

    /* the rest implementation of support function */
}

Since the representation of the key in GiST is flexible, it may depend on user-specified parameters. For instance, the length of key signature may be specified. See gtsvector_options() for example.

sortsupport

Returns a comparator function to sort data in a way that preserves locality. It is used by CREATE INDEX and REINDEX commands. The quality of the created index depends on how well the sort order determined by the comparator function preserves locality of the inputs.

The sortsupport method is optional. If it is not provided, CREATE INDEX builds the index by inserting each tuple to the tree using the penalty and picksplit functions, which is much slower.

The SQL declaration of the function must look like this:

CREATE OR REPLACE FUNCTION my_sortsupport(internal)
RETURNS void
AS 'MODULE_PATHNAME'
LANGUAGE C STRICT;

The argument is a pointer to a SortSupport struct. At a minimum, the function must fill in its comparator field. The comparator takes three arguments: two Datums to compare, and a pointer to the SortSupport struct. The Datums are the two indexed values in the format that they are stored in the index; that is, in the format returned by the compress method. The full API is defined in src/include/utils/sortsupport.h.

The matching code in the C module could then follow this skeleton:

PG_FUNCTION_INFO_V1(my_sortsupport);

static int
my_fastcmp(Datum x, Datum y, SortSupport ssup)
{
  /* establish order between x and y by computing some sorting value z */

  int z1 = ComputeSpatialCode(x);
  int z2 = ComputeSpatialCode(y);

  return z1 == z2 ? 0 : z1 > z2 ? 1 : -1;
}

Datum
my_sortsupport(PG_FUNCTION_ARGS)
{
  SortSupport ssup = (SortSupport) PG_GETARG_POINTER(0);

  ssup->comparator = my_fastcmp;
  PG_RETURN_VOID();
}

All the GiST support methods are normally called in short-lived memory contexts; that is, CurrentMemoryContext will get reset after each tuple is processed. It is therefore not very important to worry about pfree'ing everything you palloc. However, in some cases it's useful for a support method to cache data across repeated calls. To do that, allocate the longer-lived data in fcinfo->flinfo->fn_mcxt, and keep a pointer to it in fcinfo->flinfo->fn_extra. Such data will survive for the life of the index operation (e.g., a single GiST index scan, index build, or index tuple insertion). Be careful to pfree the previous value when replacing a fn_extra value, or the leak will accumulate for the duration of the operation.

Prev | Up | Next
65.2. Built-in Operator Classes | Home | 65.4. Implementation


65.4. Implementation
Prev Up Chapter 65. GiST Indexes Home Next

65.4. Implementation

65.4.1. GiST Index Build Methods

65.4.1. GiST Index Build Methods

The simplest way to build a GiST index is just to insert all the entries, one by one. This tends to be slow for large indexes, because if the index tuples are scattered across the index and the index is large enough to not fit in cache, a lot of random I/O will be needed. PostgreSQL supports two alternative methods for initial build of a GiST index: sorted and buffered modes.

The sorted method is only available if each of the opclasses used by the index provides a sortsupport function, as described in Section 65.3. If they do, this method is usually the best, so it is used by default.

The buffered method works by not inserting tuples directly into the index right away. It can dramatically reduce the amount of random I/O needed for non-ordered data sets. For well-ordered data sets the benefit is smaller or non-existent, because only a small number of pages receive new tuples at a time, and those pages fit in cache even if the index as a whole does not.

The buffered method needs to call the penalty function more often than the simple method does, which consumes some extra CPU resources. Also, the buffers need temporary disk space, up to the size of the resulting index. Buffering can also influence the quality of the resulting index, in both positive and negative directions. That influence depends on various factors, like the distribution of the input data and the operator class implementation.

If sorting is not possible, then by default a GiST index build switches to the buffering method when the index size reaches effective_cache_size. Buffering can be manually forced or prevented by the buffering parameter to the CREATE INDEX command. The default behavior is good for most cases, but turning buffering off might speed up the build somewhat if the input data is ordered.

Prev | Up | Next
65.3. Extensibility | Home | 65.5. Examples


65.5. Examples
Prev Up Chapter 65. GiST Indexes Home Next

65.5. Examples

The PostgreSQL source distribution includes several examples of index methods implemented using GiST. The core system currently provides text search support (indexing for tsvector and tsquery) as well as R-Tree equivalent functionality for some of the built-in geometric data types (see src/backend/access/gist/gistproc.c). The following contrib modules also contain GiST operator classes:

btree_gist

B-tree equivalent functionality for several data types

cube

Indexing for multidimensional cubes

hstore

Module for storing (key, value) pairs

intarray

RD-Tree for one-dimensional array of int4 values

ltree

Indexing for tree-like structures

pg_trgm

Text similarity using trigram matching

seg

Indexing for “float ranges”

Prev | Up | Next
65.4. Implementation | Home | Chapter 66. SP-GiST Indexes


Chapter 67. GIN Indexes
Prev Up Part VII. Internals Home Next

Chapter 67. GIN Indexes

Table of Contents

67.1. Introduction
67.2. Built-in Operator Classes
67.3. Extensibility
67.4. Implementation
67.4.1. GIN Fast Update Technique
67.4.2. Partial Match Algorithm
67.5. GIN Tips and Tricks
67.6. Limitations
67.7. Examples
Prev | Up | Next
66.5. Examples | Home | 67.1. Introduction


67.1. Introduction
Prev Up Chapter 67. GIN Indexes Home Next

67.1. Introduction

GIN stands for Generalized Inverted Index. GIN is designed for handling cases where the items to be indexed are composite values, and the queries to be handled by the index need to search for element values that appear within the composite items. For example, the items could be documents, and the queries could be searches for documents containing specific words.

We use the word item to refer to a composite value that is to be indexed, and the word key to refer to an element value. GIN always stores and searches for keys, not item values per se.

A GIN index stores a set of (key, posting list) pairs, where a posting list is a set of row IDs in which the key occurs. The same row ID can appear in multiple posting lists, since an item can contain more than one key. Each key value is stored only once, so a GIN index is very compact for cases where the same key appears many times.

GIN is generalized in the sense that the GIN access method code does not need to know the specific operations that it accelerates. Instead, it uses custom strategies defined for particular data types. The strategy defines how keys are extracted from indexed items and query conditions, and how to determine whether a row that contains some of the key values in a query actually satisfies the query.

One advantage of GIN is that it allows the development of custom data types with the appropriate access methods, by an expert in the domain of the data type, rather than a database expert. This is much the same advantage as using GiST.

The GIN implementation in PostgreSQL is primarily maintained by Teodor Sigaev and Oleg Bartunov. There is more information about GIN on their website.

Prev | Up | Next
Chapter 67. GIN Indexes | Home | 67.2. Built-in Operator Classes


67.2. Built-in Operator Classes
Prev Up Chapter 67. GIN Indexes Home Next

67.2. Built-in Operator Classes

The core PostgreSQL distribution includes the GIN operator classes shown in Table 67.1. (Some of the optional modules described in Appendix F provide additional GIN operator classes.)

Table 67.1. Built-in GIN Operator Classes

Name | Indexable Operators
array_ops | && (anyarray,anyarray)
 | @> (anyarray,anyarray)
 | <@ (anyarray,anyarray)
 | = (anyarray,anyarray)
jsonb_ops | @> (jsonb,jsonb)
 | @? (jsonb,jsonpath)
 | @@ (jsonb,jsonpath)
 | ? (jsonb,text)
 | ?| (jsonb,text[])
 | ?& (jsonb,text[])
jsonb_path_ops | @> (jsonb,jsonb)
 | @? (jsonb,jsonpath)
 | @@ (jsonb,jsonpath)
tsvector_ops | @@ (tsvector,tsquery)
 | @@@ (tsvector,tsquery)

Of the two operator classes for type jsonb, jsonb_ops is the default. jsonb_path_ops supports fewer operators but offers better performance for those operators. See Section 8.14.4 for details.

Prev | Up | Next
67.1. Introduction | Home | 67.3. Extensibility


67.3. Extensibility
Prev Up Chapter 67. GIN Indexes Home Next

67.3. Extensibility

The GIN interface has a high level of abstraction, requiring the access method implementer only to implement the semantics of the data type being accessed. The GIN layer itself takes care of concurrency, logging and searching the tree structure.

All it takes to get a GIN access method working is to implement a few user-defined methods, which define the behavior of keys in the tree and the relationships between keys, indexed items, and indexable queries. In short, GIN combines extensibility with generality, code reuse, and a clean interface.

There are two methods that an operator class for GIN must provide:

Datum *extractValue(Datum itemValue, int32 *nkeys, bool **nullFlags)

Returns a palloc'd array of keys given an item to be indexed. The number of returned keys must be stored into *nkeys. If any of the keys can be null, also palloc an array of *nkeys bool fields, store its address at *nullFlags, and set these null flags as needed. *nullFlags can be left NULL (its initial value) if all keys are non-null. The return value can be NULL if the item contains no keys.

Datum *extractQuery(Datum query, int32 *nkeys, StrategyNumber n, bool **pmatch, Pointer **extra_data, bool **nullFlags, int32 *searchMode)

Returns a palloc'd array of keys given a value to be queried; that is, query is the value on the right-hand side of an indexable operator whose left-hand side is the indexed column. n is the strategy number of the operator within the operator class (see Section 38.16.2). Often, extractQuery will need to consult n to determine the data type of query and the method it should use to extract key values. The number of returned keys must be stored into *nkeys. If any of the keys can be null, also palloc an array of *nkeys bool fields, store its address at *nullFlags, and set these null flags as needed. *nullFlags can be left NULL (its initial value) if all keys are non-null. The return value can be NULL if the query contains no keys.

searchMode is an output argument that allows extractQuery to specify details about how the search will be done. If *searchMode is set to GIN_SEARCH_MODE_DEFAULT (which is the value it is initialized to before call), only items that match at least one of the returned keys are considered candidate matches. If *searchMode is set to GIN_SEARCH_MODE_INCLUDE_EMPTY, then in addition to items containing at least one matching key, items that contain no keys at all are considered candidate matches. (This mode is useful for implementing is-subset-of operators, for example.) If *searchMode is set to GIN_SEARCH_MODE_ALL, then all non-null items in the index are considered candidate matches, whether they match any of the returned keys or not. (This mode is much slower than the other two choices, since it requires scanning essentially the entire index, but it may be necessary to implement corner cases correctly. An operator that needs this mode in most cases is probably not a good candidate for a GIN operator class.) The symbols to use for setting this mode are defined in access/gin.h.

pmatch is an output argument for use when partial match is supported. To use it, extractQuery must allocate an array of *nkeys bools and store its address at *pmatch. Each element of the array should be set to true if the corresponding key requires partial match, false if not. If *pmatch is set to NULL then GIN assumes partial match is not required. The variable is initialized to NULL before call, so this argument can simply be ignored by operator classes that do not support partial match.

extra_data is an output argument that allows extractQuery to pass additional data to the consistent and comparePartial methods. To use it, extractQuery must allocate an array of *nkeys pointers and store its address at *extra_data, then store whatever it wants to into the individual pointers. The variable is initialized to NULL before call, so this argument can simply be ignored by operator classes that do not require extra data. If *extra_data is set, the whole array is passed to the consistent method, and the appropriate element to the comparePartial method.

An operator class must also provide a function to check if an indexed item matches the query. It comes in two flavors, a Boolean consistent function, and a ternary triConsistent function. triConsistent covers the functionality of both, so providing triConsistent alone is sufficient. However, if the Boolean variant is significantly cheaper to calculate, it can be advantageous to provide both. If only the Boolean variant is provided, some optimizations that depend on refuting index items before fetching all the keys are disabled.

bool consistent(bool check[], StrategyNumber n, Datum query, int32 nkeys, Pointer extra_data[], bool *recheck, Datum queryKeys[], bool nullFlags[])

Returns true if an indexed item satisfies the query operator with strategy number n (or might satisfy it, if the recheck indication is returned). This function does not have direct access to the indexed item's value, since GIN does not store items explicitly. Rather, what is available is knowledge about which key values extracted from the query appear in a given indexed item. The check array has length nkeys, which is the same as the number of keys previously returned by extractQuery for this query datum. Each element of the check array is true if the indexed item contains the corresponding query key, i.e., if (check[i] == true) the i-th key of the extractQuery result array is present in the indexed item. The original query datum is passed in case the consistent method needs to consult it, and so are the queryKeys[] and nullFlags[] arrays previously returned by extractQuery. extra_data is the extra-data array returned by extractQuery, or NULL if none.

When extractQuery returns a null key in queryKeys[], the corresponding check[] element is true if the indexed item contains a null key; that is, the semantics of check[] are like IS NOT DISTINCT FROM. The consistent function can examine the corresponding nullFlags[] element if it needs to tell the difference between a regular value match and a null match.

On success, *recheck should be set to true if the heap tuple needs to be rechecked against the query operator, or false if the index test is exact. That is, a false return value guarantees that the heap tuple does not match the query; a true return value with *recheck set to false guarantees that the heap tuple does match the query; and a true return value with *recheck set to true means that the heap tuple might match the query, so it needs to be fetched and rechecked by evaluating the query operator directly against the originally indexed item.

GinTernaryValue triConsistent(GinTernaryValue check[], StrategyNumber n, Datum query, int32 nkeys, Pointer extra_data[], Datum queryKeys[], bool nullFlags[])

triConsistent is similar to consistent, but instead of Booleans in the check vector, there are three possible values for each key: GIN_TRUE, GIN_FALSE and GIN_MAYBE. GIN_FALSE and GIN_TRUE have the same meaning as regular Boolean values, while GIN_MAYBE means that the presence of that key is not known. When GIN_MAYBE values are present, the function should only return GIN_TRUE if the item certainly matches whether or not the index item contains the corresponding query keys. Likewise, the function must return GIN_FALSE only if the item certainly does not match, whether or not it contains the GIN_MAYBE keys. If the result depends on the GIN_MAYBE entries, i.e., the match cannot be confirmed or refuted based on the known query keys, the function must return GIN_MAYBE.

When there are no GIN_MAYBE values in the check vector, a GIN_MAYBE return value is the equivalent of setting the recheck flag in the Boolean consistent function.

In addition, GIN must have a way to sort the key values stored in the index. The operator class can define the sort ordering by specifying a comparison method:

int compare(Datum a, Datum b)

Compares two keys (not indexed items!) and returns an integer less than zero, zero, or greater than zero, indicating whether the first key is less than, equal to, or greater than the second. Null keys are never passed to this function.

Alternatively, if the operator class does not provide a compare method, GIN will look up the default btree operator class for the index key data type, and use its comparison function. It is recommended to specify the comparison function in a GIN operator class that is meant for just one data type, as looking up the btree operator class costs a few cycles. However, polymorphic GIN operator classes (such as array_ops) typically cannot specify a single comparison function.

An operator class for GIN can optionally supply the following methods:

int comparePartial(Datum partial_key, Datum key, StrategyNumber n, Pointer extra_data)

Compare a partial-match query key to an index key. Returns an integer whose sign indicates the result: less than zero means the index key does not match the query, but the index scan should continue; zero means that the index key does match the query; greater than zero indicates that the index scan should stop because no more matches are possible. The strategy number n of the operator that generated the partial match query is provided, in case its semantics are needed to determine when to end the scan. Also, extra_data is the corresponding element of the extra-data array made by extractQuery, or NULL if none. Null keys are never passed to this function.

void options(local_relopts *relopts)

Defines a set of user-visible parameters that control operator class behavior.

The options function is passed a pointer to a local_relopts struct, which needs to be filled with a set of operator class specific options. The options can be accessed from other support functions using the PG_HAS_OPCLASS_OPTIONS() and PG_GET_OPCLASS_OPTIONS() macros.

Since both key extraction of indexed values and representation of the key in GIN are flexible, they may depend on user-specified parameters.

To support “partial match” queries, an operator class must provide the comparePartial method, and its extractQuery method must set the pmatch parameter when a partial-match query is encountered. See Section 67.4.2 for details.

The actual data types of the various Datum values mentioned above vary depending on the operator class. The item values passed to extractValue are always of the operator class's input type, and all key values must be of the class's STORAGE type. The type of the query argument passed to extractQuery, consistent and triConsistent is whatever is the right-hand input type of the class member operator identified by the strategy number. This need not be the same as the indexed type, so long as key values of the correct type can be extracted from it. However, it is recommended that the SQL declarations of these three support functions use the opclass's indexed data type for the query argument, even though the actual type might be something else depending on the operator.

Prev | Up | Next
67.2. Built-in Operator Classes | Home | 67.4. Implementation


67.4. Implementation
Prev Up Chapter 67. GIN Indexes Home Next

67.4. Implementation

67.4.1. GIN Fast Update Technique
67.4.2. Partial Match Algorithm

Internally, a GIN index contains a B-tree index constructed over keys, where each key is an element of one or more indexed items (a member of an array, for example) and where each tuple in a leaf page contains either a pointer to a B-tree of heap pointers (a “posting tree”), or a simple list of heap pointers (a “posting list”) when the list is small enough to fit into a single index tuple along with the key value. Figure 67.1 illustrates these components of a GIN index.

As of PostgreSQL 9.1, null key values can be included in the index. Also, placeholder nulls are included in the index for indexed items that are null or contain no keys according to extractValue. This allows searches that should find empty items to do so.

Multicolumn GIN indexes are implemented by building a single B-tree over composite values (column number, key value). The key values for different columns can be of different types.

Figure 67.1. GIN Internals

67.4.1. GIN Fast Update Technique

Updating a GIN index tends to be slow because of the intrinsic nature of inverted indexes: inserting or updating one heap row can cause many inserts into the index (one for each key extracted from the indexed item). GIN is capable of postponing much of this work by inserting new tuples into a temporary, unsorted list of pending entries. When the table is vacuumed or autoanalyzed, or when gin_clean_pending_list function is called, or if the pending list becomes larger than gin_pending_list_limit, the entries are moved to the main GIN data structure using the same bulk insert techniques used during initial index creation. This greatly improves GIN index update speed, even counting the additional vacuum overhead. Moreover the overhead work can be done by a background process instead of in foreground query processing.

The main disadvantage of this approach is that searches must scan the list of pending entries in addition to searching the regular index, and so a large list of pending entries will slow searches significantly. Another disadvantage is that, while most updates are fast, an update that causes the pending list to become “too large” will incur an immediate cleanup cycle and thus be much slower than other updates. Proper use of autovacuum can minimize both of these problems.

If consistent response time is more important than update speed, use of pending entries can be disabled by turning off the fastupdate storage parameter for a GIN index. See CREATE INDEX for details.

67.4.2. Partial Match Algorithm

GIN can support “partial match” queries, in which the query does not determine an exact match for one or more keys, but the possible matches fall within a reasonably narrow range of key values (within the key sorting order determined by the compare support method). The extractQuery method, instead of returning a key value to be matched exactly, returns a key value that is the lower bound of the range to be searched, and sets the pmatch flag true. The key range is then scanned using the comparePartial method. comparePartial must return zero for a matching index key, less than zero for a non-match that is still within the range to be searched, or greater than zero if the index key is past the range that could match.

Prev | Up | Next
67.3. Extensibility | Home | 67.5. GIN Tips and Tricks


67.5. GIN Tips and Tricks
Prev Up Chapter 67. GIN Indexes Home Next

67.5. GIN Tips and Tricks

Create vs. insert

Insertion into a GIN index can be slow due to the likelihood of many keys being inserted for each item. So, for bulk insertions into a table it is advisable to drop the GIN index and recreate it after finishing bulk insertion.

When fastupdate is enabled for GIN (see Section 67.4.1 for details), the penalty is less than when it is not. But for very large updates it may still be best to drop and recreate the index.

maintenance_work_mem

Build time for a GIN index is very sensitive to the maintenance_work_mem setting; it doesn't pay to skimp on work memory during index creation.

gin_pending_list_limit

During a series of insertions into an existing GIN index that has fastupdate enabled, the system will clean up the pending-entry list whenever the list grows larger than gin_pending_list_limit. To avoid fluctuations in observed response time, it's desirable to have pending-list cleanup occur in the background (i.e., via autovacuum). Foreground cleanup operations can be avoided by increasing gin_pending_list_limit or making autovacuum more aggressive. However, enlarging the threshold of the cleanup operation means that if a foreground cleanup does occur, it will take even longer.

gin_pending_list_limit can be overridden for individual GIN indexes by changing storage parameters, which allows each GIN index to have its own cleanup threshold. For example, it's possible to increase the threshold only for the GIN index which can be updated heavily, and decrease it otherwise.

gin_fuzzy_search_limit

The primary goal of developing GIN indexes was to create support for highly scalable full-text search in PostgreSQL, and there are often situations when a full-text search returns a very large set of results. Moreover, this often happens when the query contains very frequent words, so that the large result set is not even useful. Since reading many tuples from the disk and sorting them could take a lot of time, this is unacceptable for production. (Note that the index search itself is very fast.)

To facilitate controlled execution of such queries, GIN has a configurable soft upper limit on the number of rows returned: the gin_fuzzy_search_limit configuration parameter. It is set to 0 (meaning no limit) by default. If a non-zero limit is set, then the returned set is a subset of the whole result set, chosen at random.

“Soft” means that the actual number of returned results could differ somewhat from the specified limit, depending on the query and the quality of the system's random number generator.

From experience, values in the thousands (e.g., 5000 — 20000) work well.

Prev | Up | Next
67.4. Implementation | Home | 67.6. Limitations


67.6. Limitations
Prev Up Chapter 67. GIN Indexes Home Next

67.6. Limitations

GIN assumes that indexable operators are strict. This means that extractValue will not be called at all on a null item value (instead, a placeholder index entry is created automatically), and extractQuery will not be called on a null query value either (instead, the query is presumed to be unsatisfiable). Note however that null key values contained within a non-null composite item or query value are supported.

Prev | Up | Next
67.5. GIN Tips and Tricks | Home | 67.7. Examples


67.7. Examples
Prev Up Chapter 67. GIN Indexes Home Next

67.7. Examples

The core PostgreSQL distribution includes the GIN operator classes previously shown in Table 67.1. The following contrib modules also contain GIN operator classes:

btree_gin

B-tree equivalent functionality for several data types

hstore

Module for storing (key, value) pairs

intarray

Enhanced support for int[]

pg_trgm

Text similarity using trigram matching

Prev | Up | Next
67.6. Limitations | Home | Chapter 68. BRIN Indexes


Chapter 68. BRIN Indexes
Prev Up Part VII. Internals Home Next

Chapter 68. BRIN Indexes

Table of Contents

68.1. Introduction
68.1.1. Index Maintenance
68.2. Built-in Operator Classes
68.2.1. Operator Class Parameters
68.3. Extensibility
Prev | Up | Next
67.7. Examples | Home | 68.1. Introduction


68.1. Introduction
Prev Up Chapter 68. BRIN Indexes Home Next

68.1. Introduction

68.1.1. Index Maintenance

BRIN stands for Block Range Index. BRIN is designed for handling very large tables in which certain columns have some natural correlation with their physical location within the table. A block range is a group of pages that are physically adjacent in the table; for each block range, some summary info is stored by the index. For example, a table storing a store's sale orders might have a date column on which each order was placed, and most of the time the entries for earlier orders will appear earlier in the table as well; a table storing a ZIP code column might have all codes for a city grouped together naturally.

BRIN indexes can satisfy queries via regular bitmap index scans, and will return all tuples in all pages within each range if the summary info stored by the index is consistent with the query conditions. The query executor is in charge of rechecking these tuples and discarding those that do not match the query conditions — in other words, these indexes are lossy. Because a BRIN index is very small, scanning the index adds little overhead compared to a sequential scan, but may avoid scanning large parts of the table that are known not to contain matching tuples.

The specific data that a BRIN index will store, as well as the specific queries that the index will be able to satisfy, depend on the operator class selected for each column of the index. Data types having a linear sort order can have operator classes that store the minimum and maximum value within each block range, for instance; geometrical types might store the bounding box for all the objects in the block range.

The size of the block range is determined at index creation time by the pages_per_range storage parameter. The number of index entries will be equal to the size of the relation in pages divided by the selected value for pages_per_range. Therefore, the smaller the number, the larger the index becomes (because of the need to store more index entries), but at the same time the summary data stored can be more precise and more data blocks can be skipped during an index scan.

68.1.1. Index Maintenance

At the time of creation, all existing heap pages are scanned and a summary index tuple is created for each range, including the possibly-incomplete range at the end. As new pages are filled with data, page ranges that are already summarized will cause the summary information to be updated with data from the new tuples. When a new page is created that does not fall within the last summarized range, that range does not automatically acquire a summary tuple; those tuples remain unsummarized until a summarization run is invoked later, creating initial summaries. This process can be invoked manually using the brin_summarize_range(regclass, bigint) or brin_summarize_new_values(regclass) functions; automatically when VACUUM processes the table; or by automatic summarization executed by autovacuum, as insertions occur. (This last trigger is disabled by default and can be enabled with the autosummarize parameter.) Conversely, a range can be de-summarized using the brin_desummarize_range(regclass, bigint) function, which is useful when the index tuple is no longer a very good representation because the existing values have changed.

When autosummarization is enabled, each time a page range is filled a request is sent to autovacuum for it to execute a targeted summarization for that range, to be fulfilled at the end of the next worker run on the same database. If the request queue is full, the request is not recorded and a message is sent to the server log:

LOG:  request for BRIN range summarization for index "brin_wi_idx" page 128 was not recorded

When this happens, the range will be summarized normally during the next regular vacuum of the table.

Prev | Up | Next
Chapter 68. BRIN Indexes | Home | 68.2. Built-in Operator Classes


68.2. Built-in Operator Classes
Prev Up Chapter 68. BRIN Indexes Home Next

68.2. Built-in Operator Classes

68.2.1. Operator Class Parameters

The core PostgreSQL distribution includes the BRIN operator classes shown in Table 68.1.

The minmax operator classes store the minimum and the maximum values appearing in the indexed column within the range. The inclusion operator classes store a value which includes the values in the indexed column within the range. The bloom operator classes build a Bloom filter for all values in the range. The minmax-multi operator classes store multiple minimum and maximum values, representing values appearing in the indexed column within the range.

Table 68.1. Built-in BRIN Operator Classes

Name | Indexable Operators
bit_minmax_ops | = (bit,bit)
 | < (bit,bit)
 | > (bit,bit)
 | <= (bit,bit)
 | >= (bit,bit)
box_inclusion_ops | @> (box,point)
 | << (box,box)
 | &< (box,box)
 | &> (box,box)
 | >> (box,box)
 | <@ (box,box)
 | @> (box,box)
 | ~= (box,box)
 | && (box,box)
 | <<| (box,box)
 | &<| (box,box)
 | |&> (box,box)
 | |>> (box,box)
bpchar_bloom_ops | = (character,character)
bpchar_minmax_ops | = (character,character)
 | < (character,character)
 | <= (character,character)
 | > (character,character)
 | >= (character,character)
bytea_bloom_ops | = (bytea,bytea)
bytea_minmax_ops | = (bytea,bytea)
 | < (bytea,bytea)
 | <= (bytea,bytea)
 | > (bytea,bytea)
 | >= (bytea,bytea)
char_bloom_ops | = ("char","char")
char_minmax_ops | = ("char","char")
 | < ("char","char")
 | <= ("char","char")
 | > ("char","char")
 | >= ("char","char")
date_bloom_ops | = (date,date)
date_minmax_ops | = (date,date)
 | < (date,date)
 | <= (date,date)
 | > (date,date)
 | >= (date,date)
date_minmax_multi_ops | = (date,date)
 | < (date,date)
 | <= (date,date)
 | > (date,date)
 | >= (date,date)
float4_bloom_ops | = (float4,float4)
float4_minmax_ops | = (float4,float4)
 | < (float4,float4)
 | > (float4,float4)
 | <= (float4,float4)
 | >= (float4,float4)
float4_minmax_multi_ops | = (float4,float4)
 | < (float4,float4)
 | > (float4,float4)
 | <= (float4,float4)
 | >= (float4,float4)
float8_bloom_ops | = (float8,float8)
float8_minmax_ops | = (float8,float8)
 | < (float8,float8)
 | <= (float8,float8)
 | > (float8,float8)
 | >= (float8,float8)
float8_minmax_multi_ops | = (float8,float8)
 | < (float8,float8)
 | <= (float8,float8)
 | > (float8,float8)
 | >= (float8,float8)
inet_inclusion_ops | << (inet,inet)
 | <<= (inet,inet)
 | >> (inet,inet)
 | >>= (inet,inet)
 | = (inet,inet)
 | && (inet,inet)
inet_bloom_ops | = (inet,inet)
inet_minmax_ops | = (inet,inet)
 | < (inet,inet)
 | <= (inet,inet)
 | > (inet,inet)
 | >= (inet,inet)
inet_minmax_multi_ops | = (inet,inet)
 | < (inet,inet)
 | <= (inet,inet)
 | > (inet,inet)
 | >= (inet,inet)
int2_bloom_ops | = (int2,int2)
int2_minmax_ops | = (int2,int2)
 | < (int2,int2)
 | > (int2,int2)
 | <= (int2,int2)
 | >= (int2,int2)
int2_minmax_multi_ops | = (int2,int2)
 | < (int2,int2)
 | > (int2,int2)
 | <= (int2,int2)
 | >= (int2,int2)
int4_bloom_ops | = (int4,int4)
int4_minmax_ops | = (int4,int4)
 | < (int4,int4)
 | > (int4,int4)
 | <= (int4,int4)
 | >= (int4,int4)
int4_minmax_multi_ops | = (int4,int4)
 | < (int4,int4)
 | > (int4,int4)
 | <= (int4,int4)
 | >= (int4,int4)
int8_bloom_ops | = (bigint,bigint)
int8_minmax_ops | = (bigint,bigint)
 | < (bigint,bigint)
 | > (bigint,bigint)
 | <= (bigint,bigint)
 | >= (bigint,bigint)
int8_minmax_multi_ops | = (bigint,bigint)
 | < (bigint,bigint)
 | > (bigint,bigint)
 | <= (bigint,bigint)
 | >= (bigint,bigint)
interval_bloom_ops | = (interval,interval)
interval_minmax_ops | = (interval,interval)
 | < (interval,interval)
 | <= (interval,interval)
 | > (interval,interval)
 | >= (interval,interval)
interval_minmax_multi_ops | = (interval,interval)
 | < (interval,interval)
 | <= (interval,interval)
 | > (interval,interval)
 | >= (interval,interval)
macaddr_bloom_ops | = (macaddr,macaddr)
macaddr_minmax_ops | = (macaddr,macaddr)
 | < (macaddr,macaddr)
 | <= (macaddr,macaddr)
 | > (macaddr,macaddr)
 | >= (macaddr,macaddr)
macaddr_minmax_multi_ops | = (macaddr,macaddr)
 | < (macaddr,macaddr)
 | <= (macaddr,macaddr)
 | > (macaddr,macaddr)
 | >= (macaddr,macaddr)
macaddr8_bloom_ops | = (macaddr8,macaddr8)
macaddr8_minmax_ops | = (macaddr8,macaddr8)
 | < (macaddr8,macaddr8)
 | <= (macaddr8,macaddr8)
 | > (macaddr8,macaddr8)
 | >= (macaddr8,macaddr8)
macaddr8_minmax_multi_ops | = (macaddr8,macaddr8)
 | < (macaddr8,macaddr8)
 | <= (macaddr8,macaddr8)
 | > (macaddr8,macaddr8)
 | >= (macaddr8,macaddr8)
name_bloom_ops | = (name,name)
name_minmax_ops | = (name,name)
 | < (name,name)
 | <= (name,name)
 | > (name,name)
 | >= (name,name)
numeric_bloom_ops | = (numeric,numeric)
numeric_minmax_ops | = (numeric,numeric)
 | < (numeric,numeric)
 | <= (numeric,numeric)
 | > (numeric,numeric)
 | >= (numeric,numeric)
numeric_minmax_multi_ops | = (numeric,numeric)
 | < (numeric,numeric)
 | <= (numeric,numeric)
 | > (numeric,numeric)
 | >= (numeric,numeric)
oid_bloom_ops | = (oid,oid)
oid_minmax_ops | = (oid,oid)
 | < (oid,oid)
 | > (oid,oid)
 | <= (oid,oid)
 | >= (oid,oid)
oid_minmax_multi_ops | = (oid,oid)
 | < (oid,oid)
 | > (oid,oid)
 | <= (oid,oid)
 | >= (oid,oid)
pg_lsn_bloom_ops | = (pg_lsn,pg_lsn)
pg_lsn_minmax_ops | = (pg_lsn,pg_lsn)
 | < (pg_lsn,pg_lsn)
 | > (pg_lsn,pg_lsn)
 | <= (pg_lsn,pg_lsn)
 | >= (pg_lsn,pg_lsn)
pg_lsn_minmax_multi_ops | = (pg_lsn,pg_lsn)
 | < (pg_lsn,pg_lsn)
 | > (pg_lsn,pg_lsn)
 | <= (pg_lsn,pg_lsn)
 | >= (pg_lsn,pg_lsn)
range_inclusion_ops | = (anyrange,anyrange)
 | < (anyrange,anyrange)
 | <= (anyrange,anyrange)
 | >= (anyrange,anyrange)
 | > (anyrange,anyrange)
 | && (anyrange,anyrange)
 | @> (anyrange,anyelement)
 | @> (anyrange,anyrange)
 | <@ (anyrange,anyrange)
 | << (anyrange,anyrange)
 | >> (anyrange,anyrange)
 | &< (anyrange,anyrange)
 | &> (anyrange,anyrange)
 | -|- (anyrange,anyrange)
text_bloom_ops | = (text,text)
text_minmax_ops | = (text,text)
 | < (text,text)
 | <= (text,text)
 | > (text,text)
 | >= (text,text)
tid_bloom_ops | = (tid,tid)
tid_minmax_ops | = (tid,tid)
 | < (tid,tid)
 | > (tid,tid)
 | <= (tid,tid)
 | >= (tid,tid)
tid_minmax_multi_ops | = (tid,tid)
 | < (tid,tid)
 | > (tid,tid)
 | <= (tid,tid)
 | >= (tid,tid)
timestamp_bloom_ops | = (timestamp,timestamp)
timestamp_minmax_ops | = (timestamp,timestamp)
 | < (timestamp,timestamp)
 | <= (timestamp,timestamp)
 | > (timestamp,timestamp)
 | >= (timestamp,timestamp)
timestamp_minmax_multi_ops | = (timestamp,timestamp)
 | < (timestamp,timestamp)
 | <= (timestamp,timestamp)
 | > (timestamp,timestamp)
 | >= (timestamp,timestamp)
timestamptz_bloom_ops | = (timestamptz,timestamptz)
timestamptz_minmax_ops | = (timestamptz,timestamptz)
 | < (timestamptz,timestamptz)
 | <= (timestamptz,timestamptz)
 | > (timestamptz,timestamptz)
 | >= (timestamptz,timestamptz)
timestamptz_minmax_multi_ops | = (timestamptz,timestamptz)
 | < (timestamptz,timestamptz)
 | <= (timestamptz,timestamptz)
 | > (timestamptz,timestamptz)
 | >= (timestamptz,timestamptz)
time_bloom_ops | = (time,time)
time_minmax_ops | = (time,time)
 | < (time,time)
 | <= (time,time)
 | > (time,time)
 | >= (time,time)
time_minmax_multi_ops | = (time,time)
 | < (time,time)
 | <= (time,time)
 | > (time,time)
 | >= (time,time)
timetz_bloom_ops | = (timetz,timetz)
timetz_minmax_ops | = (timetz,timetz)
 | < (timetz,timetz)
 | <= (timetz,timetz)
 | > (timetz,timetz)
 | >= (timetz,timetz)
timetz_minmax_multi_ops | = (timetz,timetz)
 | < (timetz,timetz)
 | <= (timetz,timetz)
 | > (timetz,timetz)
 | >= (timetz,timetz)
uuid_bloom_ops | = (uuid,uuid)
uuid_minmax_ops | = (uuid,uuid)
 | < (uuid,uuid)
 | > (uuid,uuid)
 | <= (uuid,uuid)
 | >= (uuid,uuid)
uuid_minmax_multi_ops | = (uuid,uuid)
 | < (uuid,uuid)
 | > (uuid,uuid)
 | <= (uuid,uuid)
 | >= (uuid,uuid)
varbit_minmax_ops | = (varbit,varbit)
 | < (varbit,varbit)
 | > (varbit,varbit)
 | <= (varbit,varbit)
 | >= (varbit,varbit)

68.2.1. Operator Class Parameters

Some of the built-in operator classes allow specifying parameters affecting behavior of the operator class. Each operator class has its own set of allowed parameters. Only the bloom and minmax-multi operator classes allow specifying parameters:

bloom operator classes accept these parameters:

n_distinct_per_range

Defines the estimated number of distinct non-null values in the block range, used by BRIN bloom indexes for sizing of the Bloom filter. It behaves similarly to n_distinct option for ALTER TABLE. When set to a positive value, each block range is assumed to contain this number of distinct non-null values. When set to a negative value, which must be greater than or equal to -1, the number of distinct non-null values is assumed to grow linearly with the maximum possible number of tuples in the block range (about 290 rows per block). The default value is -0.1, and the minimum number of distinct non-null values is 16.

false_positive_rate

Defines the desired false positive rate used by BRIN bloom indexes for sizing of the Bloom filter. The values must be between 0.0001 and 0.25. The default value is 0.01, which is 1% false positive rate.

minmax-multi operator classes accept these parameters:

values_per_range

Defines the maximum number of values stored by BRIN minmax indexes to summarize a block range. Each value may represent either a point, or a boundary of an interval. Values must be between 8 and 256, and the default value is 32.

Prev | Up | Next
68.1. Introduction | Home | 68.3. Extensibility


68.3. Extensibility
Prev Up Chapter 68. BRIN Indexes Home Next

68.3. Extensibility

The BRIN interface has a high level of abstraction, requiring the access method implementer only to implement the semantics of the data type being accessed. The BRIN layer itself takes care of concurrency, logging and searching the index structure.

All it takes to get a BRIN access method working is to implement a few user-defined methods, which define the behavior of summary values stored in the index and the way they interact with scan keys. In short, BRIN combines extensibility with generality, code reuse, and a clean interface.

There are four methods that an operator class for BRIN must provide:

BrinOpcInfo *opcInfo(Oid type_oid)

Returns internal information about the indexed columns' summary data. The return value must point to a palloc'd BrinOpcInfo, which has this definition:

typedef struct BrinOpcInfo
{
    /* Number of columns stored in an index column of this opclass */
    uint16      oi_nstored;

    /* Opaque pointer for the opclass' private use */
    void       *oi_opaque;

    /* Type cache entries of the stored columns */
    TypeCacheEntry *oi_typcache[FLEXIBLE_ARRAY_MEMBER];
} BrinOpcInfo;

BrinOpcInfo.oi_opaque can be used by the operator class routines to pass information between support functions during an index scan.

bool consistent(BrinDesc *bdesc, BrinValues *column, ScanKey *keys, int nkeys)

Returns whether all the ScanKey entries are consistent with the given indexed values for a range. The attribute number to use is passed as part of the scan key. Multiple scan keys for the same attribute may be passed at once; the number of entries is determined by the nkeys parameter.

bool consistent(BrinDesc *bdesc, BrinValues *column, ScanKey key)

Returns whether the ScanKey is consistent with the given indexed values for a range. The attribute number to use is passed as part of the scan key. This is an older backward-compatible variant of the consistent function.

bool addValue(BrinDesc *bdesc, BrinValues *column, Datum newval, bool isnull)

Given an index tuple and an indexed value, modifies the indicated attribute of the tuple so that it additionally represents the new value. If any modification was done to the tuple, true is returned.

bool unionTuples(BrinDesc *bdesc, BrinValues *a, BrinValues *b)

Consolidates two index tuples. Given two index tuples, modifies the indicated attribute of the first of them so that it represents both tuples. The second tuple is not modified.

An operator class for BRIN can optionally specify the following method:

void options(local_relopts *relopts)

Defines a set of user-visible parameters that control operator class behavior.

The options function is passed a pointer to a local_relopts struct, which needs to be filled with a set of operator class specific options. The options can be accessed from other support functions using the PG_HAS_OPCLASS_OPTIONS() and PG_GET_OPCLASS_OPTIONS() macros.

Since both key extraction of indexed values and representation of the key in BRIN are flexible, they may depend on user-specified parameters.

The core distribution includes support for four types of operator classes: minmax, minmax-multi, inclusion and bloom. Operator class definitions using them are shipped for in-core data types as appropriate. Additional operator classes can be defined by the user for other data types using equivalent definitions, without having to write any source code; appropriate catalog entries being declared is enough. Note that assumptions about the semantics of operator strategies are embedded in the support functions' source code.

Operator classes that implement completely different semantics are also possible, provided implementations of the four main support functions described above are written. Note that backwards compatibility across major releases is not guaranteed: for example, additional support functions might be required in later releases.

To write an operator class for a data type that implements a totally ordered set, it is possible to use the minmax support functions alongside the corresponding operators, as shown in Table 68.2. All operator class members (functions and operators) are mandatory.

Table 68.2. Function and Support Numbers for Minmax Operator Classes

Operator class member | Object
Support Function 1 | internal function brin_minmax_opcinfo()
Support Function 2 | internal function brin_minmax_add_value()
Support Function 3 | internal function brin_minmax_consistent()
Support Function 4 | internal function brin_minmax_union()
Operator Strategy 1 | operator less-than
Operator Strategy 2 | operator less-than-or-equal-to
Operator Strategy 3 | operator equal-to
Operator Strategy 4 | operator greater-than-or-equal-to
Operator Strategy 5 | operator greater-than

To write an operator class for a complex data type which has values included within another type, it's possible to use the inclusion support functions alongside the corresponding operators, as shown in Table 68.3. It requires only a single additional function, which can be written in any language. More functions can be defined for additional functionality. All operators are optional. Some operators require other operators, as shown as dependencies on the table.

Table 68.3. Function and Support Numbers for Inclusion Operator Classes

Operator class member | Object | Dependency
Support Function 1 | internal function brin_inclusion_opcinfo() |
Support Function 2 | internal function brin_inclusion_add_value() |
Support Function 3 | internal function brin_inclusion_consistent() |
Support Function 4 | internal function brin_inclusion_union() |
Support Function 11 | function to merge two elements |
Support Function 12 | optional function to check whether two elements are mergeable |
Support Function 13 | optional function to check if an element is contained within another |
Support Function 14 | optional function to check whether an element is empty |
Operator Strategy 1 | operator left-of | Operator Strategy 4
Operator Strategy 2 | operator does-not-extend-to-the-right-of | Operator Strategy 5
Operator Strategy 3 | operator overlaps |
Operator Strategy 4 | operator does-not-extend-to-the-left-of | Operator Strategy 1
Operator Strategy 5 | operator right-of | Operator Strategy 2
Operator Strategy 6, 18 | operator same-as-or-equal-to | Operator Strategy 7
Operator Strategy 7, 16, 24, 25 | operator contains-or-equal-to |
Operator Strategy 8, 26, 27 | operator is-contained-by-or-equal-to | Operator Strategy 3
Operator Strategy 9 | operator does-not-extend-above | Operator Strategy 11
Operator Strategy 10 | operator is-below | Operator Strategy 12
Operator Strategy 11 | operator is-above | Operator Strategy 9
Operator Strategy 12 | operator does-not-extend-below | Operator Strategy 10
Operator Strategy 20 | operator less-than | Operator Strategy 5
Operator Strategy 21 | operator less-than-or-equal-to | Operator Strategy 5
Operator Strategy 22 | operator greater-than | Operator Strategy 1
Operator Strategy 23 | operator greater-than-or-equal-to | Operator Strategy 1

Support function numbers 1 through 10 are reserved for the BRIN internal functions, so the SQL level functions start with number 11. Support function number 11 is the main function required to build the index. It should accept two arguments with the same data type as the operator class, and return the union of them. The inclusion operator class can store union values with different data types if it is defined with the STORAGE parameter. The return value of the union function should match the STORAGE data type.

Support function numbers 12 and 14 are provided to support irregularities of built-in data types. Function number 12 is used to support network addresses from different families which are not mergeable. Function number 14 is used to support empty ranges. Function number 13 is an optional but recommended one, which allows the new value to be checked before it is passed to the union function. As the BRIN framework can shortcut some operations when the union is not changed, using this function can improve index performance.

To write an operator class for a data type that implements only an equality operator and supports hashing, it is possible to use the bloom support procedures alongside the corresponding operators, as shown in Table 68.4. All operator class members (procedures and operators) are mandatory.

Table 68.4. Procedure and Support Numbers for Bloom Operator Classes

Operator class member | Object
Support Procedure 1 | internal function brin_bloom_opcinfo()
Support Procedure 2 | internal function brin_bloom_add_value()
Support Procedure 3 | internal function brin_bloom_consistent()
Support Procedure 4 | internal function brin_bloom_union()
Support Procedure 11 | function to compute hash of an element
Operator Strategy 1 | operator equal-to

Support procedure numbers 1-10 are reserved for the BRIN internal functions, so the SQL level functions start with number 11. Support function number 11 is the main function required to build the index. It should accept one argument with the same data type as the operator class, and return a hash of the value.

The minmax-multi operator class is also intended for data types implementing a totally ordered set, and may be seen as a simple extension of the minmax operator class. While minmax operator class summarizes values from each block range into a single contiguous interval, minmax-multi allows summarization into multiple smaller intervals to improve handling of outlier values. It is possible to use the minmax-multi support procedures alongside the corresponding operators, as shown in Table 68.5. All operator class members (procedures and operators) are mandatory.

Table 68.5. Procedure and Support Numbers for minmax-multi Operator Classes

Operator class member | Object
Support Procedure 1 | internal function brin_minmax_multi_opcinfo()
Support Procedure 2 | internal function brin_minmax_multi_add_value()
Support Procedure 3 | internal function brin_minmax_multi_consistent()
Support Procedure 4 | internal function brin_minmax_multi_union()
Support Procedure 11 | function to compute distance between two values (length of a range)
Operator Strategy 1 | operator less-than
Operator Strategy 2 | operator less-than-or-equal-to
Operator Strategy 3 | operator equal-to
Operator Strategy 4 | operator greater-than-or-equal-to
Operator Strategy 5 | operator greater-than

Both minmax and inclusion operator classes support cross-data-type operators, though with these the dependencies become more complicated. The minmax operator class requires a full set of operators to be defined with both arguments having the same data type. It allows additional data types to be supported by defining extra sets of operators. Inclusion operator class operator strategies are dependent on another operator strategy as shown in Table 68.3, or the same operator strategy as themselves. They require the dependency operator to be defined with the STORAGE data type as the left-hand-side argument and the other supported data type to be the right-hand-side argument of the supported operator. See float4_minmax_ops as an example of minmax, and box_inclusion_ops as an example of inclusion.

Prev | Up | Next
68.2. Built-in Operator Classes | Home | Chapter 69. Hash Indexes


Chapter 69. Hash Indexes
Prev Up Part VII. Internals Home Next

Chapter 69. Hash Indexes

Table of Contents

69.1. Overview
69.2. Implementation
Prev | Up | Next
68.3. Extensibility | Home | 69.1. Overview


69.1. Overview
Prev Up Chapter 69. Hash Indexes Home Next

69.1. Overview

PostgreSQL includes an implementation of persistent on-disk hash indexes, which are fully crash recoverable. Any data type can be indexed by a hash index, including data types that do not have a well-defined linear ordering. Hash indexes store only the hash value of the data being indexed, thus there are no restrictions on the size of the data column being indexed.

Hash indexes support only single-column indexes and do not allow uniqueness checking.

Hash indexes support only the = operator, so WHERE clauses that specify range operations will not be able to take advantage of hash indexes.

Each hash index tuple stores just the 4-byte hash value, not the actual column value. As a result, hash indexes may be much smaller than B-trees when indexing longer data items such as UUIDs, URLs, etc. The absence of the column value also makes all hash index scans lossy. Hash indexes may take part in bitmap index scans and backward scans.

Hash indexes are best optimized for SELECT and UPDATE-heavy workloads that use equality scans on larger tables. In a B-tree index, searches must descend through the tree until the leaf page is found. In tables with millions of rows, this descent can increase access time to data. The equivalent of a leaf page in a hash index is referred to as a bucket page. In contrast, a hash index allows accessing the bucket pages directly, thereby potentially reducing index access time in larger tables. This reduction in "logical I/O" becomes even more pronounced on indexes/data larger than shared_buffers/RAM.

Hash indexes have been designed to cope with uneven distributions of hash values. Direct access to the bucket pages works well if the hash values are evenly distributed. When inserts mean that the bucket page becomes full, additional overflow pages are chained to that specific bucket page, locally expanding the storage for index tuples that match that hash value. When scanning a hash bucket during queries, we need to scan through all of the overflow pages. Thus an unbalanced hash index might actually be worse than a B-tree in terms of number of block accesses required, for some data.

As a result of the overflow cases, we can say that hash indexes are most suitable for unique, nearly unique data or data with a low number of rows per hash bucket. One possible way to avoid problems is to exclude highly non-unique values from the index using a partial index condition, but this may not be suitable in many cases.

Like B-Trees, hash indexes perform simple index tuple deletion. This is a deferred maintenance operation that deletes index tuples that are known to be safe to delete (those whose item identifier's LP_DEAD bit is already set). If an insert finds no space is available on a page we try to avoid creating a new overflow page by attempting to remove dead index tuples. Removal cannot occur if the page is pinned at that time. Deletion of dead index pointers also occurs during VACUUM.

If it can, VACUUM will also try to squeeze the index tuples onto as few overflow pages as possible, minimizing the overflow chain. If an overflow page becomes empty, overflow pages can be recycled for reuse in other buckets, though we never return them to the operating system. There is currently no provision to shrink a hash index, other than by rebuilding it with REINDEX. There is no provision for reducing the number of buckets, either.

Hash indexes may expand the number of bucket pages as the number of rows indexed grows. The hash key-to-bucket-number mapping is chosen so that the index can be incrementally expanded. When a new bucket is to be added to the index, exactly one existing bucket will need to be "split", with some of its tuples being transferred to the new bucket according to the updated key-to-bucket-number mapping.

The expansion occurs in the foreground, which could increase execution time for user inserts. Thus, hash indexes may not be suitable for tables with rapidly increasing number of rows.

Prev | Up | Next
Chapter 69. Hash Indexes | Home | 69.2. Implementation


69.2. Implementation
Prev Up Chapter 69. Hash Indexes Home Next

69.2. Implementation

There are four kinds of pages in a hash index: the meta page (page zero), which contains statically allocated control information; primary bucket pages; overflow pages; and bitmap pages, which keep track of overflow pages that have been freed and are available for re-use. For addressing purposes, bitmap pages are regarded as a subset of the overflow pages.

Both scanning the index and inserting tuples require locating the bucket where a given tuple ought to be located. To do this, we need the bucket count, highmask, and lowmask from the metapage; however, it's undesirable for performance reasons to have to have to lock and pin the metapage for every such operation. Instead, we retain a cached copy of the metapage in each backend's relcache entry. This will produce the correct bucket mapping as long as the target bucket hasn't been split since the last cache refresh.

Primary bucket pages and overflow pages are allocated independently since any given index might need more or fewer overflow pages relative to its number of buckets. The hash code uses an interesting set of addressing rules to support a variable number of overflow pages while not having to move primary bucket pages around after they are created.

Each row in the table indexed is represented by a single index tuple in the hash index. Hash index tuples are stored in bucket pages, and if they exist, overflow pages. We speed up searches by keeping the index entries in any one index page sorted by hash code, thus allowing binary search to be used within an index page. Note however that there is *no* assumption about the relative ordering of hash codes across different index pages of a bucket.

The bucket splitting algorithms to expand the hash index are too complex to be worthy of mention here, though are described in more detail in src/backend/access/hash/README. The split algorithm is crash safe and can be restarted if not completed successfully.

Prev | Up | Next
69.1. Overview | Home | Chapter 70. Database Physical Storage


Chapter 71. System Catalog Declarations and Initial Contents
Prev Up Part VII. Internals Home Next

Chapter 71. System Catalog Declarations and Initial Contents

Table of Contents

71.1. System Catalog Declaration Rules
71.2. System Catalog Initial Data
71.2.1. Data File Format
71.2.2. OID Assignment
71.2.3. OID Reference Lookup
71.2.4. Automatic Creation of Array Types
71.2.5. Recipes for Editing Data Files
71.3. BKI File Format
71.4. BKI Commands
71.5. Structure of the Bootstrap BKI File
71.6. BKI Example

PostgreSQL uses many different system catalogs to keep track of the existence and properties of database objects, such as tables and functions. Physically there is no difference between a system catalog and a plain user table, but the backend C code knows the structure and properties of each catalog, and can manipulate it directly at a low level. Thus, for example, it is inadvisable to attempt to alter the structure of a catalog on-the-fly; that would break assumptions built into the C code about how rows of the catalog are laid out. But the structure of the catalogs can change between major versions.

The structures of the catalogs are declared in specially formatted C header files in the src/include/catalog/ directory of the source tree. For each catalog there is a header file named after the catalog (e.g., pg_class.h for pg_class), which defines the set of columns the catalog has, as well as some other basic properties such as its OID.

Many of the catalogs have initial data that must be loaded into them during the “bootstrap” phase of initdb, to bring the system up to a point where it is capable of executing SQL commands. (For example, pg_class.h must contain an entry for itself, as well as one for each other system catalog and index.) This initial data is kept in editable form in data files that are also stored in the src/include/catalog/ directory. For example, pg_proc.dat describes all the initial rows that must be inserted into the pg_proc catalog.

To create the catalog files and load this initial data into them, a backend running in bootstrap mode reads a BKI (Backend Interface) file containing commands and initial data. The postgres.bki file used in this mode is prepared from the aforementioned header and data files, while building a PostgreSQL distribution, by a Perl script named genbki.pl. Although it's specific to a particular PostgreSQL release, postgres.bki is platform-independent and is installed in the share subdirectory of the installation tree.

genbki.pl also produces a derived header file for each catalog, for example pg_class_d.h for the pg_class catalog. This file contains automatically-generated macro definitions, and may contain other macros, enum declarations, and so on that can be useful for client C code that reads a particular catalog.

Most PostgreSQL developers don't need to be directly concerned with the BKI file, but almost any nontrivial feature addition in the backend will require modifying the catalog header files and/or initial data files. The rest of this chapter gives some information about that, and for completeness describes the BKI file format.

Prev | Up | Next
70.6. Database Page Layout | Home | 71.1. System Catalog Declaration Rules


71.3. BKI File Format
Prev Up Chapter 71. System Catalog Declarations and Initial Contents Home Next

71.3. BKI File Format

This section describes how the PostgreSQL backend interprets BKI files. This description will be easier to understand if the postgres.bki file is at hand as an example.

BKI input consists of a sequence of commands. Commands are made up of a number of tokens, depending on the syntax of the command. Tokens are usually separated by whitespace, but need not be if there is no ambiguity. There is no special command separator; the next token that syntactically cannot belong to the preceding command starts a new one. (Usually you would put a new command on a new line, for clarity.) Tokens can be certain key words, special characters (parentheses, commas, etc.), identifiers, numbers, or single-quoted strings. Everything is case sensitive.

Lines starting with # are ignored.

Prev | Up | Next
71.2. System Catalog Initial Data | Home | 71.4. BKI Commands


71.4. BKI Commands
Prev Up Chapter 71. System Catalog Declarations and Initial Contents Home Next

71.4. BKI Commands

create tablename tableoid [bootstrap] [shared_relation] [rowtype_oid oid] (name1 = type1 [FORCE NOT NULL | FORCE NULL ] [, name2 = type2 [FORCE NOT NULL | FORCE NULL ], ...])

Create a table named tablename, and having the OID tableoid, with the columns given in parentheses.

The following column types are supported directly by bootstrap.c: bool, bytea, char (1 byte), name, int2, int4, regproc, regclass, regtype, text, oid, tid, xid, cid, int2vector, oidvector, _int4 (array), _text (array), _oid (array), _char (array), _aclitem (array). Although it is possible to create tables containing columns of other types, this cannot be done until after pg_type has been created and filled with appropriate entries. (That effectively means that only these column types can be used in bootstrap catalogs, but non-bootstrap catalogs can contain any built-in type.)

When bootstrap is specified, the table will only be created on disk; nothing is entered into pg_class, pg_attribute, etc, for it. Thus the table will not be accessible by ordinary SQL operations until such entries are made the hard way (with insert commands). This option is used for creating pg_class etc themselves.

The table is created as shared if shared_relation is specified. The table's row type OID (pg_type OID) can optionally be specified via the rowtype_oid clause; if not specified, an OID is automatically generated for it. (The rowtype_oid clause is useless if bootstrap is specified, but it can be provided anyway for documentation.)

open tablename

Open the table named tablename for insertion of data. Any currently open table is closed.

close tablename

Close the open table. The name of the table must be given as a cross-check.

insert ( [oid_value] value1 value2 ... )

Insert a new row into the open table using value1, value2, etc., for its column values.

NULL values can be specified using the special key word _null_. Values that do not look like identifiers or digit strings must be single-quoted. (To include a single quote in a value, write it twice. Escape-string-style backslash escapes are allowed in the string, too.)

declare [unique] index indexname indexoid on tablename using amname ( opclass1 name1 [, ...] )

Create an index named indexname, having OID indexoid, on the table named tablename, using the amname access method. The fields to index are called name1, name2 etc., and the operator classes to use are opclass1, opclass2 etc., respectively. The index file is created and appropriate catalog entries are made for it, but the index contents are not initialized by this command.

declare toast toasttableoid toastindexoid on tablename

Create a TOAST table for the table named tablename. The TOAST table is assigned OID toasttableoid and its index is assigned OID toastindexoid. As with declare index, filling of the index is postponed.

build indices

Fill in the indices that have previously been declared.

Prev | Up | Next
71.3. BKI File Format | Home | 71.5. Structure of the Bootstrap BKI File


71.5. Structure of the Bootstrap BKI File
Prev Up Chapter 71. System Catalog Declarations and Initial Contents Home Next

71.5. Structure of the Bootstrap BKI File

The open command cannot be used until the tables it uses exist and have entries for the table that is to be opened. (These minimum tables are pg_class, pg_attribute, pg_proc, and pg_type.) To allow those tables themselves to be filled, create with the bootstrap option implicitly opens the created table for data insertion.

Also, the declare index and declare toast commands cannot be used until the system catalogs they need have been created and filled in.

Thus, the structure of the postgres.bki file has to be:

  1. create bootstrap one of the critical tables

  2. insert data describing at least the critical tables

  3. close

  4. Repeat for the other critical tables.

  5. create (without bootstrap) a noncritical table

  6. open

  7. insert desired data

  8. close

  9. Repeat for the other noncritical tables.

  10. Define indexes and toast tables.

  11. build indices

There are doubtless other, undocumented ordering dependencies.

Prev | Up | Next
71.4. BKI Commands | Home | 71.6. BKI Example


71.6. BKI Example
Prev Up Chapter 71. System Catalog Declarations and Initial Contents Home Next

71.6. BKI Example

The following sequence of commands will create the table test_table with OID 420, having three columns oid, cola and colb of type oid, int4 and text, respectively, and insert two rows into the table:

create test_table 420 (oid = oid, cola = int4, colb = text)
open test_table
insert ( 421 1 'value 1' )
insert ( 422 2 _null_ )
close test_table

Prev | Up | Next
71.5. Structure of the Bootstrap BKI File | Home | Chapter 72. How the Planner Uses Statistics


Chapter 72. How the Planner Uses Statistics
Prev Up Part VII. Internals Home Next

Chapter 72. How the Planner Uses Statistics

Table of Contents

72.1. Row Estimation Examples
72.2. Multivariate Statistics Examples
72.2.1. Functional Dependencies
72.2.2. Multivariate N-Distinct Counts
72.2.3. MCV Lists
72.3. Planner Statistics and Security

This chapter builds on the material covered in Section 14.1 and Section 14.2 to show some additional details about how the planner uses the system statistics to estimate the number of rows each part of a query might return. This is a significant part of the planning process, providing much of the raw material for cost calculation.

The intent of this chapter is not to document the code in detail, but to present an overview of how it works. This will perhaps ease the learning curve for someone who subsequently wishes to read the code.

Prev | Up | Next
71.6. BKI Example | Home | 72.1. Row Estimation Examples


72.2. Multivariate Statistics Examples
Prev Up Chapter 72. How the Planner Uses Statistics Home Next

72.2. Multivariate Statistics Examples

72.2.1. Functional Dependencies
72.2.2. Multivariate N-Distinct Counts
72.2.3. MCV Lists

72.2.1. Functional Dependencies

Multivariate correlation can be demonstrated with a very simple data set — a table with two columns, both containing the same values:

CREATE TABLE t (a INT, b INT);
INSERT INTO t SELECT i % 100, i % 100 FROM generate_series(1, 10000) s(i);
ANALYZE t;

As explained in Section 14.2, the planner can determine cardinality of t using the number of pages and rows obtained from pg_class:

SELECT relpages, reltuples FROM pg_class WHERE relname = 't';

 relpages | reltuples
----------+-----------
       45 |     10000

The data distribution is very simple; there are only 100 distinct values in each column, uniformly distributed.

The following example shows the result of estimating a WHERE condition on the a column:

EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1;
                                 QUERY PLAN
-------------------------------------------------------------------​------------
 Seq Scan on t  (cost=0.00..170.00 rows=100 width=8) (actual rows=100 loops=1)
   Filter: (a = 1)
   Rows Removed by Filter: 9900

The planner examines the condition and determines the selectivity of this clause to be 1%. By comparing this estimate and the actual number of rows, we see that the estimate is very accurate (in fact exact, as the table is very small). Changing the WHERE condition to use the b column, an identical plan is generated. But observe what happens if we apply the same condition on both columns, combining them with AND:

EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 1;
                                 QUERY PLAN
-------------------------------------------------------------------​----------
 Seq Scan on t  (cost=0.00..195.00 rows=1 width=8) (actual rows=100 loops=1)
   Filter: ((a = 1) AND (b = 1))
   Rows Removed by Filter: 9900

The planner estimates the selectivity for each condition individually, arriving at the same 1% estimates as above. Then it assumes that the conditions are independent, and so it multiplies their selectivities, producing a final selectivity estimate of just 0.01%. This is a significant underestimate, as the actual number of rows matching the conditions (100) is two orders of magnitude higher.

This problem can be fixed by creating a statistics object that directs ANALYZE to calculate functional-dependency multivariate statistics on the two columns:

CREATE STATISTICS stts (dependencies) ON a, b FROM t;
ANALYZE t;
EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 1;
                                  QUERY PLAN
-------------------------------------------------------------------​------------
 Seq Scan on t  (cost=0.00..195.00 rows=100 width=8) (actual rows=100 loops=1)
   Filter: ((a = 1) AND (b = 1))
   Rows Removed by Filter: 9900

72.2.2. Multivariate N-Distinct Counts

A similar problem occurs with estimation of the cardinality of sets of multiple columns, such as the number of groups that would be generated by a GROUP BY clause. When GROUP BY lists a single column, the n-distinct estimate (which is visible as the estimated number of rows returned by the HashAggregate node) is very accurate:

EXPLAIN (ANALYZE, TIMING OFF) SELECT COUNT(*) FROM t GROUP BY a;
                                       QUERY PLAN
-------------------------------------------------------------------​----------------------
 HashAggregate  (cost=195.00..196.00 rows=100 width=12) (actual rows=100 loops=1)
   Group Key: a
   ->  Seq Scan on t  (cost=0.00..145.00 rows=10000 width=4) (actual rows=10000 loops=1)

But without multivariate statistics, the estimate for the number of groups in a query with two columns in GROUP BY, as in the following example, is off by an order of magnitude:

EXPLAIN (ANALYZE, TIMING OFF) SELECT COUNT(*) FROM t GROUP BY a, b;
                                       QUERY PLAN
-------------------------------------------------------------------​-------------------------
 HashAggregate  (cost=220.00..230.00 rows=1000 width=16) (actual rows=100 loops=1)
   Group Key: a, b
   ->  Seq Scan on t  (cost=0.00..145.00 rows=10000 width=8) (actual rows=10000 loops=1)

By redefining the statistics object to include n-distinct counts for the two columns, the estimate is much improved:

DROP STATISTICS stts;
CREATE STATISTICS stts (dependencies, ndistinct) ON a, b FROM t;
ANALYZE t;
EXPLAIN (ANALYZE, TIMING OFF) SELECT COUNT(*) FROM t GROUP BY a, b;
                                       QUERY PLAN
-------------------------------------------------------------------​-------------------------
 HashAggregate  (cost=220.00..221.00 rows=100 width=16) (actual rows=100 loops=1)
   Group Key: a, b
   ->  Seq Scan on t  (cost=0.00..145.00 rows=10000 width=8) (actual rows=10000 loops=1)

72.2.3. MCV Lists

As explained in Section 72.2.1, functional dependencies are very cheap and efficient type of statistics, but their main limitation is their global nature (only tracking dependencies at the column level, not between individual column values).

This section introduces multivariate variant of MCV (most-common values) lists, a straightforward extension of the per-column statistics described in Section 72.1. These statistics address the limitation by storing individual values, but it is naturally more expensive, both in terms of building the statistics in ANALYZE, storage and planning time.

Let's look at the query from Section 72.2.1 again, but this time with a MCV list created on the same set of columns (be sure to drop the functional dependencies, to make sure the planner uses the newly created statistics).

DROP STATISTICS stts;
CREATE STATISTICS stts2 (mcv) ON a, b FROM t;
ANALYZE t;
EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 1;
                                   QUERY PLAN
-------------------------------------------------------------------​------------
 Seq Scan on t  (cost=0.00..195.00 rows=100 width=8) (actual rows=100 loops=1)
   Filter: ((a = 1) AND (b = 1))
   Rows Removed by Filter: 9900

The estimate is as accurate as with the functional dependencies, mostly thanks to the table being fairly small and having a simple distribution with a low number of distinct values. Before looking at the second query, which was not handled by functional dependencies particularly well, let's inspect the MCV list a bit.

Inspecting the MCV list is possible using pg_mcv_list_items set-returning function.

SELECT m.* FROM pg_statistic_ext join pg_statistic_ext_data on (oid = stxoid),
                pg_mcv_list_items(stxdmcv) m WHERE stxname = 'stts2';
 index |  values  | nulls | frequency | base_frequency
-------+----------+-------+-----------+----------------
     0 | {0, 0}   | {f,f} |      0.01 |         0.0001
     1 | {1, 1}   | {f,f} |      0.01 |         0.0001
   ...
    49 | {49, 49} | {f,f} |      0.01 |         0.0001
    50 | {50, 50} | {f,f} |      0.01 |         0.0001
   ...
    97 | {97, 97} | {f,f} |      0.01 |         0.0001
    98 | {98, 98} | {f,f} |      0.01 |         0.0001
    99 | {99, 99} | {f,f} |      0.01 |         0.0001
(100 rows)

This confirms there are 100 distinct combinations in the two columns, and all of them are about equally likely (1% frequency for each one). The base frequency is the frequency computed from per-column statistics, as if there were no multi-column statistics. Had there been any null values in either of the columns, this would be identified in the nulls column.

When estimating the selectivity, the planner applies all the conditions on items in the MCV list, and then sums the frequencies of the matching ones. See mcv_clauselist_selectivity in src/backend/statistics/mcv.c for details.

Compared to functional dependencies, MCV lists have two major advantages. Firstly, the list stores actual values, making it possible to decide which combinations are compatible.

EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a = 1 AND b = 10;
                                 QUERY PLAN
-------------------------------------------------------------------​--------
 Seq Scan on t  (cost=0.00..195.00 rows=1 width=8) (actual rows=0 loops=1)
   Filter: ((a = 1) AND (b = 10))
   Rows Removed by Filter: 10000

Secondly, MCV lists handle a wider range of clause types, not just equality clauses like functional dependencies. For example, consider the following range query for the same table:

EXPLAIN (ANALYZE, TIMING OFF) SELECT * FROM t WHERE a <= 49 AND b > 49;
                                QUERY PLAN
-------------------------------------------------------------------​--------
 Seq Scan on t  (cost=0.00..195.00 rows=1 width=8) (actual rows=0 loops=1)
   Filter: ((a <= 49) AND (b > 49))
   Rows Removed by Filter: 10000

Prev | Up | Next
72.1. Row Estimation Examples | Home | 72.3. Planner Statistics and Security


72.3. Planner Statistics and Security
Prev Up Chapter 72. How the Planner Uses Statistics Home Next

72.3. Planner Statistics and Security

Access to the table pg_statistic is restricted to superusers, so that ordinary users cannot learn about the contents of the tables of other users from it. Some selectivity estimation functions will use a user-provided operator (either the operator appearing in the query or a related operator) to analyze the stored statistics. For example, in order to determine whether a stored most common value is applicable, the selectivity estimator will have to run the appropriate = operator to compare the constant in the query to the stored value. Thus the data in pg_statistic is potentially passed to user-defined operators. An appropriately crafted operator can intentionally leak the passed operands (for example, by logging them or writing them to a different table), or accidentally leak them by showing their values in error messages, in either case possibly exposing data from pg_statistic to a user who should not be able to see it.

In order to prevent this, the following applies to all built-in selectivity estimation functions. When planning a query, in order to be able to use stored statistics, the current user must either have SELECT privilege on the table or the involved columns, or the operator used must be LEAKPROOF (more accurately, the function that the operator is based on). If not, then the selectivity estimator will behave as if no statistics are available, and the planner will proceed with default or fall-back assumptions.

If a user does not have the required privilege on the table or columns, then in many cases the query will ultimately receive a permission-denied error, in which case this mechanism is invisible in practice. But if the user is reading from a security-barrier view, then the planner might wish to check the statistics of an underlying table that is otherwise inaccessible to the user. In that case, the operator should be leak-proof or the statistics will not be used. There is no direct feedback about that, except that the plan might be suboptimal. If one suspects that this is the case, one could try running the query as a more privileged user, to see if a different plan results.

This restriction applies only to cases where the planner would need to execute a user-defined operator on one or more values from pg_statistic. Thus the planner is permitted to use generic statistical information, such as the fraction of null values or the number of distinct values in a column, regardless of access privileges.

Selectivity estimation functions contained in third-party extensions that potentially operate on statistics with user-defined operators should follow the same security rules. Consult the PostgreSQL source code for guidance.

Prev | Up | Next
72.2. Multivariate Statistics Examples | Home | Chapter 73. Backup Manifest Format


Chapter 73. Backup Manifest Format
Prev Up Part VII. Internals Home Next

Chapter 73. Backup Manifest Format

Table of Contents

73.1. Backup Manifest Top-level Object
73.2. Backup Manifest File Object
73.3. Backup Manifest WAL Range Object

The backup manifest generated by pg_basebackup is primarily intended to permit the backup to be verified using pg_verifybackup. However, it is also possible for other tools to read the backup manifest file and use the information contained therein for their own purposes. To that end, this chapter describes the format of the backup manifest file.

A backup manifest is a JSON document encoded as UTF-8. (Although in general JSON documents are required to be Unicode, PostgreSQL permits the json and jsonb data types to be used with any supported server encoding. There is no similar exception for backup manifests.) The JSON document is always an object; the keys that are present in this object are described in the next section.

Prev | Up | Next
72.3. Planner Statistics and Security | Home | 73.1. Backup Manifest Top-level Object


73.1. Backup Manifest Top-level Object
Prev Up Chapter 73. Backup Manifest Format Home Next

73.1. Backup Manifest Top-level Object

The backup manifest JSON document contains the following keys.

PostgreSQL-Backup-Manifest-Version

The associated value is always the integer 1.

Files

The associated value is always a list of objects, each describing one file that is present in the backup. No entries are present in this list for the WAL files that are needed in order to use the backup, or for the backup manifest itself. The structure of each object in the list is described in Section 73.2.

WAL-Ranges

The associated value is always a list of objects, each describing a range of WAL records that must be readable from a particular timeline in order to make use of the backup. The structure of these objects is further described in Section 73.3.

Manifest-Checksum

This key is always present on the last line of the backup manifest file. The associated value is a SHA256 checksum of all the preceding lines. We use a fixed checksum method here to make it possible for clients to do incremental parsing of the manifest. While a SHA256 checksum is significantly more expensive than a CRC32C checksum, the manifest should normally be small enough that the extra computation won't matter very much.

Prev | Up | Next
Chapter 73. Backup Manifest Format | Home | 73.2. Backup Manifest File Object


73.2. Backup Manifest File Object
Prev Up Chapter 73. Backup Manifest Format Home Next

73.2. Backup Manifest File Object

The object which describes a single file contains either a Path key or an Encoded-Path key. Normally, the Path key will be present. The associated string value is the path of the file relative to the root of the backup directory. Files located in a user-defined tablespace will have paths whose first two components are pg_tblspc and the OID of the tablespace. If the path is not a string that is legal in UTF-8, or if the user requests that encoded paths be used for all files, then the Encoded-Path key will be present instead. This stores the same data, but it is encoded as a string of hexadecimal digits. Each pair of hexadecimal digits in the string represents a single octet.

The following two keys are always present:

Size

The expected size of this file, as an integer.

Last-Modified

The last modification time of the file as reported by the server at the time of the backup. Unlike the other fields stored in the backup, this field is not used by pg_verifybackup. It is included only for informational purposes.

If the backup was taken with file checksums enabled, the following keys will be present:

Checksum-Algorithm

The checksum algorithm used to compute a checksum for this file. Currently, this will be the same for every file in the backup manifest, but this may change in future releases. At present, the supported checksum algorithms are CRC32C, SHA224, SHA256, SHA384, and SHA512.

Checksum

The checksum computed for this file, stored as a series of hexadecimal characters, two for each byte of the checksum.

Prev | Up | Next
73.1. Backup Manifest Top-level Object | Home | 73.3. Backup Manifest WAL Range Object


73.3. Backup Manifest WAL Range Object
Prev Up Chapter 73. Backup Manifest Format Home Next

73.3. Backup Manifest WAL Range Object

The object which describes a WAL range always has three keys:

Timeline

The timeline for this range of WAL records, as an integer.

Start-LSN

The LSN at which replay must begin on the indicated timeline in order to make use of this backup. The LSN is stored in the format normally used by PostgreSQL; that is, it is a string consisting of two strings of hexadecimal characters, each with a length of between 1 and 8, separated by a slash.

End-LSN

The earliest LSN at which replay on the indicated timeline may end when making use of this backup. This is stored in the same format as Start-LSN.

Ordinarily, there will be only a single WAL range. However, if a backup is taken from a standby which switches timelines during the backup due to an upstream promotion, it is possible for multiple ranges to be present, each with a different timeline. There will never be multiple WAL ranges present for the same timeline.

Prev | Up | Next
73.2. Backup Manifest File Object | Home | Part VIII. Appendixes


Part VIII. Appendixes
Prev Up PostgreSQL 14.4 Documentation Home Next

Part VIII. Appendixes

Table of Contents

A. PostgreSQL Error Codes
B. Date/Time Support
B.1. Date/Time Input Interpretation
B.2. Handling of Invalid or Ambiguous Timestamps
B.3. Date/Time Key Words
B.4. Date/Time Configuration Files
B.5. POSIX Time Zone Specifications
B.6. History of Units
B.7. Julian Dates
C. SQL Key Words
D. SQL Conformance
D.1. Supported Features
D.2. Unsupported Features
D.3. XML Limits and Conformance to SQL/XML
E. Release Notes
E.1. Release 14.4
E.2. Release 14.3
E.3. Release 14.2
E.4. Release 14.1
E.5. Release 14
E.6. Prior Releases
F. Additional Supplied Modules
F.1. adminpack
F.2. amcheck
F.3. auth_delay
F.4. auto_explain
F.5. bloom
F.6. btree_gin
F.7. btree_gist
F.8. citext
F.9. cube
F.10. dblink
F.11. dict_int
F.12. dict_xsyn
F.13. earthdistance
F.14. file_fdw
F.15. fuzzystrmatch
F.16. hstore
F.17. intagg
F.18. intarray
F.19. isn
F.20. lo
F.21. ltree
F.22. old_snapshot
F.23. pageinspect
F.24. passwordcheck
F.25. pg_buffercache
F.26. pgcrypto
F.27. pg_freespacemap
F.28. pg_prewarm
F.29. pgrowlocks
F.30. pg_stat_statements
F.31. pgstattuple
F.32. pg_surgery
F.33. pg_trgm
F.34. pg_visibility
F.35. postgres_fdw
F.36. seg
F.37. sepgsql
F.38. spi
F.39. sslinfo
F.40. tablefunc
F.41. tcn
F.42. test_decoding
F.43. tsm_system_rows
F.44. tsm_system_time
F.45. unaccent
F.46. uuid-ossp
F.47. xml2
G. Additional Supplied Programs
G.1. Client Applications
G.2. Server Applications
H. External Projects
H.1. Client Interfaces
H.2. Administration Tools
H.3. Procedural Languages
H.4. Extensions
I. The Source Code Repository
I.1. Getting the Source via Git
J. Documentation
J.1. DocBook
J.2. Tool Sets
J.3. Building the Documentation
J.4. Documentation Authoring
J.5. Style Guide
K. PostgreSQL Limits
L. Acronyms
M. Glossary
N. Color Support
N.1. When Color is Used
N.2. Configuring the Colors
O. Obsolete or Renamed Features
O.1. recovery.conf file merged into postgresql.conf
O.2. Default Roles Renamed to Predefined Roles
O.3. pg_xlogdump renamed to pg_waldump
O.4. pg_resetxlog renamed to pg_resetwal
O.5. pg_receivexlog renamed to pg_receivewal
Prev | Up | Next
73.3. Backup Manifest WAL Range Object | Home | Appendix A. PostgreSQL Error Codes


Appendix A. PostgreSQL Error Codes
Prev Up Part VIII. Appendixes Home Next

Appendix A. PostgreSQL Error Codes

All messages emitted by the PostgreSQL server are assigned five-character error codes that follow the SQL standard's conventions for “SQLSTATE” codes. Applications that need to know which error condition has occurred should usually test the error code, rather than looking at the textual error message. The error codes are less likely to change across PostgreSQL releases, and also are not subject to change due to localization of error messages. Note that some, but not all, of the error codes produced by PostgreSQL are defined by the SQL standard; some additional error codes for conditions not defined by the standard have been invented or borrowed from other databases.

According to the standard, the first two characters of an error code denote a class of errors, while the last three characters indicate a specific condition within that class. Thus, an application that does not recognize the specific error code might still be able to infer what to do from the error class.

Table A.1 lists all the error codes defined in PostgreSQL 14.4. (Some are not actually used at present, but are defined by the SQL standard.) The error classes are also shown. For each error class there is a “standard” error code having the last three characters 000. This code is used only for error conditions that fall within the class but do not have any more-specific code assigned.

The symbol shown in the column “Condition Name” is the condition name to use in PL/pgSQL. Condition names can be written in either upper or lower case. (Note that PL/pgSQL does not recognize warning, as opposed to error, condition names; those are classes 00, 01, and 02.)

For some types of errors, the server reports the name of a database object (a table, table column, data type, or constraint) associated with the error; for example, the name of the unique constraint that caused a unique_violation error. Such names are supplied in separate fields of the error report message so that applications need not try to extract them from the possibly-localized human-readable text of the message. As of PostgreSQL 9.3, complete coverage for this feature exists only for errors in SQLSTATE class 23 (integrity constraint violation), but this is likely to be expanded in future.

Table A.1. PostgreSQL Error Codes

Error Code | Condition Name
Class 00 — Successful Completion
00000 | successful_completion
Class 01 — Warning
01000 | warning
0100C | dynamic_result_sets_returned
01008 | implicit_zero_bit_padding
01003 | null_value_eliminated_in_set_function
01007 | privilege_not_granted
01006 | privilege_not_revoked
01004 | string_data_right_truncation
01P01 | deprecated_feature
Class 02 — No Data (this is also a warning class per the SQL standard)
02000 | no_data
02001 | no_additional_dynamic_result_sets_returned
Class 03 — SQL Statement Not Yet Complete
03000 | sql_statement_not_yet_complete
Class 08 — Connection Exception
08000 | connection_exception
08003 | connection_does_not_exist
08006 | connection_failure
08001 | sqlclient_unable_to_establish_sqlconnection
08004 | sqlserver_rejected_establishment_of_sqlconnection
08007 | transaction_resolution_unknown
08P01 | protocol_violation
Class 09 — Triggered Action Exception
09000 | triggered_action_exception
Class 0A — Feature Not Supported
0A000 | feature_not_supported
Class 0B — Invalid Transaction Initiation
0B000 | invalid_transaction_initiation
Class 0F — Locator Exception
0F000 | locator_exception
0F001 | invalid_locator_specification
Class 0L — Invalid Grantor
0L000 | invalid_grantor
0LP01 | invalid_grant_operation
Class 0P — Invalid Role Specification
0P000 | invalid_role_specification
Class 0Z — Diagnostics Exception
0Z000 | diagnostics_exception
0Z002 | stacked_diagnostics_accessed_without_active_handler
Class 20 — Case Not Found
20000 | case_not_found
Class 21 — Cardinality Violation
21000 | cardinality_violation
Class 22 — Data Exception
22000 | data_exception
2202E | array_subscript_error
22021 | character_not_in_repertoire
22008 | datetime_field_overflow
22012 | division_by_zero
22005 | error_in_assignment
2200B | escape_character_conflict
22022 | indicator_overflow
22015 | interval_field_overflow
2201E | invalid_argument_for_logarithm
22014 | invalid_argument_for_ntile_function
22016 | invalid_argument_for_nth_value_function
2201F | invalid_argument_for_power_function
2201G | invalid_argument_for_width_bucket_function
22018 | invalid_character_value_for_cast
22007 | invalid_datetime_format
22019 | invalid_escape_character
2200D | invalid_escape_octet
22025 | invalid_escape_sequence
22P06 | nonstandard_use_of_escape_character
22010 | invalid_indicator_parameter_value
22023 | invalid_parameter_value
22013 | invalid_preceding_or_following_size
2201B | invalid_regular_expression
2201W | invalid_row_count_in_limit_clause
2201X | invalid_row_count_in_result_offset_clause
2202H | invalid_tablesample_argument
2202G | invalid_tablesample_repeat
22009 | invalid_time_zone_displacement_value
2200C | invalid_use_of_escape_character
2200G | most_specific_type_mismatch
22004 | null_value_not_allowed
22002 | null_value_no_indicator_parameter
22003 | numeric_value_out_of_range
2200H | sequence_generator_limit_exceeded
22026 | string_data_length_mismatch
22001 | string_data_right_truncation
22011 | substring_error
22027 | trim_error
22024 | unterminated_c_string
2200F | zero_length_character_string
22P01 | floating_point_exception
22P02 | invalid_text_representation
22P03 | invalid_binary_representation
22P04 | bad_copy_file_format
22P05 | untranslatable_character
2200L | not_an_xml_document
2200M | invalid_xml_document
2200N | invalid_xml_content
2200S | invalid_xml_comment
2200T | invalid_xml_processing_instruction
22030 | duplicate_json_object_key_value
22031 | invalid_argument_for_sql_json_datetime_function
22032 | invalid_json_text
22033 | invalid_sql_json_subscript
22034 | more_than_one_sql_json_item
22035 | no_sql_json_item
22036 | non_numeric_sql_json_item
22037 | non_unique_keys_in_a_json_object
22038 | singleton_sql_json_item_required
22039 | sql_json_array_not_found
2203A | sql_json_member_not_found
2203B | sql_json_number_not_found
2203C | sql_json_object_not_found
2203D | too_many_json_array_elements
2203E | too_many_json_object_members
2203F | sql_json_scalar_required
Class 23 — Integrity Constraint Violation
23000 | integrity_constraint_violation
23001 | restrict_violation
23502 | not_null_violation
23503 | foreign_key_violation
23505 | unique_violation
23514 | check_violation
23P01 | exclusion_violation
Class 24 — Invalid Cursor State
24000 | invalid_cursor_state
Class 25 — Invalid Transaction State
25000 | invalid_transaction_state
25001 | active_sql_transaction
25002 | branch_transaction_already_active
25008 | held_cursor_requires_same_isolation_level
25003 | inappropriate_access_mode_for_branch_transaction
25004 | inappropriate_isolation_level_for_branch_transaction
25005 | no_active_sql_transaction_for_branch_transaction
25006 | read_only_sql_transaction
25007 | schema_and_data_statement_mixing_not_supported
25P01 | no_active_sql_transaction
25P02 | in_failed_sql_transaction
25P03 | idle_in_transaction_session_timeout
Class 26 — Invalid SQL Statement Name
26000 | invalid_sql_statement_name
Class 27 — Triggered Data Change Violation
27000 | triggered_data_change_violation
Class 28 — Invalid Authorization Specification
28000 | invalid_authorization_specification
28P01 | invalid_password
Class 2B — Dependent Privilege Descriptors Still Exist
2B000 | dependent_privilege_descriptors_still_exist
2BP01 | dependent_objects_still_exist
Class 2D — Invalid Transaction Termination
2D000 | invalid_transaction_termination
Class 2F — SQL Routine Exception
2F000 | sql_routine_exception
2F005 | function_executed_no_return_statement
2F002 | modifying_sql_data_not_permitted
2F003 | prohibited_sql_statement_attempted
2F004 | reading_sql_data_not_permitted
Class 34 — Invalid Cursor Name
34000 | invalid_cursor_name
Class 38 — External Routine Exception
38000 | external_routine_exception
38001 | containing_sql_not_permitted
38002 | modifying_sql_data_not_permitted
38003 | prohibited_sql_statement_attempted
38004 | reading_sql_data_not_permitted
Class 39 — External Routine Invocation Exception
39000 | external_routine_invocation_exception
39001 | invalid_sqlstate_returned
39004 | null_value_not_allowed
39P01 | trigger_protocol_violated
39P02 | srf_protocol_violated
39P03 | event_trigger_protocol_violated
Class 3B — Savepoint Exception
3B000 | savepoint_exception
3B001 | invalid_savepoint_specification
Class 3D — Invalid Catalog Name
3D000 | invalid_catalog_name
Class 3F — Invalid Schema Name
3F000 | invalid_schema_name
Class 40 — Transaction Rollback
40000 | transaction_rollback
40002 | transaction_integrity_constraint_violation
40001 | serialization_failure
40003 | statement_completion_unknown
40P01 | deadlock_detected
Class 42 — Syntax Error or Access Rule Violation
42000 | syntax_error_or_access_rule_violation
42601 | syntax_error
42501 | insufficient_privilege
42846 | cannot_coerce
42803 | grouping_error
42P20 | windowing_error
42P19 | invalid_recursion
42830 | invalid_foreign_key
42602 | invalid_name
42622 | name_too_long
42939 | reserved_name
42804 | datatype_mismatch
42P18 | indeterminate_datatype
42P21 | collation_mismatch
42P22 | indeterminate_collation
42809 | wrong_object_type
428C9 | generated_always
42703 | undefined_column
42883 | undefined_function
42P01 | undefined_table
42P02 | undefined_parameter
42704 | undefined_object
42701 | duplicate_column
42P03 | duplicate_cursor
42P04 | duplicate_database
42723 | duplicate_function
42P05 | duplicate_prepared_statement
42P06 | duplicate_schema
42P07 | duplicate_table
42712 | duplicate_alias
42710 | duplicate_object
42702 | ambiguous_column
42725 | ambiguous_function
42P08 | ambiguous_parameter
42P09 | ambiguous_alias
42P10 | invalid_column_reference
42611 | invalid_column_definition
42P11 | invalid_cursor_definition
42P12 | invalid_database_definition
42P13 | invalid_function_definition
42P14 | invalid_prepared_statement_definition
42P15 | invalid_schema_definition
42P16 | invalid_table_definition
42P17 | invalid_object_definition
Class 44 — WITH CHECK OPTION Violation
44000 | with_check_option_violation
Class 53 — Insufficient Resources
53000 | insufficient_resources
53100 | disk_full
53200 | out_of_memory
53300 | too_many_connections
53400 | configuration_limit_exceeded
Class 54 — Program Limit Exceeded
54000 | program_limit_exceeded
54001 | statement_too_complex
54011 | too_many_columns
54023 | too_many_arguments
Class 55 — Object Not In Prerequisite State
55000 | object_not_in_prerequisite_state
55006 | object_in_use
55P02 | cant_change_runtime_param
55P03 | lock_not_available
55P04 | unsafe_new_enum_value_usage
Class 57 — Operator Intervention
57000 | operator_intervention
57014 | query_canceled
57P01 | admin_shutdown
57P02 | crash_shutdown
57P03 | cannot_connect_now
57P04 | database_dropped
57P05 | idle_session_timeout
Class 58 — System Error (errors external to PostgreSQL itself)
58000 | system_error
58030 | io_error
58P01 | undefined_file
58P02 | duplicate_file
Class 72 — Snapshot Failure
72000 | snapshot_too_old
Class F0 — Configuration File Error
F0000 | config_file_error
F0001 | lock_file_exists
Class HV — Foreign Data Wrapper Error (SQL/MED)
HV000 | fdw_error
HV005 | fdw_column_name_not_found
HV002 | fdw_dynamic_parameter_value_needed
HV010 | fdw_function_sequence_error
HV021 | fdw_inconsistent_descriptor_information
HV024 | fdw_invalid_attribute_value
HV007 | fdw_invalid_column_name
HV008 | fdw_invalid_column_number
HV004 | fdw_invalid_data_type
HV006 | fdw_invalid_data_type_descriptors
HV091 | fdw_invalid_descriptor_field_identifier
HV00B | fdw_invalid_handle
HV00C | fdw_invalid_option_index
HV00D | fdw_invalid_option_name
HV090 | fdw_invalid_string_length_or_buffer_length
HV00A | fdw_invalid_string_format
HV009 | fdw_invalid_use_of_null_pointer
HV014 | fdw_too_many_handles
HV001 | fdw_out_of_memory
HV00P | fdw_no_schemas
HV00J | fdw_option_name_not_found
HV00K | fdw_reply_handle
HV00Q | fdw_schema_not_found
HV00R | fdw_table_not_found
HV00L | fdw_unable_to_create_execution
HV00M | fdw_unable_to_create_reply
HV00N | fdw_unable_to_establish_connection
Class P0 — PL/pgSQL Error
P0000 | plpgsql_error
P0001 | raise_exception
P0002 | no_data_found
P0003 | too_many_rows
P0004 | assert_failure
Class XX — Internal Error
XX000 | internal_error
XX001 | data_corrupted
XX002 | index_corrupted

Prev | Up | Next
Part VIII. Appendixes | Home | Appendix B. Date/Time Support


Appendix B. Date/Time Support
Prev Up Part VIII. Appendixes Home Next

Appendix B. Date/Time Support

Table of Contents

B.1. Date/Time Input Interpretation
B.2. Handling of Invalid or Ambiguous Timestamps
B.3. Date/Time Key Words
B.4. Date/Time Configuration Files
B.5. POSIX Time Zone Specifications
B.6. History of Units
B.7. Julian Dates

PostgreSQL uses an internal heuristic parser for all date/time input support. Dates and times are input as strings, and are broken up into distinct fields with a preliminary determination of what kind of information can be in the field. Each field is interpreted and either assigned a numeric value, ignored, or rejected. The parser contains internal lookup tables for all textual fields, including months, days of the week, and time zones.

This appendix includes information on the content of these lookup tables and describes the steps used by the parser to decode dates and times.

Prev | Up | Next
Appendix A. PostgreSQL Error Codes | Home | B.1. Date/Time Input Interpretation


B.1. Date/Time Input Interpretation
Prev Up Appendix B. Date/Time Support Home Next

B.1. Date/Time Input Interpretation

Date/time input strings are decoded using the following procedure.

  1. Break the input string into tokens and categorize each token as a string, time, time zone, or number.

    1. If the numeric token contains a colon (:), this is a time string. Include all subsequent digits and colons.

    2. If the numeric token contains a dash (-), slash (/), or two or more dots (.), this is a date string which might have a text month. If a date token has already been seen, it is instead interpreted as a time zone name (e.g., America/New_York).

    3. If the token is numeric only, then it is either a single field or an ISO 8601 concatenated date (e.g., 19990113 for January 13, 1999) or time (e.g., 141516 for 14:15:16).

    4. If the token starts with a plus (+) or minus (-), then it is either a numeric time zone or a special field.

  2. If the token is an alphabetic string, match up with possible strings:

    1. See if the token matches any known time zone abbreviation. These abbreviations are supplied by the configuration file described in Section B.4.

    2. If not found, search an internal table to match the token as either a special string (e.g., today), day (e.g., Thursday), month (e.g., January), or noise word (e.g., at, on).

    3. If still not found, throw an error.

  3. When the token is a number or number field:

    1. If there are eight or six digits, and if no other date fields have been previously read, then interpret as a “concatenated date” (e.g., 19990118 or 990118). The interpretation is YYYYMMDD or YYMMDD.

    2. If the token is three digits and a year has already been read, then interpret as day of year.

    3. If four or six digits and a year has already been read, then interpret as a time (HHMM or HHMMSS).

    4. If three or more digits and no date fields have yet been found, interpret as a year (this forces yy-mm-dd ordering of the remaining date fields).

    5. Otherwise the date field ordering is assumed to follow the DateStyle setting: mm-dd-yy, dd-mm-yy, or yy-mm-dd. Throw an error if a month or day field is found to be out of range.

  4. If BC has been specified, negate the year and add one for internal storage. (There is no year zero in the Gregorian calendar, so numerically 1 BC becomes year zero.)

  5. If BC was not specified, and if the year field was two digits in length, then adjust the year to four digits. If the field is less than 70, then add 2000, otherwise add 1900.

    Tip

    Gregorian years AD 1–99 can be entered by using 4 digits with leading zeros (e.g., 0099 is AD 99).

Prev | Up | Next
Appendix B. Date/Time Support | Home | B.2. Handling of Invalid or Ambiguous Timestamps


B.2. Handling of Invalid or Ambiguous Timestamps
Prev Up Appendix B. Date/Time Support Home Next

B.2. Handling of Invalid or Ambiguous Timestamps

Ordinarily, if a date/time string is syntactically valid but contains out-of-range field values, an error will be thrown. For example, input specifying the 31st of February will be rejected.

During a daylight-savings-time transition, it is possible for a seemingly valid timestamp string to represent a nonexistent or ambiguous timestamp. Such cases are not rejected; the ambiguity is resolved by determining which UTC offset to apply. For example, supposing that the TimeZone parameter is set to America/New_York, consider

=> SELECT '2018-03-11 02:30'::timestamptz;
      timestamptz
------------------------
 2018-03-11 03:30:00-04
(1 row)

Because that day was a spring-forward transition date in that time zone, there was no civil time instant 2:30AM; clocks jumped forward from 2AM EST to 3AM EDT. PostgreSQL interprets the given time as if it were standard time (UTC-5), which then renders as 3:30AM EDT (UTC-4).

Conversely, consider the behavior during a fall-back transition:

=> SELECT '2018-11-04 01:30'::timestamptz;
      timestamptz
------------------------
 2018-11-04 01:30:00-05
(1 row)

On that date, there were two possible interpretations of 1:30AM; there was 1:30AM EDT, and then an hour later after clocks jumped back from 2AM EDT to 1AM EST, there was 1:30AM EST. Again, PostgreSQL interprets the given time as if it were standard time (UTC-5). We can force the other interpretation by specifying daylight-savings time:

=> SELECT '2018-11-04 01:30 EDT'::timestamptz;
      timestamptz
------------------------
 2018-11-04 01:30:00-04
(1 row)

The precise rule that is applied in such cases is that an invalid timestamp that appears to fall within a jump-forward daylight savings transition is assigned the UTC offset that prevailed in the time zone just before the transition, while an ambiguous timestamp that could fall on either side of a jump-back transition is assigned the UTC offset that prevailed just after the transition. In most time zones this is equivalent to saying that “the standard-time interpretation is preferred when in doubt”.

In all cases, the UTC offset associated with a timestamp can be specified explicitly, using either a numeric UTC offset or a time zone abbreviation that corresponds to a fixed UTC offset. The rule just given applies only when it is necessary to infer a UTC offset for a time zone in which the offset varies.

Prev | Up | Next
B.1. Date/Time Input Interpretation | Home | B.3. Date/Time Key Words


B.3. Date/Time Key Words
Prev Up Appendix B. Date/Time Support Home Next

B.3. Date/Time Key Words

Table B.1 shows the tokens that are recognized as names of months.

Table B.1. Month Names

Month | Abbreviations
January | Jan
February | Feb
March | Mar
April | Apr
May |
June | Jun
July | Jul
August | Aug
September | Sep, Sept
October | Oct
November | Nov
December | Dec

Table B.2 shows the tokens that are recognized as names of days of the week.

Table B.2. Day of the Week Names

Day | Abbreviations
Sunday | Sun
Monday | Mon
Tuesday | Tue, Tues
Wednesday | Wed, Weds
Thursday | Thu, Thur, Thurs
Friday | Fri
Saturday | Sat

Table B.3 shows the tokens that serve various modifier purposes.

Table B.3. Date/Time Field Modifiers

Identifier | Description
AM | Time is before 12:00
AT | Ignored
JULIAN, JD, J | Next field is Julian Date
ON | Ignored
PM | Time is on or after 12:00
T | Next field is time

Prev | Up | Next
B.2. Handling of Invalid or Ambiguous Timestamps | Home | B.4. Date/Time Configuration Files


B.4. Date/Time Configuration Files
Prev Up Appendix B. Date/Time Support Home Next

B.4. Date/Time Configuration Files

Since timezone abbreviations are not well standardized, PostgreSQL provides a means to customize the set of abbreviations accepted by the server. The timezone_abbreviations run-time parameter determines the active set of abbreviations. While this parameter can be altered by any database user, the possible values for it are under the control of the database administrator — they are in fact names of configuration files stored in .../share/timezonesets/ of the installation directory. By adding or altering files in that directory, the administrator can set local policy for timezone abbreviations.

timezone_abbreviations can be set to any file name found in .../share/timezonesets/, if the file's name is entirely alphabetic. (The prohibition against non-alphabetic characters in timezone_abbreviations prevents reading files outside the intended directory, as well as reading editor backup files and other extraneous files.)

A timezone abbreviation file can contain blank lines and comments beginning with #. Non-comment lines must have one of these formats:

zone_abbreviation offset
zone_abbreviation offset D
zone_abbreviation time_zone_name
@INCLUDE file_name
@OVERRIDE

A zone_abbreviation is just the abbreviation being defined. An offset is an integer giving the equivalent offset in seconds from UTC, positive being east from Greenwich and negative being west. For example, -18000 would be five hours west of Greenwich, or North American east coast standard time. D indicates that the zone name represents local daylight-savings time rather than standard time.

Alternatively, a time_zone_name can be given, referencing a zone name defined in the IANA timezone database. The zone's definition is consulted to see whether the abbreviation is or has been in use in that zone, and if so, the appropriate meaning is used — that is, the meaning that was currently in use at the timestamp whose value is being determined, or the meaning in use immediately before that if it wasn't current at that time, or the oldest meaning if it was used only after that time. This behavior is essential for dealing with abbreviations whose meaning has historically varied. It is also allowed to define an abbreviation in terms of a zone name in which that abbreviation does not appear; then using the abbreviation is just equivalent to writing out the zone name.

Tip

Using a simple integer offset is preferred when defining an abbreviation whose offset from UTC has never changed, as such abbreviations are much cheaper to process than those that require consulting a time zone definition.

The @INCLUDE syntax allows inclusion of another file in the .../share/timezonesets/ directory. Inclusion can be nested, to a limited depth.

The @OVERRIDE syntax indicates that subsequent entries in the file can override previous entries (typically, entries obtained from included files). Without this, conflicting definitions of the same timezone abbreviation are considered an error.

In an unmodified installation, the file Default contains all the non-conflicting time zone abbreviations for most of the world. Additional files Australia and India are provided for those regions: these files first include the Default file and then add or modify abbreviations as needed.

For reference purposes, a standard installation also contains files Africa.txt, America.txt, etc, containing information about every time zone abbreviation known to be in use according to the IANA timezone database. The zone name definitions found in these files can be copied and pasted into a custom configuration file as needed. Note that these files cannot be directly referenced as timezone_abbreviations settings, because of the dot embedded in their names.

Note

If an error occurs while reading the time zone abbreviation set, no new value is applied and the old set is kept. If the error occurs while starting the database, startup fails.

Caution

Time zone abbreviations defined in the configuration file override non-timezone meanings built into PostgreSQL. For example, the Australia configuration file defines SAT (for South Australian Standard Time). When this file is active, SAT will not be recognized as an abbreviation for Saturday.

Caution

If you modify files in .../share/timezonesets/, it is up to you to make backups — a normal database dump will not include this directory.

Prev | Up | Next
B.3. Date/Time Key Words | Home | B.5. POSIX Time Zone Specifications


B.5. POSIX Time Zone Specifications
Prev Up Appendix B. Date/Time Support Home Next

B.5. POSIX Time Zone Specifications

PostgreSQL can accept time zone specifications that are written according to the POSIX standard's rules for the TZ environment variable. POSIX time zone specifications are inadequate to deal with the complexity of real-world time zone history, but there are sometimes reasons to use them.

A POSIX time zone specification has the form

STD offset [ DST [ dstoffset ] [ , rule ] ]

(For readability, we show spaces between the fields, but spaces should not be used in practice.) The fields are:

  • STD is the zone abbreviation to be used for standard time.

  • offset is the zone's standard-time offset from UTC.

  • DST is the zone abbreviation to be used for daylight-savings time. If this field and the following ones are omitted, the zone uses a fixed UTC offset with no daylight-savings rule.

  • dstoffset is the daylight-savings offset from UTC. This field is typically omitted, since it defaults to one hour less than the standard-time offset, which is usually the right thing.

  • rule defines the rule for when daylight savings is in effect, as described below.

In this syntax, a zone abbreviation can be a string of letters, such as EST, or an arbitrary string surrounded by angle brackets, such as <UTC-05>. Note that the zone abbreviations given here are only used for output, and even then only in some timestamp output formats. The zone abbreviations recognized in timestamp input are determined as explained in Section B.4.

The offset fields specify the hours, and optionally minutes and seconds, difference from UTC. They have the format hh[:mm[:ss]] optionally with a leading sign (+ or -). The positive sign is used for zones west of Greenwich. (Note that this is the opposite of the ISO-8601 sign convention used elsewhere in PostgreSQL.) hh can have one or two digits; mm and ss (if used) must have two.

The daylight-savings transition rule has the format

dstdate [ / dsttime ] , stddate [ / stdtime ]

(As before, spaces should not be included in practice.) The dstdate and dsttime fields define when daylight-savings time starts, while stddate and stdtime define when standard time starts. (In some cases, notably in zones south of the equator, the former might be later in the year than the latter.) The date fields have one of these formats:

n

A plain integer denotes a day of the year, counting from zero to 364, or to 365 in leap years.

J n

In this form, n counts from 1 to 365, and February 29 is not counted even if it is present. (Thus, a transition occurring on February 29 could not be specified this way. However, days after February have the same numbers whether it's a leap year or not, so that this form is usually more useful than the plain-integer form for transitions on fixed dates.)

M m.n.d

This form specifies a transition that always happens during the same month and on the same day of the week. m identifies the month, from 1 to 12. n specifies the n'th occurrence of the weekday identified by d. n is a number between 1 and 4, or 5 meaning the last occurrence of that weekday in the month (which could be the fourth or the fifth). d is a number between 0 and 6, with 0 indicating Sunday. For example, M3.2.0 means “the second Sunday in March”.

Note

The M format is sufficient to describe many common daylight-savings transition laws. But note that none of these variants can deal with daylight-savings law changes, so in practice the historical data stored for named time zones (in the IANA time zone database) is necessary to interpret past time stamps correctly.

The time fields in a transition rule have the same format as the offset fields described previously, except that they cannot contain signs. They define the current local time at which the change to the other time occurs. If omitted, they default to 02:00:00.

If a daylight-savings abbreviation is given but the transition rule field is omitted, the fallback behavior is to use the rule M3.2.0,M11.1.0, which corresponds to USA practice as of 2020 (that is, spring forward on the second Sunday of March, fall back on the first Sunday of November, both transitions occurring at 2AM prevailing time). Note that this rule does not give correct USA transition dates for years before 2007.

As an example, CET-1CEST,M3.5.0,M10.5.0/3 describes current (as of 2020) timekeeping practice in Paris. This specification says that standard time has the abbreviation CET and is one hour ahead (east) of UTC; daylight savings time has the abbreviation CEST and is implicitly two hours ahead of UTC; daylight savings time begins on the last Sunday in March at 2AM CET and ends on the last Sunday in October at 3AM CEST.

The four timezone names EST5EDT, CST6CDT, MST7MDT, and PST8PDT look like they are POSIX zone specifications. However, they actually are treated as named time zones because (for historical reasons) there are files by those names in the IANA time zone database. The practical implication of this is that these zone names will produce valid historical USA daylight-savings transitions, even when a plain POSIX specification would not.

One should be wary that it is easy to misspell a POSIX-style time zone specification, since there is no check on the reasonableness of the zone abbreviation(s). For example, SET TIMEZONE TO FOOBAR0 will work, leaving the system effectively using a rather peculiar abbreviation for UTC.

Prev | Up | Next
B.4. Date/Time Configuration Files | Home | B.6. History of Units


B.6. History of Units
Prev Up Appendix B. Date/Time Support Home Next

B.6. History of Units

The SQL standard states that “Within the definition of a ‘datetime literal’, the ‘datetime values’ are constrained by the natural rules for dates and times according to the Gregorian calendar”. PostgreSQL follows the SQL standard's lead by counting dates exclusively in the Gregorian calendar, even for years before that calendar was in use. This rule is known as the proleptic Gregorian calendar.

The Julian calendar was introduced by Julius Caesar in 45 BC. It was in common use in the Western world until the year 1582, when countries started changing to the Gregorian calendar. In the Julian calendar, the tropical year is approximated as 365 1/4 days = 365.25 days. This gives an error of about 1 day in 128 years.

The accumulating calendar error prompted Pope Gregory XIII to reform the calendar in accordance with instructions from the Council of Trent. In the Gregorian calendar, the tropical year is approximated as 365 + 97 / 400 days = 365.2425 days. Thus it takes approximately 3300 years for the tropical year to shift one day with respect to the Gregorian calendar.

The approximation 365+97/400 is achieved by having 97 leap years every 400 years, using the following rules:

Every year divisible by 4 is a leap year.
However, every year divisible by 100 is not a leap year.
However, every year divisible by 400 is a leap year after all.

So, 1700, 1800, 1900, 2100, and 2200 are not leap years. But 1600, 2000, and 2400 are leap years. By contrast, in the older Julian calendar all years divisible by 4 are leap years.

The papal bull of February 1582 decreed that 10 days should be dropped from October 1582 so that 15 October should follow immediately after 4 October. This was observed in Italy, Poland, Portugal, and Spain. Other Catholic countries followed shortly after, but Protestant countries were reluctant to change, and the Greek Orthodox countries didn't change until the start of the 20th century. The reform was observed by Great Britain and its dominions (including what is now the USA) in 1752. Thus 2 September 1752 was followed by 14 September 1752. This is why Unix systems that have the cal program produce the following:

$ cal 9 1752
   September 1752
 S  M Tu  W Th  F  S
       1  2 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30

But, of course, this calendar is only valid for Great Britain and dominions, not other places. Since it would be difficult and confusing to try to track the actual calendars that were in use in various places at various times, PostgreSQL does not try, but rather follows the Gregorian calendar rules for all dates, even though this method is not historically accurate.

Different calendars have been developed in various parts of the world, many predating the Gregorian system. For example, the beginnings of the Chinese calendar can be traced back to the 14th century BC. Legend has it that the Emperor Huangdi invented that calendar in 2637 BC. The People's Republic of China uses the Gregorian calendar for civil purposes. The Chinese calendar is used for determining festivals.

Prev | Up | Next
B.5. POSIX Time Zone Specifications | Home | B.7. Julian Dates


B.7. Julian Dates
Prev Up Appendix B. Date/Time Support Home Next

B.7. Julian Dates

The Julian Date system is a method for numbering days. It is unrelated to the Julian calendar, though it is confusingly named similarly to that calendar. The Julian Date system was invented by the French scholar Joseph Justus Scaliger (1540–1609) and probably takes its name from Scaliger's father, the Italian scholar Julius Caesar Scaliger (1484–1558).

In the Julian Date system, each day has a sequential number, starting from JD 0 (which is sometimes called the Julian Date). JD 0 corresponds to 1 January 4713 BC in the Julian calendar, or 24 November 4714 BC in the Gregorian calendar. Julian Date counting is most often used by astronomers for labeling their nightly observations, and therefore a date runs from noon UTC to the next noon UTC, rather than from midnight to midnight: JD 0 designates the 24 hours from noon UTC on 24 November 4714 BC to noon UTC on 25 November 4714 BC.

Although PostgreSQL supports Julian Date notation for input and output of dates (and also uses Julian dates for some internal datetime calculations), it does not observe the nicety of having dates run from noon to noon. PostgreSQL treats a Julian Date as running from local midnight to local midnight, the same as a normal date.

This definition does, however, provide a way to obtain the astronomical definition when you need it: do the arithmetic in time zone UTC+12. For example,

=> SELECT extract(julian from '2021-06-23 7:00:00-04'::timestamptz at time zone 'UTC+12');
           extract
------------------------------
 2459388.95833333333333333333
(1 row)
=> SELECT extract(julian from '2021-06-23 8:00:00-04'::timestamptz at time zone 'UTC+12');
               extract
--------------------------------------
 2459389.0000000000000000000000000000
(1 row)
=> SELECT extract(julian from date '2021-06-23');
 extract
---------
 2459389
(1 row)

Prev | Up | Next
B.6. History of Units | Home | Appendix C. SQL Key Words


Appendix D. SQL Conformance
Prev Up Part VIII. Appendixes Home Next

Appendix D. SQL Conformance

Table of Contents

D.1. Supported Features
D.2. Unsupported Features
D.3. XML Limits and Conformance to SQL/XML
D.3.1. Queries Are Restricted to XPath 1.0
D.3.2. Incidental Limits of the Implementation

This section attempts to outline to what extent PostgreSQL conforms to the current SQL standard. The following information is not a full statement of conformance, but it presents the main topics in as much detail as is both reasonable and useful for users.

The formal name of the SQL standard is ISO/IEC 9075 “Database Language SQL”. A revised version of the standard is released from time to time; the most recent update appearing in 2016. The 2016 version is referred to as ISO/IEC 9075:2016, or simply as SQL:2016. The versions prior to that were SQL:2011, SQL:2008, SQL:2006, SQL:2003, SQL:1999, and SQL-92. Each version replaces the previous one, so claims of conformance to earlier versions have no official merit. PostgreSQL development aims for conformance with the latest official version of the standard where such conformance does not contradict traditional features or common sense. Many of the features required by the SQL standard are supported, though sometimes with slightly differing syntax or function. Further moves towards conformance can be expected over time.

SQL-92 defined three feature sets for conformance: Entry, Intermediate, and Full. Most database management systems claiming SQL standard conformance were conforming at only the Entry level, since the entire set of features in the Intermediate and Full levels was either too voluminous or in conflict with legacy behaviors.

Starting with SQL:1999, the SQL standard defines a large set of individual features rather than the ineffectively broad three levels found in SQL-92. A large subset of these features represents the “Core” features, which every conforming SQL implementation must supply. The rest of the features are purely optional.

The standard versions beginning with SQL:2003 are also split into a number of parts. Each is known by a shorthand name. Note that these parts are not consecutively numbered.

  • ISO/IEC 9075-1 Framework (SQL/Framework)

  • ISO/IEC 9075-2 Foundation (SQL/Foundation)

  • ISO/IEC 9075-3 Call Level Interface (SQL/CLI)

  • ISO/IEC 9075-4 Persistent Stored Modules (SQL/PSM)

  • ISO/IEC 9075-9 Management of External Data (SQL/MED)

  • ISO/IEC 9075-10 Object Language Bindings (SQL/OLB)

  • ISO/IEC 9075-11 Information and Definition Schemas (SQL/Schemata)

  • ISO/IEC 9075-13 Routines and Types using the Java Language (SQL/JRT)

  • ISO/IEC 9075-14 XML-related specifications (SQL/XML)

  • ISO/IEC 9075-15 Multi-dimensional arrays (SQL/MDA)

The PostgreSQL core covers parts 1, 2, 9, 11, and 14. Part 3 is covered by the ODBC driver, and part 13 is covered by the PL/Java plug-in, but exact conformance is currently not being verified for these components. There are currently no implementations of parts 4, 10, and 15 for PostgreSQL.

PostgreSQL supports most of the major features of SQL:2016. Out of 177 mandatory features required for full Core conformance, PostgreSQL conforms to at least 170. In addition, there is a long list of supported optional features. It might be worth noting that at the time of writing, no current version of any database management system claims full conformance to Core SQL:2016.

In the following two sections, we provide a list of those features that PostgreSQL supports, followed by a list of the features defined in SQL:2016 which are not yet supported in PostgreSQL. Both of these lists are approximate: There might be minor details that are nonconforming for a feature that is listed as supported, and large parts of an unsupported feature might in fact be implemented. The main body of the documentation always contains the most accurate information about what does and does not work.

Note

Feature codes containing a hyphen are subfeatures. Therefore, if a particular subfeature is not supported, the main feature is listed as unsupported even if some other subfeatures are supported.

Prev | Up | Next
Appendix C. SQL Key Words | Home | D.1. Supported Features


D.1. Supported Features
Prev Up Appendix D. SQL Conformance Home Next

D.1. Supported Features

Identifier | Core? | Description | Comment
B012 |  | Embedded C |
B021 |  | Direct SQL |
E011 | Core | Numeric data types |
E011-01 | Core | INTEGER and SMALLINT data types |
E011-02 | Core | REAL, DOUBLE PRECISION, and FLOAT data types |
E011-03 | Core | DECIMAL and NUMERIC data types |
E011-04 | Core | Arithmetic operators |
E011-05 | Core | Numeric comparison |
E011-06 | Core | Implicit casting among the numeric data types |
E021 | Core | Character data types |
E021-01 | Core | CHARACTER data type |
E021-02 | Core | CHARACTER VARYING data type |
E021-03 | Core | Character literals |
E021-04 | Core | CHARACTER_LENGTH function | trims trailing spaces from CHARACTER values before counting
E021-05 | Core | OCTET_LENGTH function |
E021-06 | Core | SUBSTRING function |
E021-07 | Core | Character concatenation |
E021-08 | Core | UPPER and LOWER functions |
E021-09 | Core | TRIM function |
E021-10 | Core | Implicit casting among the character string types |
E021-11 | Core | POSITION function |
E021-12 | Core | Character comparison |
E031 | Core | Identifiers |
E031-01 | Core | Delimited identifiers |
E031-02 | Core | Lower case identifiers |
E031-03 | Core | Trailing underscore |
E051 | Core | Basic query specification |
E051-01 | Core | SELECT DISTINCT |
E051-02 | Core | GROUP BY clause |
E051-04 | Core | GROUP BY can contain columns not in <select list> |
E051-05 | Core | Select list items can be renamed |
E051-06 | Core | HAVING clause |
E051-07 | Core | Qualified * in select list |
E051-08 | Core | Correlation names in the FROM clause |
E051-09 | Core | Rename columns in the FROM clause |
E061 | Core | Basic predicates and search conditions |
E061-01 | Core | Comparison predicate |
E061-02 | Core | BETWEEN predicate |
E061-03 | Core | IN predicate with list of values |
E061-04 | Core | LIKE predicate |
E061-05 | Core | LIKE predicate ESCAPE clause |
E061-06 | Core | NULL predicate |
E061-07 | Core | Quantified comparison predicate |
E061-08 | Core | EXISTS predicate |
E061-09 | Core | Subqueries in comparison predicate |
E061-11 | Core | Subqueries in IN predicate |
E061-12 | Core | Subqueries in quantified comparison predicate |
E061-13 | Core | Correlated subqueries |
E061-14 | Core | Search condition |
E071 | Core | Basic query expressions |
E071-01 | Core | UNION DISTINCT table operator |
E071-02 | Core | UNION ALL table operator |
E071-03 | Core | EXCEPT DISTINCT table operator |
E071-05 | Core | Columns combined via table operators need not have exactly the same data type |
E071-06 | Core | Table operators in subqueries |
E081 | Core | Basic Privileges |
E081-01 | Core | SELECT privilege |
E081-02 | Core | DELETE privilege |
E081-03 | Core | INSERT privilege at the table level |
E081-04 | Core | UPDATE privilege at the table level |
E081-05 | Core | UPDATE privilege at the column level |
E081-06 | Core | REFERENCES privilege at the table level |
E081-07 | Core | REFERENCES privilege at the column level |
E081-08 | Core | WITH GRANT OPTION |
E081-09 | Core | USAGE privilege |
E081-10 | Core | EXECUTE privilege |
E091 | Core | Set functions |
E091-01 | Core | AVG |
E091-02 | Core | COUNT |
E091-03 | Core | MAX |
E091-04 | Core | MIN |
E091-05 | Core | SUM |
E091-06 | Core | ALL quantifier |
E091-07 | Core | DISTINCT quantifier |
E101 | Core | Basic data manipulation |
E101-01 | Core | INSERT statement |
E101-03 | Core | Searched UPDATE statement |
E101-04 | Core | Searched DELETE statement |
E111 | Core | Single row SELECT statement |
E121 | Core | Basic cursor support |
E121-01 | Core | DECLARE CURSOR |
E121-02 | Core | ORDER BY columns need not be in select list |
E121-03 | Core | Value expressions in ORDER BY clause |
E121-04 | Core | OPEN statement |
E121-06 | Core | Positioned UPDATE statement |
E121-07 | Core | Positioned DELETE statement |
E121-08 | Core | CLOSE statement |
E121-10 | Core | FETCH statement implicit NEXT |
E121-17 | Core | WITH HOLD cursors |
E131 | Core | Null value support (nulls in lieu of values) |
E141 | Core | Basic integrity constraints |
E141-01 | Core | NOT NULL constraints |
E141-02 | Core | UNIQUE constraints of NOT NULL columns |
E141-03 | Core | PRIMARY KEY constraints |
E141-04 | Core | Basic FOREIGN KEY constraint with the NO ACTION default for both referential delete action and referential update action |
E141-06 | Core | CHECK constraints |
E141-07 | Core | Column defaults |
E141-08 | Core | NOT NULL inferred on PRIMARY KEY |
E141-10 | Core | Names in a foreign key can be specified in any order |
E151 | Core | Transaction support |
E151-01 | Core | COMMIT statement |
E151-02 | Core | ROLLBACK statement |
E152 | Core | Basic SET TRANSACTION statement |
E152-01 | Core | SET TRANSACTION statement: ISOLATION LEVEL SERIALIZABLE clause |
E152-02 | Core | SET TRANSACTION statement: READ ONLY and READ WRITE clauses |
E153 | Core | Updatable queries with subqueries |
E161 | Core | SQL comments using leading double minus |
E171 | Core | SQLSTATE support |
E182 | Core | Host language binding |
F021 | Core | Basic information schema |
F021-01 | Core | COLUMNS view |
F021-02 | Core | TABLES view |
F021-03 | Core | VIEWS view |
F021-04 | Core | TABLE_CONSTRAINTS view |
F021-05 | Core | REFERENTIAL_CONSTRAINTS view |
F021-06 | Core | CHECK_CONSTRAINTS view |
F031 | Core | Basic schema manipulation |
F031-01 | Core | CREATE TABLE statement to create persistent base tables |
F031-02 | Core | CREATE VIEW statement |
F031-03 | Core | GRANT statement |
F031-04 | Core | ALTER TABLE statement: ADD COLUMN clause |
F031-13 | Core | DROP TABLE statement: RESTRICT clause |
F031-16 | Core | DROP VIEW statement: RESTRICT clause |
F031-19 | Core | REVOKE statement: RESTRICT clause |
F032 |  | CASCADE drop behavior |
F033 |  | ALTER TABLE statement: DROP COLUMN clause |
F034 |  | Extended REVOKE statement |
F034-01 |  | REVOKE statement performed by other than the owner of a schema object |
F034-02 |  | REVOKE statement: GRANT OPTION FOR clause |
F034-03 |  | REVOKE statement to revoke a privilege that the grantee has WITH GRANT OPTION |
F041 | Core | Basic joined table |
F041-01 | Core | Inner join (but not necessarily the INNER keyword) |
F041-02 | Core | INNER keyword |
F041-03 | Core | LEFT OUTER JOIN |
F041-04 | Core | RIGHT OUTER JOIN |
F041-05 | Core | Outer joins can be nested |
F041-07 | Core | The inner table in a left or right outer join can also be used in an inner join |
F041-08 | Core | All comparison operators are supported (rather than just =) |
F051 | Core | Basic date and time |
F051-01 | Core | DATE data type (including support of DATE literal) |
F051-02 | Core | TIME data type (including support of TIME literal) with fractional seconds precision of at least 0 |
F051-03 | Core | TIMESTAMP data type (including support of TIMESTAMP literal) with fractional seconds precision of at least 0 and 6 |
F051-04 | Core | Comparison predicate on DATE, TIME, and TIMESTAMP data types |
F051-05 | Core | Explicit CAST between datetime types and character string types |
F051-06 | Core | CURRENT_DATE |
F051-07 | Core | LOCALTIME |
F051-08 | Core | LOCALTIMESTAMP |
F052 |  | Intervals and datetime arithmetic |
F053 |  | OVERLAPS predicate |
F081 | Core | UNION and EXCEPT in views |
F111 |  | Isolation levels other than SERIALIZABLE |
F111-01 |  | READ UNCOMMITTED isolation level |
F111-02 |  | READ COMMITTED isolation level |
F111-03 |  | REPEATABLE READ isolation level |
F131 | Core | Grouped operations |
F131-01 | Core | WHERE, GROUP BY, and HAVING clauses supported in queries with grouped views |
F131-02 | Core | Multiple tables supported in queries with grouped views |
F131-03 | Core | Set functions supported in queries with grouped views |
F131-04 | Core | Subqueries with GROUP BY and HAVING clauses and grouped views |
F131-05 | Core | Single row SELECT with GROUP BY and HAVING clauses and grouped views |
F171 |  | Multiple schemas per user |
F181 | Core | Multiple module support |
F191 |  | Referential delete actions |
F200 |  | TRUNCATE TABLE statement |
F201 | Core | CAST function |
F202 |  | TRUNCATE TABLE: identity column restart option |
F221 | Core | Explicit defaults |
F222 |  | INSERT statement: DEFAULT VALUES clause |
F231 |  | Privilege tables |
F231-01 |  | TABLE_PRIVILEGES view |
F231-02 |  | COLUMN_PRIVILEGES view |
F231-03 |  | USAGE_PRIVILEGES view |
F251 |  | Domain support |
F261 | Core | CASE expression |
F261-01 | Core | Simple CASE |
F261-02 | Core | Searched CASE |
F261-03 | Core | NULLIF |
F261-04 | Core | COALESCE |
F262 |  | Extended CASE expression |
F271 |  | Compound character literals |
F281 |  | LIKE enhancements |
F302 |  | INTERSECT table operator |
F302-01 |  | INTERSECT DISTINCT table operator |
F302-02 |  | INTERSECT ALL table operator |
F304 |  | EXCEPT ALL table operator |
F311 | Core | Schema definition statement |
F311-01 | Core | CREATE SCHEMA |
F311-02 | Core | CREATE TABLE for persistent base tables |
F311-03 | Core | CREATE VIEW |
F311-04 | Core | CREATE VIEW: WITH CHECK OPTION |
F311-05 | Core | GRANT statement |
F321 |  | User authorization |
F341 |  | Usage tables |
F361 |  | Subprogram support |
F381 |  | Extended schema manipulation |
F381-01 |  | ALTER TABLE statement: ALTER COLUMN clause |
F381-02 |  | ALTER TABLE statement: ADD CONSTRAINT clause |
F381-03 |  | ALTER TABLE statement: DROP CONSTRAINT clause |
F382 |  | Alter column data type |
F383 |  | Set column not null clause |
F384 |  | Drop identity property clause |
F385 |  | Drop column generation expression clause |
F386 |  | Set identity column generation clause |
F391 |  | Long identifiers |
F392 |  | Unicode escapes in identifiers |
F393 |  | Unicode escapes in literals |
F394 |  | Optional normal form specification |
F401 |  | Extended joined table |
F401-01 |  | NATURAL JOIN |
F401-02 |  | FULL OUTER JOIN |
F401-04 |  | CROSS JOIN |
F402 |  | Named column joins for LOBs, arrays, and multisets |
F404 |  | Range variable for common column names |
F411 |  | Time zone specification | differences regarding literal interpretation
F421 |  | National character |
F431 |  | Read-only scrollable cursors |
F431-01 |  | FETCH with explicit NEXT |
F431-02 |  | FETCH FIRST |
F431-03 |  | FETCH LAST |
F431-04 |  | FETCH PRIOR |
F431-05 |  | FETCH ABSOLUTE |
F431-06 |  | FETCH RELATIVE |
F441 |  | Extended set function support |
F442 |  | Mixed column references in set functions |
F471 | Core | Scalar subquery values |
F481 | Core | Expanded NULL predicate |
F491 |  | Constraint management |
F501 | Core | Features and conformance views |
F501-01 | Core | SQL_FEATURES view |
F501-02 | Core | SQL_SIZING view |
F502 |  | Enhanced documentation tables |
F531 |  | Temporary tables |
F555 |  | Enhanced seconds precision |
F561 |  | Full value expressions |
F571 |  | Truth value tests |
F591 |  | Derived tables |
F611 |  | Indicator data types |
F641 |  | Row and table constructors |
F651 |  | Catalog name qualifiers |
F661 |  | Simple tables |
F672 |  | Retrospective check constraints |
F690 |  | Collation support | but no character set support
F692 |  | Extended collation support |
F701 |  | Referential update actions |
F711 |  | ALTER domain |
F731 |  | INSERT column privileges |
F751 |  | View CHECK enhancements |
F761 |  | Session management |
F762 |  | CURRENT_CATALOG |
F763 |  | CURRENT_SCHEMA |
F771 |  | Connection management |
F781 |  | Self-referencing operations |
F791 |  | Insensitive cursors |
F801 |  | Full set function |
F850 |  | Top-level <order by clause> in <query expression> |
F851 |  | <order by clause> in subqueries |
F852 |  | Top-level <order by clause> in views |
F855 |  | Nested <order by clause> in <query expression> |
F856 |  | Nested <fetch first clause> in <query expression> |
F857 |  | Top-level <fetch first clause> in <query expression> |
F858 |  | <fetch first clause> in subqueries |
F859 |  | Top-level <fetch first clause> in views |
F860 |  | <fetch first row count> in <fetch first clause> |
F861 |  | Top-level <result offset clause> in <query expression> |
F862 |  | <result offset clause> in subqueries |
F863 |  | Nested <result offset clause> in <query expression> |
F864 |  | Top-level <result offset clause> in views |
F865 |  | <offset row count> in <result offset clause> |
F867 |  | FETCH FIRST clause: WITH TIES option |
S071 |  | SQL paths in function and type name resolution |
S092 |  | Arrays of user-defined types |
S095 |  | Array constructors by query |
S096 |  | Optional array bounds |
S098 |  | ARRAY_AGG |
S111 |  | ONLY in query expressions |
S201 |  | SQL-invoked routines on arrays |
S201-01 |  | Array parameters |
S201-02 |  | Array as result type of functions |
S211 |  | User-defined cast functions |
S301 |  | Enhanced UNNEST |
S404 |  | TRIM_ARRAY |
T031 |  | BOOLEAN data type |
T071 |  | BIGINT data type |
T121 |  | WITH (excluding RECURSIVE) in query expression |
T122 |  | WITH (excluding RECURSIVE) in subquery |
T131 |  | Recursive query |
T132 |  | Recursive query in subquery |
T133 |  | Enhanced cycle mark values | SQL:202x draft
T141 |  | SIMILAR predicate |
T151 |  | DISTINCT predicate |
T152 |  | DISTINCT predicate with negation |
T171 |  | LIKE clause in table definition |
T172 |  | AS subquery clause in table definition |
T173 |  | Extended LIKE clause in table definition |
T174 |  | Identity columns |
T177 |  | Sequence generator support: simple restart option |
T178 |  | Identity columns: simple restart option |
T191 |  | Referential action RESTRICT |
T201 |  | Comparable data types for referential constraints |
T211-01 |  | Triggers activated on UPDATE, INSERT, or DELETE of one base table |
T211-02 |  | BEFORE triggers |
T211-03 |  | AFTER triggers |
T211-04 |  | FOR EACH ROW triggers |
T211-05 |  | Ability to specify a search condition that must be true before the trigger is invoked |
T211-07 |  | TRIGGER privilege |
T212 |  | Enhanced trigger capability |
T213 |  | INSTEAD OF triggers |
T241 |  | START TRANSACTION statement |
T261 |  | Chained transactions |
T271 |  | Savepoints |
T281 |  | SELECT privilege with column granularity |
T285 |  | Enhanced derived column names |
T312 |  | OVERLAY function |
T321-01 | Core | User-defined functions with no overloading |
T321-02 | Core | User-defined stored procedures with no overloading |
T321-03 | Core | Function invocation |
T321-04 | Core | CALL statement |
T321-06 | Core | ROUTINES view |
T321-07 | Core | PARAMETERS view |
T323 |  | Explicit security for external routines |
T325 |  | Qualified SQL parameter references |
T331 |  | Basic roles |
T332 |  | Extended roles |
T341 |  | Overloading of SQL-invoked functions and procedures |
T351 |  | Bracketed SQL comments (/*...*/ comments) |
T431 |  | Extended grouping capabilities |
T432 |  | Nested and concatenated GROUPING SETS |
T433 |  | Multiargument GROUPING function |
T434 |  | GROUP BY DISTINCT |
T441 |  | ABS and MOD functions |
T461 |  | Symmetric BETWEEN predicate |
T491 |  | LATERAL derived table |
T501 |  | Enhanced EXISTS predicate |
T521 |  | Named arguments in CALL statement |
T523 |  | Default values for INOUT parameters of SQL-invoked procedures |
T524 |  | Named arguments in routine invocations other than a CALL statement |
T525 |  | Default values for parameters of SQL-invoked functions |
T551 |  | Optional key words for default syntax |
T581 |  | Regular expression substring function |
T591 |  | UNIQUE constraints of possibly null columns |
T611 |  | Elementary OLAP operations |
T612 |  | Advanced OLAP operations |
T613 |  | Sampling |
T614 |  | NTILE function |
T615 |  | LEAD and LAG functions |
T617 |  | FIRST_VALUE and LAST_VALUE function |
T620 |  | WINDOW clause: GROUPS option |
T621 |  | Enhanced numeric functions |
T622 |  | Trigonometric functions |
T623 |  | General logarithm functions |
T624 |  | Common logarithm functions |
T631 | Core | IN predicate with one list element |
T651 |  | SQL-schema statements in SQL routines |
T653 |  | SQL-schema statements in external routines |
T655 |  | Cyclically dependent routines |
T831 |  | SQL/JSON path language: strict mode |
T832 |  | SQL/JSON path language: item method |
T833 |  | SQL/JSON path language: multiple subscripts |
T834 |  | SQL/JSON path language: wildcard member accessor |
T835 |  | SQL/JSON path language: filter expressions |
T836 |  | SQL/JSON path language: starts with predicate |
T837 |  | SQL/JSON path language: regex_like predicate |
X010 |  | XML type |
X011 |  | Arrays of XML type |
X014 |  | Attributes of XML type |
X016 |  | Persistent XML values |
X020 |  | XMLConcat |
X031 |  | XMLElement |
X032 |  | XMLForest |
X034 |  | XMLAgg |
X035 |  | XMLAgg: ORDER BY option |
X036 |  | XMLComment |
X037 |  | XMLPI |
X040 |  | Basic table mapping |
X041 |  | Basic table mapping: nulls absent |
X042 |  | Basic table mapping: null as nil |
X043 |  | Basic table mapping: table as forest |
X044 |  | Basic table mapping: table as element |
X045 |  | Basic table mapping: with target namespace |
X046 |  | Basic table mapping: data mapping |
X047 |  | Basic table mapping: metadata mapping |
X048 |  | Basic table mapping: base64 encoding of binary strings |
X049 |  | Basic table mapping: hex encoding of binary strings |
X050 |  | Advanced table mapping |
X051 |  | Advanced table mapping: nulls absent |
X052 |  | Advanced table mapping: null as nil |
X053 |  | Advanced table mapping: table as forest |
X054 |  | Advanced table mapping: table as element |
X055 |  | Advanced table mapping: with target namespace |
X056 |  | Advanced table mapping: data mapping |
X057 |  | Advanced table mapping: metadata mapping |
X058 |  | Advanced table mapping: base64 encoding of binary strings |
X059 |  | Advanced table mapping: hex encoding of binary strings |
X060 |  | XMLParse: character string input and CONTENT option |
X061 |  | XMLParse: character string input and DOCUMENT option |
X070 |  | XMLSerialize: character string serialization and CONTENT option |
X071 |  | XMLSerialize: character string serialization and DOCUMENT option |
X072 |  | XMLSerialize: character string serialization |
X090 |  | XML document predicate |
X120 |  | XML parameters in SQL routines |
X121 |  | XML parameters in external routines |
X221 |  | XML passing mechanism BY VALUE |
X301 |  | XMLTable: derived column list option |
X302 |  | XMLTable: ordinality column option |
X303 |  | XMLTable: column default option |
X304 |  | XMLTable: passing a context item | must be XML DOCUMENT
X400 |  | Name and identifier mapping |
X410 |  | Alter column data type: XML type |

Prev | Up | Next
Appendix D. SQL Conformance | Home | D.2. Unsupported Features


E.1. Release 14.4
Prev Up Appendix E. Release Notes Home Next

E.1. Release 14.4

E.1.1. Migration to Version 14.4
E.1.2. Changes

Release date: 2022-06-16

This release contains a variety of fixes from 14.3. For information about new features in major release 14, see Section E.5.

E.1.1. Migration to Version 14.4

A dump/restore is not required for those running 14.X.

However, if you have any indexes that were created using the CONCURRENTLY option under 14.X, you should re-index them after updating. See the first changelog entry below.

Also, if you are upgrading from a version earlier than 14.3, see Section E.2.

E.1.2. Changes

  • Prevent possible corruption of indexes created or rebuilt with the CONCURRENTLY option (Álvaro Herrera)

    An optimization added in v14 caused CREATE INDEX ... CONCURRENTLY and REINDEX ... CONCURRENTLY to sometimes miss indexing rows that were updated during the index build. Revert that optimization. It is recommended that any indexes made with the CONCURRENTLY option be rebuilt after installing this update. (Alternatively, rebuild them without CONCURRENTLY.)

  • Harden Memoize plan node against non-deterministic equality functions (David Rowley)

    Memoize could crash if a data type's equality or hash functions gave inconsistent results across different calls. Throw a runtime error instead.

  • Fix incorrect cost estimates for Memoize plans (David Rowley)

    This mistake could lead to Memoize being used when it isn't really the best plan, or to very long executor startup times due to initializing an overly-large hash table for a Memoize node.

  • Fix queries in which a “whole-row variable” references the result of a function that returns a domain over composite type (Tom Lane)

  • Fix “variable not found in subplan target list” planner error when pulling up a sub-SELECT that's referenced in a GROUPING function (Richard Guo)

  • Prevent pg_stat_get_subscription() from possibly returning an extra row containing garbage values (Kuntal Ghosh)

  • Fix COPY FROM's error checking in the case where the database encoding is SQL_ASCII while the client's encoding is a multi-byte encoding (Heikki Linnakangas)

    This mistake could lead to false complaints of invalidly-encoded input data.

  • Avoid crashing if too many column aliases are attached to an XMLTABLE or JSON_TABLE construct (Álvaro Herrera)

  • When decompiling a view or rule, show a SELECT output column's AS "?column?" alias clause if it could be referenced elsewhere (Tom Lane)

    Previously, this auto-generated alias was always hidden; but there are corner cases where doing so results in a non-restorable view or rule definition.

  • Report implicitly-created operator families to event triggers (Masahiko Sawada)

    If CREATE OPERATOR CLASS results in the implicit creation of an operator family, that object was not reported to event triggers that should capture such events.

  • Fix control file updates made when a restartpoint is running during promotion of a standby server (Kyotaro Horiguchi)

    Previously, when the restartpoint completed it could incorrectly update the last-checkpoint fields of the control file, potentially leading to PANIC and failure to restart if the server crashes before the next normal checkpoint completes.

  • Prevent triggering of standby's wal_receiver_timeout during logical replication of large transactions (Wang Wei, Amit Kapila)

    If a large transaction on the primary server sends no data to the standby (perhaps because no table it changes is published), it was possible for the standby to timeout. Fix that by ensuring we send keepalive messages periodically in such situations.

  • Prevent open-file leak when reading an invalid timezone abbreviation file (Kyotaro Horiguchi)

    Such cases could result in harmless warning messages.

  • Allow custom server parameters to have short descriptions that are NULL (Steve Chavez)

    Previously, although extensions could choose to create such settings, some code paths would crash while processing them.

  • Remove misguided SSL key file ownership check in libpq (Tom Lane)

    In the previous minor releases, we copied the server's permission checking rules for SSL private key files into libpq. But we should not have also copied the server's file-ownership check. While that works in normal use-cases, it can result in an unexpected failure for clients running as root, and perhaps in other cases.

  • Ensure ecpg reports server connection loss sanely (Tom Lane)

    Misprocessing of a libpq-generated error result, such as a report of lost connection, would lead to printing “(null)” instead of a useful error message; or in older releases it would lead to a crash.

  • Prevent crash after server connection loss in pg_amcheck (Tom Lane)

    Misprocessing of a libpq-generated error result, such as a report of lost connection, would lead to a crash.

  • Adjust PL/Perl test case so it will work under Perl 5.36 (Dagfinn Ilmari Mannsåker)

  • Avoid incorrectly using an out-of-date libldap_r library when multiple OpenLDAP installations are present while building PostgreSQL (Tom Lane)

Prev | Up | Next
Appendix E. Release Notes | Home | E.2. Release 14.3


E.2. Release 14.3
Prev Up Appendix E. Release Notes Home Next

E.2. Release 14.3

E.2.1. Migration to Version 14.3
E.2.2. Changes

Release date: 2022-05-12

This release contains a variety of fixes from 14.2. For information about new features in major release 14, see Section E.5.

E.2.1. Migration to Version 14.3

A dump/restore is not required for those running 14.X.

However, if you have any GiST indexes on columns of type ltree (supplied by the contrib/ltree extension), you should re-index them after updating. See the second changelog entry below.

Also, if you are upgrading from a version earlier than 14.2, see Section E.3.

E.2.2. Changes

  • Confine additional operations within “security restricted operation” sandboxes (Sergey Shinderuk, Noah Misch)

    Autovacuum, CLUSTER, CREATE INDEX, REINDEX, REFRESH MATERIALIZED VIEW, and pg_amcheck activated the “security restricted operation” protection mechanism too late, or even not at all in some code paths. A user having permission to create non-temporary objects within a database could define an object that would execute arbitrary SQL code with superuser permissions the next time that autovacuum processed the object, or that some superuser ran one of the affected commands against it.

    The PostgreSQL Project thanks Alexander Lakhin for reporting this problem. (CVE-2022-1552)

  • Fix default signature length for gist_ltree_ops indexes (Tomas Vondra, Alexander Korotkov)

    The default signature length (hash size) for GiST indexes on ltree columns was accidentally changed while upgrading that operator class to support operator class parameters. If any operations had been done on such an index without first upgrading the ltree extension to version 1.2, they were done assuming that the signature length was 28 bytes rather than the intended 8. This means it is very likely that such indexes are now corrupt. For safety we recommend re-indexing all GiST indexes on ltree columns after installing this update. (Note that GiST indexes on ltree[] columns, that is arrays of ltree, are not affected.)

  • Stop using query-provided column aliases for the columns of whole-row variables that refer to plain tables (Tom Lane)

    The column names in tuples produced by a whole-row variable (such as tbl.* in contexts other than the top level of a SELECT list) are now always those of the associated named composite type, if there is one. We'd previously attempted to make them track any column aliases that had been applied to the FROM entry the variable refers to. But that's semantically dubious, because really then the output of the variable is not at all of the composite type it claims to be. Previous attempts to deal with that inconsistency had bad results up to and including storing unreadable data on disk, so just give up on the whole idea.

    In cases where it's important to be able to relabel such columns, a workaround is to introduce an extra level of sub-SELECT, so that the whole-row variable is referring to the sub-SELECT's output and not to a plain table. Then the variable is of type record to begin with and there's no issue.

  • Fix incorrect roundoff when extracting epoch values from intervals (Peter Eisentraut)

    The new numeric-based code for EXTRACT() failed to yield results equivalent to the old float-based code, as a result of accidentally truncating the DAYS_PER_YEAR value to an integer.

  • Defend against pg_stat_get_replication_slot(NULL) (Andres Freund)

    This function should be marked strict in the catalog data, but it was not in v14, so add a run-time check instead.

  • Fix incorrect output for types timestamptz and timetz in table_to_xmlschema() and allied functions (Renan Soares Lopes)

    The xmlschema output for these types included a malformed regular expression.

  • Avoid core dump in parser for a VALUES clause with zero columns (Tom Lane)

  • Fix planner failure when a Result plan node appears immediately underneath an Append node (Etsuro Fujita)

    Recently-added code to support asynchronous remote queries failed to handle this case, leading to crashes or errors about unrecognized node types.

  • Fix planner failure if a query using SEARCH or CYCLE features contains a duplicate CTE name (Tom Lane, Kyotaro Horiguchi)

    When the name of the recursive WITH query is re-used within itself, the planner could crash or report odd errors such as “could not find attribute 2 in subquery targetlist”.

  • Fix planner errors for GROUPING() constructs that reference outer query levels (Richard Guo, Tom Lane)

  • Fix plan generation for index-only scans on indexes with both returnable and non-returnable columns (Tom Lane)

    The previous coding could try to read non-returnable columns in addition to the returnable ones. This was fairly harmless because it didn't actually do anything with the bogus values, but it fell foul of a recently-added error check that rejected such a plan.

  • Avoid accessing a no-longer-pinned shared buffer while attempting to lock an outdated tuple during EvalPlanQual (Tom Lane)

    The code would touch the buffer a couple more times after releasing its pin. In theory another process could recycle the buffer (or more likely, try to defragment its free space) as soon as the pin is gone, probably leading to failure to find the newer version of the tuple.

  • Fix query-lifespan memory leak in an IndexScan node that is performing reordering (Aliaksandr Kalenik)

  • Fix ALTER FUNCTION to support changing a function's parallelism property and its SET-variable list in the same command (Tom Lane)

    The parallelism property change was lost if the same command also updated the function's SET clause.

  • Tighten lookup of the index “owned by” a constraint (Tom Lane, Japin Li)

    Some code paths mistook the index depended on by a foreign key constraint for one owned by a unique or primary key constraint, resulting in odd errors during certain ALTER TABLE operations on tables having foreign key constraints.

  • Fix bogus errors from attempts to alter system columns of tables (Tom Lane)

    The system should just tell you that you can't do it, but sometimes it would report “no owned sequence found” instead.

  • Fix mis-sorting of table rows when CLUSTERing using an index whose leading key is an expression (Peter Geoghegan, Thomas Munro)

    The table would be rebuilt with the correct data, but in an order having little to do with the index order.

  • Prevent data loss if a system crash occurs shortly after a sorted GiST index build (Heikki Linnakangas)

    The code path for building GiST indexes using sorting neglected to fsync the file upon completion. This could result in a corrupted index if the operating system crashed shortly later.

  • Fix risk of deadlock failures while dropping a partitioned index (Jimmy Yih, Gaurab Dey, Tom Lane)

    Ensure that the required table and index locks are taken in the standard order (parents before children, tables before indexes). The previous coding for DROP INDEX did it differently, and so could deadlock against concurrent queries taking these locks in the standard order.

  • Fix race condition between DROP TABLESPACE and checkpointing (Nathan Bossart)

    The checkpoint forced by DROP TABLESPACE could sometimes fail to remove all dead files from the tablespace's directory, leading to a bogus “tablespace is not empty” error.

  • Fix possible trouble in crash recovery after a TRUNCATE command that overlaps a checkpoint (Kyotaro Horiguchi, Heikki Linnakangas, Robert Haas)

    TRUNCATE must ensure that the table's disk file is truncated before the checkpoint is allowed to complete. Otherwise, replay starting from that checkpoint might find unexpected data in the supposedly-removed pages, possibly causing replay failure.

  • Fix unsafe toast-data accesses during temporary object cleanup (Andres Freund)

    Temporary-object deletion during server process exit could fail with “FATAL: cannot fetch toast data without an active snapshot”. This was usually harmless since the next use of that temporary schema would clean up successfully.

  • Re-allow underscore as the first character in a custom parameter name (Japin Li)

    Such names were unintentionally disallowed in v14.

  • Add regress option for the compute_query_id parameter (Michael Paquier)

    This is intended to facilitate testing, by allowing query IDs to be computed but not shown in EXPLAIN output.

  • Improve wait logic in RegisterSyncRequest (Thomas Munro)

    If we run out of space in the checkpointer sync request queue (which is hopefully rare on real systems, but is common when testing with a very small buffer pool), we wait for it to drain. While waiting, we should report that as a wait event so that users know what is going on, and also watch for postmaster death, since otherwise the loop might never terminate if the checkpointer has already exited.

  • Wake up for latch events when the checkpointer is waiting between writes (Thomas Munro)

    This improves responsiveness to backends sending sync requests. The change also creates a proper wait event class for these waits.

  • Fix “PANIC: xlog flush request is not satisfied” failure during standby promotion when there is a missing WAL continuation record (Sami Imseih)

  • Fix possibility of self-deadlock in hot standby conflict handling (Andres Freund)

    With unlucky timing, the WAL-applying process could get stuck while waiting for some other process to release a buffer lock.

  • Fix possible mis-identification of the correct ancestor relation to publish logical replication changes through (Tomas Vondra, Hou zj, Amit Kapila)

    If publish_via_partition_root is enabled, and there are multiple publications naming different ancestors of the currently-modified relation, the wrong ancestor might be chosen for reporting the change.

  • Ensure that logical replication apply workers can be restarted even when we're up against the max_sync_workers_per_subscription limit (Amit Kapila)

    Faulty coding of the limit check caused a restarted worker to exit immediately, leaving fewer workers than there should be.

  • Include unchanged replica identity key columns in the WAL log for an update, if they are stored out-of-line (Dilip Kumar, Amit Kapila)

    Otherwise subscribers cannot see the values and will fail to replicate the update.

  • Cope correctly with platforms that have no support for altering the server process's display in ps(1) (Andrew Dunstan)

    Few platforms are like this (the only supported one is Cygwin), so we'd managed not to notice that refactoring introduced a potential memory clobber.

  • Make the server more robust against missed timer interrupts (Michael Harris, Tom Lane)

    An optimization added in v14 meant that if a server process somehow missed a timer interrupt, it would never again ask the kernel for another one, thus breaking timeout detection for the remainder of the session. This seems unduly fragile, so add a recovery path.

  • Disallow execution of SPI functions during PL/Perl function compilation (Tom Lane)

    Perl can be convinced to execute user-defined code during compilation of a PL/Perl function. However, it's not okay for such code to try to invoke SQL operations via SPI. That results in a crash, and if it didn't crash it would be a security hazard, because we really don't want code execution during function validation. Put in a check to give a friendlier error message instead.

  • Make libpq accept root-owned SSL private key files (David Steele)

    This change synchronizes libpq's rules for safe ownership and permissions of SSL key files with the rules the server has used since release 9.6. Namely, in addition to the current rules, allow the case where the key file is owned by root and has permissions rw-r----- or less. This is helpful for system-wide management of key files.

  • Fix behavior of libpq's PQisBusy() function after a connection failure (Tom Lane)

    If we'd detected a write failure, PQisBusy() would always return true, which is the wrong thing: we want input processing to carry on normally until we've read whatever is available from the server. The practical effect of this error is that applications using libpq's async-query API would typically detect connection loss only when PQconsumeInput() returns a hard failure. With this fix, a connection loss will normally be reported via an error PGresult object, which is a much cleaner behavior for most applications.

  • Re-allow database.schema.table patterns in psql, pg_dump, and pg_amcheck (Mark Dilger)

    Versions before v14 silently ignored all but the schema and table fragments of a pattern containing more than one dot. Refactoring in v14 accidentally broke that use-case. Reinstate it, but now complain if the first fragment is not the name of the current database.

  • Make pg_ctl recheck postmaster aliveness while waiting for stop/restart/promote actions (Tom Lane)

    pg_ctl would verify that the postmaster is alive as a side-effect of sending the stop or promote signal, but then it just naively waited to see the on-disk state change. If the postmaster died uncleanly without having removed its PID file or updated the control file, pg_ctl would wait until timeout. Instead make it recheck every so often that the postmaster process is still there.

  • Fix error handling in pg_waldump (Kyotaro Horiguchi, Andres Freund)

    While trying to read a WAL file to determine the WAL segment size, pg_waldump would report an incorrect error for the case of a too-short file. In addition, the file name reported in this and related error messages could be garbage.

  • Ensure that contrib/pageinspect functions cope with all-zero pages (Michael Paquier)

    This is a legitimate edge case, but the module was mostly unprepared for it. Arrange to return nulls, or no rows, as appropriate; that seems more useful than raising an error.

  • In contrib/pageinspect, add defenses against incorrect page “special space” contents, tighten checks for correct page size, and add some missing checks that an index is of the expected type (Michael Paquier, Justin Pryzby, Julien Rouhaud)

    These changes make it less likely that the module will crash on bad data.

  • In contrib/postgres_fdw, disable batch insertion when BEFORE INSERT ... FOR EACH ROW triggers exist on the foreign table (Etsuro Fujita)

    Such a trigger might query the table it's on and expect to see previously-inserted rows. With batch insertion, those rows might not be visible yet, so disable the feature to avoid unexpected behavior.

  • In contrib/postgres_fdw, verify that ORDER BY clauses are safe to ship before requesting a remotely-ordered query, and include a USING clause if necessary (Ronan Dunklau)

    This fix prevents situations where the remote server might sort in a different order than we intend. While sometimes that would be only cosmetic, it could produce thoroughly wrong results if the remote data is used as input for a locally-performed merge join.

  • Fix configure to handle platforms that have sys/epoll.h but not sys/signalfd.h (Tom Lane)

  • Update JIT code to work with LLVM 14 (Thomas Munro)

  • Clean up assorted failures under clang's -fsanitize=undefined checks (Tom Lane, Andres Freund, Zhihong Yu)

    Most of these changes are just for pro-forma compliance with the letter of the C and POSIX standards, and are unlikely to have any effect on production builds.

  • Do not add OpenSSL dependencies to libpq's pkg-config file when building without OpenSSL (Fabrice Fontaine)

  • Fix PL/Perl so it builds on C compilers that don't support statements nested within expressions (Tom Lane)

  • Fix possible build failure of pg_dumpall on Windows, when not using MSVC to build (Andres Freund)

  • In Windows builds, use gendef instead of pexports to build DEF files (Andrew Dunstan)

    This adapts the build process to work on recent MSys tool chains.

  • Prevent extra expansion of shell wildcard patterns in programs built under MinGW (Andrew Dunstan)

    For some reason the C library provided by MinGW will expand shell wildcard characters in a program's command-line arguments by default. This is confusing, not least because it doesn't happen under MSVC, so turn it off.

  • Update time zone data files to tzdata release 2022a for DST law changes in Palestine, plus historical corrections for Chile and Ukraine.

Prev | Up | Next
E.1. Release 14.4 | Home | E.3. Release 14.2


E.3. Release 14.2
Prev Up Appendix E. Release Notes Home Next

E.3. Release 14.2

E.3.1. Migration to Version 14.2
E.3.2. Changes

Release date: 2022-02-10

This release contains a variety of fixes from 14.1. For information about new features in major release 14, see Section E.5.

E.3.1. Migration to Version 14.2

A dump/restore is not required for those running 14.X.

However, some bugs have been found that may have resulted in corrupted indexes, as explained in the first two changelog entries. If any of those cases apply to you, it's recommended to reindex possibly-affected indexes after updating.

Also, if you are upgrading from a version earlier than 14.1, see Section E.4.

E.3.2. Changes

  • Enforce standard locking protocol for TOAST table updates, to prevent problems with REINDEX CONCURRENTLY (Michael Paquier)

    If applied to a TOAST table or TOAST table's index, REINDEX CONCURRENTLY tended to produce a corrupted index. This happened because sessions updating TOAST entries released their ROW EXCLUSIVE locks immediately, rather than holding them until transaction commit as all other updates do. The fix is to make TOAST updates hold the table lock according to the normal rule. Any existing corrupted indexes can be repaired by reindexing again.

  • Fix corruption of HOT chains when a RECENTLY_DEAD tuple changes state to fully DEAD during page pruning (Andres Freund)

    It was possible for VACUUM to remove a recently-dead tuple while leaving behind a redirect item that pointed to it. When the tuple's item slot is later re-used by some new tuple, that tuple would be seen as part of the pre-existing HOT chain, creating a form of index corruption. If this has happened, reindexing the table should repair the damage. However, this is an extremely low-probability scenario, so we do not recommend reindexing just on the chance that it might have happened.

  • Fix crash in EvalPlanQual rechecks for tables with a mix of local and foreign partitions (Etsuro Fujita)

  • Fix dangling pointer in COPY TO (Bharath Rupireddy)

    This oversight could cause an incorrect error message or a crash after an error in COPY.

  • Avoid null-pointer crash in ALTER STATISTICS when the statistics object is dropped concurrently (Tomas Vondra)

  • Correctly handle alignment padding when extracting a range from a multirange (Alexander Korotkov)

    This error could cause crashes when handling multiranges over variable-length data types.

  • Fix over-optimistic use of hashing for anonymous RECORD data types (Tom Lane)

    This prevents some cases of “could not identify a hash function for type record” errors.

  • Fix incorrect plan creation for parallel single-child Append nodes (David Rowley)

    In some cases the Append would be simplified away when it should not be, leading to wrong query results (duplicated rows).

  • Fix index-only scan plans for cases where not all index columns can be returned (Tom Lane)

    If an index has both returnable and non-returnable columns, and one of the non-returnable columns is an expression using a table column that appears in a returnable index column, then a query using that expression could result in an index-only scan plan that attempts to read the non-returnable column, instead of recomputing the expression from the returnable column as intended. The non-returnable column would read as NULL, resulting in wrong query results.

  • Fix Memoize plan nodes to handle subplans that use parameters coming from above the Memoize (David Rowley)

  • Fix Memoize plan nodes to work correctly with non-hashable join operators (David Rowley)

  • Ensure that casting to an unspecified typmod generates a RelabelType node rather than a length-coercion function call (Tom Lane)

    While the coercion function should do the right thing (nothing), this translation is undesirably inefficient.

  • Fix checking of anycompatible-family data type matches (Tom Lane)

    In some cases the parser would think that a function or operator with anycompatible-family polymorphic parameters matches a set of arguments that it really shouldn't match. In reported cases, that led to matching more than one operator to a call, leading to ambiguous-operator errors; but a failure later on is also possible.

  • Fix WAL replay failure when database consistency is reached exactly at a WAL page boundary (Álvaro Herrera)

  • Fix startup of a physical replica to tolerate transaction ID wraparound (Abhijit Menon-Sen, Tomas Vondra)

    If a replica server is started while the set of active transactions on the primary crosses a wraparound boundary (so that there are some newer transactions with smaller XIDs than older ones), the replica would fail with “out-of-order XID insertion in KnownAssignedXids”. The replica would retry, but could never get past that error.

  • In logical replication, avoid double transmission of a child table's data (Hou Zhijie)

    If a publication includes both child and parent tables, and has the publish_via_partition_root option set, subscribers uselessly initiated synchronization on both child and parent tables. Ensure that only the parent table is synchronized in such cases.

  • Remove lexical limitations for SQL commands issued on a logical replication connection (Tom Lane)

    The walsender process would fail for a SQL command containing an unquoted semicolon, or with dollar-quoted literals containing odd numbers of single or double quote marks, or when the SQL command starts with a comment. Moreover, faulty error recovery could lead to unexpected errors in later commands too.

  • Ensure that replication origin timestamp is set while replicating a ROLLBACK PREPARED operation (Masahiko Sawada)

  • Fix possible loss of the commit timestamp for the last subtransaction of a transaction (Alex Kingsborough, Kyotaro Horiguchi)

  • Be sure to fsync the pg_logical/mappings subdirectory during checkpoints (Nathan Bossart)

    On some filesystems this oversight could lead to losing logical rewrite status files after a system crash.

  • Build extended statistics for partitioned tables (Justin Pryzby)

    A previous bug fix disabled building of extended statistics for old-style inheritance trees, but it also prevented building them for partitioned tables, which was an unnecessary restriction. This change allows ANALYZE to compute values for statistics objects for partitioned tables. (But note that autovacuum does not process partitioned tables as such, so you must periodically issue manual ANALYZE on the partitioned table if you want to maintain such statistics.)

  • Ignore extended statistics for inheritance trees (Justin Pryzby)

    Currently, extended statistics values are only computed locally for each table, not for entire inheritance trees. However the values were mistakenly consulted when planning queries across inheritance trees, possibly resulting in worse-than-default estimates.

  • Disallow altering data type of a partitioned table's columns when the partitioned table's row type is used as a composite type elsewhere (Tom Lane)

    This restriction has long existed for regular tables, but through an oversight it was not checked for partitioned tables.

  • Disallow ALTER TABLE ... DROP NOT NULL for a column that is part of a replica identity index (Haiying Tang, Hou Zhijie)

    The same prohibition already existed for primary key indexes.

  • Correctly update cached table state during ALTER TABLE ADD PRIMARY KEY USING INDEX (Hou Zhijie)

    Concurrent sessions failed to update their opinion of whether the table has a primary key, possibly causing incorrect logical replication behavior.

  • Correctly update cached table state when switching REPLICA IDENTITY index (Tang Haiying, Hou Zhijie)

    Concurrent sessions failed to update their opinion of which index is the replica identity one, possibly causing incorrect logical replication behavior.

  • Fix failure of SP-GiST indexes when the indexed column's data type is binary-compatible with the declared input type of the operator class (Tom Lane)

    Such cases should work, but failed with “compress method must be defined when leaf type is different from input type”.

  • Allow parallel vacuuming and concurrent index building to be ignored while computing oldest xmin (Masahiko Sawada)

    Non-parallelized instances of these operations were already ignored, but the logic did not work for parallelized cases. Holding back the xmin horizon has undesirable effects such as delaying vacuum cleanup.

  • Fix memory leak when updating expression indexes (Peter Geoghegan)

    An UPDATE affecting many rows could consume significant amounts of memory.

  • Avoid leaking memory during REASSIGN OWNED BY operations that reassign ownership of many objects (Justin Pryzby)

  • Improve performance of walsenders sending logical changes by avoiding unnecessary cache accesses (Hou Zhijie)

  • Fix display of cert authentication method's options in pg_hba_file_rules view (Magnus Hagander)

    The cert authentication method implies clientcert=verify-full, but the pg_hba_file_rules view incorrectly reported clientcert=verify-ca.

  • Ensure that the session targeted by pg_log_backend_memory_contexts() sends its results only to the server's log (Fujii Masao)

    Previously, a sufficiently high setting of client_min_messages could result in the log message also being sent to the connected client. Since that client hadn't requested it, that would be surprising (and possibly a wire protocol violation).

  • Fix display of whole-row variables appearing in INSERT ... VALUES rules (Tom Lane)

    A whole-row variable would be printed as “var.*”, but that allows it to be expanded to individual columns when the rule is reloaded, resulting in different semantics. Attach an explicit cast to prevent that, as we do elsewhere.

  • When reverse-listing a SQL-standard function body, display function parameters appropriately within INSERT ... SELECT (Tom Lane)

    Previously, they'd come out as $N even when the parameter had a name.

  • Fix one-byte buffer overrun when applying Unicode string normalization to an empty string (Michael Paquier)

    The practical impact of this is limited thanks to alignment considerations; but in debug builds, a warning was raised.

  • Fix or remove some incorrect assertions (Simon Riggs, Michael Paquier, Alexander Lakhin)

    These errors should affect only debug builds, not production.

  • Fix race condition that could lead to failure to localize error messages that are reported early in multi-threaded use of libpq or ecpglib (Tom Lane)

  • Avoid calling strerror from libpq's PQcancel function (Tom Lane)

    PQcancel is supposed to be safe to call from a signal handler, but strerror is not safe. The faulty usage only occurred in the unlikely event of failure to send the cancel message to the server, perhaps explaining the lack of reports.

  • Make psql's \password command default to setting the password for CURRENT_USER, not the connection's original user name (Tom Lane)

    This agrees with the documented behavior, and avoids probable permissions failure if SET ROLE or SET SESSION AUTHORIZATION has been done since the session began. To prevent confusion, the role name to be acted on is now included in the password prompt.

  • Fix psql \d command's query for identifying parent triggers (Justin Pryzby)

    The previous coding failed with “more than one row returned by a subquery used as an expression” if a partition had triggers and there were unrelated statement-level triggers of the same name on some parent partitioned table.

  • Make psql's \d command sort a table's extended statistics objects by name not OID (Justin Pryzby)

  • Fix psql's tab-completion of label values for enum types (Tom Lane)

  • Fix failures on Windows when using the terminal as data source or destination (Dmitry Koval, Juan José Santamaría Flecha, Michael Paquier)

    This affects psql's \copy command, as well as pg_recvlogical with -f -.

  • In psql and some other client programs, avoid trying to invoke gettext() from a control-C signal handler (Tom Lane)

    While no reported failures have been traced to this mistake, it seems highly unlikely to be a safe thing to do.

  • Allow canceling the initial password prompt in pg_receivewal and pg_recvlogical (Tom Lane, Nathan Bossart)

    Previously it was impossible to terminate these programs via control-C while they were prompting for a password.

  • Fix pg_dump's dump ordering for user-defined casts (Tom Lane)

    In rare cases, the output script might refer to a user-defined cast before it had been created.

  • Fix pg_dump's --inserts and --column-inserts modes to handle tables containing both generated columns and dropped columns (Tom Lane)

  • Fix possible mis-reporting of errors in pg_dump and pg_basebackup (Tom Lane)

    The previous code failed to check for errors from some kernel calls, and could report the wrong errno values in other cases.

  • Fix results of index-only scans on contrib/btree_gist indexes on char(N) columns (Tom Lane)

    Index-only scans returned column values with trailing spaces removed, which is not the expected behavior. That happened because that's how the data was stored in the index. This fix changes the code to store char(N) values with the expected amount of space padding. The behavior of such an index will not change immediately unless you REINDEX it; otherwise space-stripped values will be gradually replaced over time during updates. Queries that do not use index-only scan plans will be unaffected in any case.

  • Fix edge cases in postgres_fdw's handling of asynchronous queries (Etsuro Fujita)

    These errors could lead to crashes or incorrect results when attempting to parallelize scans of foreign tables.

  • Change configure to use Python's sysconfig module, rather than the deprecated distutils module, to determine how to build PL/Python (Peter Eisentraut, Tom Lane, Andres Freund)

    With Python 3.10, this avoids configure-time warnings about distutils being deprecated and scheduled for removal in Python 3.12. Presumably, once 3.12 is out, configure --with-python would fail altogether. This future-proofing does come at a cost: sysconfig did not exist before Python 2.7, nor before 3.2 in the Python 3 branch, so it is no longer possible to build PL/Python against long-dead Python versions.

  • Re-allow cross-compilation without OpenSSL (Tom Lane)

    configure should assume that /dev/urandom will be available on the target system, but it failed instead.

  • Fix PL/Perl compile failure on Windows with Perl 5.28 and later (Victor Wagner)

  • Fix PL/Python compile failure with Python 3.11 and later (Peter Eisentraut)

  • Add support for building with Visual Studio 2022 (Hans Buschmann)

  • Allow the .bat wrapper scripts in our MSVC build system to be called without first changing into their directory (Anton Voloshin, Andrew Dunstan)

Prev | Up | Next
E.2. Release 14.3 | Home | E.4. Release 14.1


E.4. Release 14.1
Prev Up Appendix E. Release Notes Home Next

E.4. Release 14.1

E.4.1. Migration to Version 14.1
E.4.2. Changes

Release date: 2021-11-11

This release contains a variety of fixes from 14.0. For information about new features in major release 14, see Section E.5.

E.4.1. Migration to Version 14.1

A dump/restore is not required for those running 14.X.

However, note that installations using physical replication should update standby servers before the primary server, as explained in the third changelog entry below.

Also, several bugs have been found that may have resulted in corrupted indexes, as explained in the next several changelog entries. If any of those cases apply to you, it's recommended to reindex possibly-affected indexes after updating.

E.4.2. Changes

  • Make the server reject extraneous data after an SSL or GSS encryption handshake (Tom Lane)

    A man-in-the-middle with the ability to inject data into the TCP connection could stuff some cleartext data into the start of a supposedly encryption-protected database session. This could be abused to send faked SQL commands to the server, although that would only work if the server did not demand any authentication data. (However, a server relying on SSL certificate authentication might well not do so.)

    The PostgreSQL Project thanks Jacob Champion for reporting this problem. (CVE-2021-23214)

  • Make libpq reject extraneous data after an SSL or GSS encryption handshake (Tom Lane)

    A man-in-the-middle with the ability to inject data into the TCP connection could stuff some cleartext data into the start of a supposedly encryption-protected database session. This could probably be abused to inject faked responses to the client's first few queries, although other details of libpq's behavior make that harder than it sounds. A different line of attack is to exfiltrate the client's password, or other sensitive data that might be sent early in the session. That has been shown to be possible with a server vulnerable to CVE-2021-23214.

    The PostgreSQL Project thanks Jacob Champion for reporting this problem. (CVE-2021-23222)

  • Fix physical replication for cases where the primary crashes after shipping a WAL segment that ends with a partial WAL record (Álvaro Herrera)

    If the primary did not survive long enough to finish writing the rest of the incomplete WAL record, then the previous crash-recovery logic had it back up and overwrite WAL starting from the beginning of the incomplete WAL record. This is problematic since standby servers may already have copies of that WAL segment. They will then see an inconsistent next segment, and will not be able to recover without manual intervention. To fix, do not back up over a WAL segment boundary when restarting after a crash. Instead write a new type of WAL record at the start of the next WAL segment, informing readers that the incomplete WAL record will never be finished and must be disregarded.

    When applying this update, it's best to update standby servers before the primary, so that they will be ready to handle this new WAL record type if the primary happens to crash.

  • Ensure that parallel VACUUM doesn't miss any indexes (Peter Geoghegan, Masahiko Sawada)

    A parallel VACUUM would fail to process indexes that are below the min_parallel_index_scan_size cutoff, if the table also has at least two indexes that are above that size. This could result in those indexes becoming corrupt, since they'd still contain references to any heap entries removed by the VACUUM; subsequent queries using such indexes would be likely to return rows they shouldn't. This problem does not affect autovacuum, since it doesn't use parallel vacuuming. However, it is advisable to reindex any manually-vacuumed tables that have the right mix of index sizes.

  • Fix CREATE INDEX CONCURRENTLY to wait for the latest prepared transactions (Andrey Borodin)

    Rows inserted by just-prepared transactions might be omitted from the new index, causing queries relying on the index to miss such rows. The previous fix for this type of problem failed to account for PREPARE TRANSACTION commands that were still in progress when CREATE INDEX CONCURRENTLY checked for them. As before, in installations that have enabled prepared transactions (max_prepared_transactions > 0), it's recommended to reindex any concurrently-built indexes in case this problem occurred when they were built.

  • Avoid race condition that can cause backends to fail to add entries for new rows to an index being built concurrently (Noah Misch, Andrey Borodin)

    While it's apparently rare in the field, this case could potentially affect any index built or reindexed with the CONCURRENTLY option. It is recommended to reindex any such indexes to make sure they are correct.

  • Fix REINDEX CONCURRENTLY to preserve operator class parameters that were attached to the target index (Michael Paquier)

  • Fix incorrect creation of shared dependencies when cloning a database that contains non-builtin objects (Aleksander Alekseev)

    The effects of this error are probably limited in practice. In principle, it could allow a role to be dropped while it still owns objects; but most installations would never want to drop a role that had been used for objects they'd added to template1.

  • Ensure that the relation cache is invalidated for a table being attached to or detached from a partitioned table (Amit Langote, Álvaro Herrera)

    This oversight could allow misbehavior of subsequent inserts/updates addressed directly to the partition, but only in currently-existing sessions.

  • Fix corruption of parse tree while creating a range type (Alex Kozhemyakin, Sergey Shinderuk)

    CREATE TYPE incorrectly freed an element of the parse tree, which could cause problems for a later event trigger, or if the CREATE TYPE command was stored in the plan cache and used again later.

  • Fix updates of element fields in arrays of domain over composite (Tom Lane)

    A command such as UPDATE tab SET fld[1].subfld = val failed if the array's elements were domains rather than plain composites.

  • Disallow the combination of FETCH FIRST WITH TIES and FOR UPDATE SKIP LOCKED (David Christensen)

    FETCH FIRST WITH TIES necessarily fetches one more row than requested, since it cannot stop until it finds a row that is not a tie. In our current implementation, if FOR UPDATE is used then that row will also get locked even though it is not returned. That results in undesirable behavior if the SKIP LOCKED option is specified. It's difficult to change this without introducing a different set of undesirable behaviors, so for now, forbid the combination.

  • Disallow ALTER INDEX index ALTER COLUMN col SET (options) (Nathan Bossart, Michael Paquier)

    While the parser accepted this, it's undocumented and doesn't actually work.

  • Fix corner-case loss of precision in numeric power() (Dean Rasheed)

    The result could be inaccurate when the first argument is very close to 1.

  • Avoid choosing the wrong hash equality operator for Memoize plans (David Rowley)

    This error could result in crashes or incorrect query results.

  • Fix planner error with pulling up subquery expressions into function rangetable entries (Tom Lane)

    If a function in FROM laterally references the output of some sub-SELECT earlier in the FROM clause, and we are able to flatten that sub-SELECT into the outer query, the expression(s) copied into the function expression were not fully processed. This could lead to crashes at execution.

  • Avoid using MCV-only statistics to estimate the range of a column (Tom Lane)

    There are corner cases in which ANALYZE will build a most-common-values (MCV) list but not a histogram, even though the MCV list does not account for all the observed values. In such cases, keep the planner from using the MCV list alone to estimate the range of column values.

  • Fix restoration of a Portal's snapshot inside a subtransaction (Bertrand Drouvot)

    If a procedure commits or rolls back a transaction, and then its next significant action is inside a new subtransaction, snapshot management went wrong, leading to a dangling pointer and probable crash. A typical example in PL/pgSQL is a COMMIT immediately followed by a BEGIN ... EXCEPTION block that performs a query.

  • Clean up correctly if a transaction fails after exporting its snapshot (Dilip Kumar)

    This oversight would only cause a problem if the same session attempted to export a snapshot again. The most likely scenario for that is creation of a replication slot (followed by rollback) and then creation of another replication slot.

  • Prevent wraparound of overflowed-subtransaction tracking on standby servers (Kyotaro Horiguchi, Alexander Korotkov)

    This oversight could cause significant performance degradation (manifesting as excessive SubtransSLRU traffic) on standby servers.

  • Ensure that prepared transactions are properly accounted for during promotion of a standby server (Michael Paquier, Andres Freund)

    There was a narrow window where a prepared transaction could be omitted from a snapshot taken by a concurrently-running session. If that session then used the snapshot to perform data updates, erroneous results or data corruption could occur.

  • Fix “could not find RecursiveUnion” error when EXPLAIN tries to print a filter condition attached to a WorkTableScan node (Tom Lane)

  • Ensure that the correct lock level is used when renaming a table (Nathan Bossart, Álvaro Herrera)

    For historical reasons, ALTER INDEX ... RENAME can be applied to any sort of relation. The lock level required to rename an index is lower than that required to rename a table or other kind of relation, but the code got this wrong and would use the weaker lock level whenever the command is spelled ALTER INDEX.

  • Avoid null-pointer-dereference crash when dropping a role that owns objects being dropped concurrently (Álvaro Herrera)

  • Prevent “snapshot reference leak” warning when lo_export() or a related function fails (Heikki Linnakangas)

  • Fix inefficient code generation for CoerceToDomain expression nodes (Ranier Vilela)

  • Avoid O(N^2) behavior in some list-manipulation operations (Nathan Bossart, Tom Lane)

    These changes fix slow processing in several scenarios, including: when a standby replays a transaction that held many exclusive locks on the primary; when many files are due to be unlinked after a checkpoint; when hash aggregation involves many batches; and when pg_trgm extracts indexable conditions from a complex regular expression. Only the first of these scenarios has actually been reported from the field, but they all seem like plausible consequences of inefficient list deletions.

  • Add more defensive checks around B-tree posting list splits (Peter Geoghegan)

    This change should help detect index corruption involving duplicate table TIDs.

  • Avoid assertion failure when inserting NaN into a BRIN float8 or float4 minmax_multi_ops index (Tomas Vondra)

    In production builds, such cases would result in a somewhat inefficient, but not actually incorrect, index.

  • Allow the autovacuum launcher process to respond to pg_log_backend_memory_contexts() requests more quickly (Koyu Tanigawa)

  • Fix memory leak in HMAC hash calculations (Sergey Shinderuk)

  • Disallow setting huge_pages to on when shared_memory_type is sysv (Thomas Munro)

    Previously, this setting was accepted, but it did nothing for lack of any implementation.

  • Fix checking of query type in PL/pgSQL's RETURN QUERY statement (Tom Lane)

    RETURN QUERY should accept any query that can return tuples, e.g. UPDATE RETURNING. v14 accidentally disallowed anything but SELECT; moreover, the RETURN QUERY EXECUTE variant failed to apply any query-type check at all.

  • Fix pg_dump to dump non-global default privileges correctly (Neil Chen, Masahiko Sawada)

    If a global (unrestricted) ALTER DEFAULT PRIVILEGES command revoked some present-by-default privilege, for example EXECUTE for functions, and then a restricted ALTER DEFAULT PRIVILEGES command granted that privilege again for a selected role or schema, pg_dump failed to dump the restricted privilege grant correctly.

  • Make pg_dump acquire shared lock on partitioned tables that are to be dumped (Tom Lane)

    This oversight was usually pretty harmless, since once pg_dump has locked any of the leaf partitions, that would suffice to prevent significant DDL on the partitioned table itself. However problems could ensue when dumping a childless partitioned table, since no relevant lock would be held.

  • Fix crash in pg_dump when attempting to dump trigger definitions from a pre-8.3 server (Tom Lane)

  • Fix incorrect filename in pg_restore's error message about an invalid large object TOC file (Daniel Gustafsson)

  • Ensure that pgbench exits with non-zero status after a socket-level failure (Yugo Nagata, Fabien Coelho)

    The desired behavior is to finish out the run but then exit with status 2. Also, fix the reporting of such errors.

  • Prevent pg_amcheck from checking temporary relations, as well as indexes that are invalid or not ready (Mark Dilger)

    This avoids unhelpful checks of relations that will almost certainly appear inconsistent.

  • Make contrib/amcheck skip unlogged tables when running on a standby server (Mark Dilger)

    It's appropriate to do this since such tables will be empty, and unlogged indexes were already handled similarly.

  • Change contrib/pg_stat_statements to read its “query texts” file in units of at most 1GB (Tom Lane)

    Such large query text files are very unusual, but if they do occur, the previous coding would fail on Windows 64 (which rejects individual read requests of more than 2GB).

  • Fix null-pointer crash when contrib/postgres_fdw tries to report a data conversion error (Tom Lane)

  • Ensure that GetSharedSecurityLabel() can be used in a newly-started session that has not yet built its critical relation cache entries (Jeff Davis)

  • When running a TAP test, include the module's own directory in PATH (Andrew Dunstan)

    This allows tests to find built programs that are not installed, such as custom test drivers.

  • Use the CLDR project's data to map Windows time zone names to IANA time zones (Tom Lane)

    When running on Windows, initdb attempts to set the new cluster's timezone parameter to the IANA time zone matching the system's prevailing time zone. We were using a mapping table that we'd generated years ago and updated only fitfully; unsurprisingly, it contained a number of errors as well as omissions of recently-added zones. It turns out that CLDR has been tracking the most appropriate mappings, so start using their data. This change will not affect any existing installation, only newly-initialized clusters.

  • Update time zone data files to tzdata release 2021e for DST law changes in Fiji, Jordan, Palestine, and Samoa, plus historical corrections for Barbados, Cook Islands, Guyana, Niue, Portugal, and Tonga.

    Also, the Pacific/Enderbury zone has been renamed to Pacific/Kanton. Also, the following zones have been merged into nearby, more-populous zones whose clocks have agreed with them since 1970: Africa/Accra, America/Atikokan, America/Blanc-Sablon, America/Creston, America/Curacao, America/Nassau, America/Port_of_Spain, Antarctica/DumontDUrville, and Antarctica/Syowa. In all these cases, the previous zone name remains as an alias.

Prev | Up | Next
E.3. Release 14.2 | Home | E.5. Release 14


Appendix F. Additional Supplied Modules
Prev Up Part VIII. Appendixes Home Next

Appendix F. Additional Supplied Modules

Table of Contents

F.1. adminpack
F.2. amcheck
F.2.1. Functions
F.2.2. Optional heapallindexed Verification
F.2.3. Using amcheck Effectively
F.2.4. Repairing Corruption
F.3. auth_delay
F.3.1. Configuration Parameters
F.3.2. Author
F.4. auto_explain
F.4.1. Configuration Parameters
F.4.2. Example
F.4.3. Author
F.5. bloom
F.5.1. Parameters
F.5.2. Examples
F.5.3. Operator Class Interface
F.5.4. Limitations
F.5.5. Authors
F.6. btree_gin
F.6.1. Example Usage
F.6.2. Authors
F.7. btree_gist
F.7.1. Example Usage
F.7.2. Authors
F.8. citext
F.8.1. Rationale
F.8.2. How to Use It
F.8.3. String Comparison Behavior
F.8.4. Limitations
F.8.5. Author
F.9. cube
F.9.1. Syntax
F.9.2. Precision
F.9.3. Usage
F.9.4. Defaults
F.9.5. Notes
F.9.6. Credits
F.10. dblink
dblink_connect — opens a persistent connection to a remote database
dblink_connect_u — opens a persistent connection to a remote database, insecurely
dblink_disconnect — closes a persistent connection to a remote database
dblink — executes a query in a remote database
dblink_exec — executes a command in a remote database
dblink_open — opens a cursor in a remote database
dblink_fetch — returns rows from an open cursor in a remote database
dblink_close — closes a cursor in a remote database
dblink_get_connections — returns the names of all open named dblink connections
dblink_error_message — gets last error message on the named connection
dblink_send_query — sends an async query to a remote database
dblink_is_busy — checks if connection is busy with an async query
dblink_get_notify — retrieve async notifications on a connection
dblink_get_result — gets an async query result
dblink_cancel_query — cancels any active query on the named connection
dblink_get_pkey — returns the positions and field names of a relation's primary key fields
dblink_build_sql_insert — builds an INSERT statement using a local tuple, replacing the primary key field values with alternative supplied values
dblink_build_sql_delete — builds a DELETE statement using supplied values for primary key field values
dblink_build_sql_update — builds an UPDATE statement using a local tuple, replacing the primary key field values with alternative supplied values
F.11. dict_int
F.11.1. Configuration
F.11.2. Usage
F.12. dict_xsyn
F.12.1. Configuration
F.12.2. Usage
F.13. earthdistance
F.13.1. Cube-Based Earth Distances
F.13.2. Point-Based Earth Distances
F.14. file_fdw
F.15. fuzzystrmatch
F.15.1. Soundex
F.15.2. Levenshtein
F.15.3. Metaphone
F.15.4. Double Metaphone
F.16. hstore
F.16.1. hstore External Representation
F.16.2. hstore Operators and Functions
F.16.3. Indexes
F.16.4. Examples
F.16.5. Statistics
F.16.6. Compatibility
F.16.7. Transforms
F.16.8. Authors
F.17. intagg
F.17.1. Functions
F.17.2. Sample Uses
F.18. intarray
F.18.1. intarray Functions and Operators
F.18.2. Index Support
F.18.3. Example
F.18.4. Benchmark
F.18.5. Authors
F.19. isn
F.19.1. Data Types
F.19.2. Casts
F.19.3. Functions and Operators
F.19.4. Examples
F.19.5. Bibliography
F.19.6. Author
F.20. lo
F.20.1. Rationale
F.20.2. How to Use It
F.20.3. Limitations
F.20.4. Author
F.21. ltree
F.21.1. Definitions
F.21.2. Operators and Functions
F.21.3. Indexes
F.21.4. Example
F.21.5. Transforms
F.21.6. Authors
F.22. old_snapshot
F.22.1. Functions
F.23. pageinspect
F.23.1. General Functions
F.23.2. Heap Functions
F.23.3. B-Tree Functions
F.23.4. BRIN Functions
F.23.5. GIN Functions
F.23.6. GiST Functions
F.23.7. Hash Functions
F.24. passwordcheck
F.25. pg_buffercache
F.25.1. The pg_buffercache View
F.25.2. Sample Output
F.25.3. Authors
F.26. pgcrypto
F.26.1. General Hashing Functions
F.26.2. Password Hashing Functions
F.26.3. PGP Encryption Functions
F.26.4. Raw Encryption Functions
F.26.5. Random-Data Functions
F.26.6. Notes
F.26.7. Author
F.27. pg_freespacemap
F.27.1. Functions
F.27.2. Sample Output
F.27.3. Author
F.28. pg_prewarm
F.28.1. Functions
F.28.2. Configuration Parameters
F.28.3. Author
F.29. pgrowlocks
F.29.1. Overview
F.29.2. Sample Output
F.29.3. Author
F.30. pg_stat_statements
F.30.1. The pg_stat_statements View
F.30.2. The pg_stat_statements_info View
F.30.3. Functions
F.30.4. Configuration Parameters
F.30.5. Sample Output
F.30.6. Authors
F.31. pgstattuple
F.31.1. Functions
F.31.2. Authors
F.32. pg_surgery
F.32.1. Functions
F.32.2. Authors
F.33. pg_trgm
F.33.1. Trigram (or Trigraph) Concepts
F.33.2. Functions and Operators
F.33.3. GUC Parameters
F.33.4. Index Support
F.33.5. Text Search Integration
F.33.6. References
F.33.7. Authors
F.34. pg_visibility
F.34.1. Functions
F.34.2. Author
F.35. postgres_fdw
F.35.1. FDW Options of postgres_fdw
F.35.2. Functions
F.35.3. Connection Management
F.35.4. Transaction Management
F.35.5. Remote Query Optimization
F.35.6. Remote Query Execution Environment
F.35.7. Cross-Version Compatibility
F.35.8. Examples
F.35.9. Author
F.36. seg
F.36.1. Rationale
F.36.2. Syntax
F.36.3. Precision
F.36.4. Usage
F.36.5. Notes
F.36.6. Credits
F.37. sepgsql
F.37.1. Overview
F.37.2. Installation
F.37.3. Regression Tests
F.37.4. GUC Parameters
F.37.5. Features
F.37.6. Sepgsql Functions
F.37.7. Limitations
F.37.8. External Resources
F.37.9. Author
F.38. spi
F.38.1. refint — Functions for Implementing Referential Integrity
F.38.2. autoinc — Functions for Autoincrementing Fields
F.38.3. insert_username — Functions for Tracking Who Changed a Table
F.38.4. moddatetime — Functions for Tracking Last Modification Time
F.39. sslinfo
F.39.1. Functions Provided
F.39.2. Author
F.40. tablefunc
F.40.1. Functions Provided
F.40.2. Author
F.41. tcn
F.42. test_decoding
F.43. tsm_system_rows
F.43.1. Examples
F.44. tsm_system_time
F.44.1. Examples
F.45. unaccent
F.45.1. Configuration
F.45.2. Usage
F.45.3. Functions
F.46. uuid-ossp
F.46.1. uuid-ossp Functions
F.46.2. Building uuid-ossp
F.46.3. Author
F.47. xml2
F.47.1. Deprecation Notice
F.47.2. Description of Functions
F.47.3. xpath_table
F.47.4. XSLT Functions
F.47.5. Author

This appendix and the next one contain information regarding the modules that can be found in the contrib directory of the PostgreSQL distribution. These include porting tools, analysis utilities, and plug-in features that are not part of the core PostgreSQL system, mainly because they address a limited audience or are too experimental to be part of the main source tree. This does not preclude their usefulness.

This appendix covers extensions and other server plug-in modules found in contrib. Appendix G covers utility programs.

When building from the source distribution, these components are not built automatically, unless you build the "world" target (see Step 2). You can build and install all of them by running:

make
make install

in the contrib directory of a configured source tree; or to build and install just one selected module, do the same in that module's subdirectory. Many of the modules have regression tests, which can be executed by running:

make check

before installation or

make installcheck

once you have a PostgreSQL server running.

If you are using a pre-packaged version of PostgreSQL, these modules are typically made available as a separate subpackage, such as postgresql-contrib.

Many modules supply new user-defined functions, operators, or types. To make use of one of these modules, after you have installed the code you need to register the new SQL objects in the database system. This is done by executing a CREATE EXTENSION command. In a fresh database, you can simply do

CREATE EXTENSION module_name;

This command registers the new SQL objects in the current database only, so you need to run it in each database that you want the module's facilities to be available in. Alternatively, run it in database template1 so that the extension will be copied into subsequently-created databases by default.

For all these modules, CREATE EXTENSION must be run by a database superuser, unless the module is considered “trusted”, in which case it can be run by any user who has CREATE privilege on the current database. Modules that are trusted are identified as such in the sections that follow. Generally, trusted modules are ones that cannot provide access to outside-the-database functionality.

Many modules allow you to install their objects in a schema of your choice. To do that, add SCHEMA schema_name to the CREATE EXTENSION command. By default, the objects will be placed in your current creation target schema, which in turn defaults to public.

Note, however, that some of these modules are not “extensions” in this sense, but are loaded into the server in some other way, for instance by way of shared_preload_libraries. See the documentation of each module for details.

Prev | Up | Next
E.6. Prior Releases | Home | F.1. adminpack


F.1. adminpack
Prev Up Appendix F. Additional Supplied Modules Home Next

F.1. adminpack

adminpack provides a number of support functions which pgAdmin and other administration and management tools can use to provide additional functionality, such as remote management of server log files. Use of all these functions is only allowed to the superuser by default but may be allowed to other users by using the GRANT command.

The functions shown in Table F.1 provide write access to files on the machine hosting the server. (See also the functions in Table 9.97, which provide read-only access.) Only files within the database cluster directory can be accessed, unless the user is a superuser or given one of the pg_read_server_files, or pg_write_server_files roles, as appropriate for the function, but either a relative or absolute path is allowable.

Table F.1. adminpack Functions

Function

Description

pg_catalog.pg_file_write ( filename text, data text, append boolean ) → bigint

Writes, or appends to, a text file.

pg_catalog.pg_file_sync ( filename text ) → void

Flushes a file or directory to disk.

pg_catalog.pg_file_rename ( oldname text, newname text [, archivename text ] ) → boolean

Renames a file.

pg_catalog.pg_file_unlink ( filename text ) → boolean

Removes a file.

pg_catalog.pg_logdir_ls () → setof record

Lists the log files in the log_directory directory.

pg_file_write writes the specified data into the file named by filename. If append is false, the file must not already exist. If append is true, the file can already exist, and will be appended to if so. Returns the number of bytes written.

pg_file_sync fsyncs the specified file or directory named by filename. An error is thrown on failure (e.g., the specified file is not present). Note that data_sync_retry has no effect on this function, and therefore a PANIC-level error will not be raised even on failure to flush database files.

pg_file_rename renames a file. If archivename is omitted or NULL, it simply renames oldname to newname (which must not already exist). If archivename is provided, it first renames newname to archivename (which must not already exist), and then renames oldname to newname. In event of failure of the second rename step, it will try to rename archivename back to newname before reporting the error. Returns true on success, false if the source file(s) are not present or not writable; other cases throw errors.

pg_file_unlink removes the specified file. Returns true on success, false if the specified file is not present or the unlink() call fails; other cases throw errors.

pg_logdir_ls returns the start timestamps and path names of all the log files in the log_directory directory. The log_filename parameter must have its default setting (postgresql-%Y-%m-%d_%H%M%S.log) to use this function.

Prev | Up | Next
Appendix F. Additional Supplied Modules | Home | F.2. amcheck


F.2. amcheck
Prev Up Appendix F. Additional Supplied Modules Home Next

F.2. amcheck

F.2.1. Functions
F.2.2. Optional heapallindexed Verification
F.2.3. Using amcheck Effectively
F.2.4. Repairing Corruption

The amcheck module provides functions that allow you to verify the logical consistency of the structure of relations.

The B-Tree checking functions verify various invariants in the structure of the representation of particular relations. The correctness of the access method functions behind index scans and other important operations relies on these invariants always holding. For example, certain functions verify, among other things, that all B-Tree pages have items in “logical” order (e.g., for B-Tree indexes on text, index tuples should be in collated lexical order). If that particular invariant somehow fails to hold, we can expect binary searches on the affected page to incorrectly guide index scans, resulting in wrong answers to SQL queries. If the structure appears to be valid, no error is raised.

Verification is performed using the same procedures as those used by index scans themselves, which may be user-defined operator class code. For example, B-Tree index verification relies on comparisons made with one or more B-Tree support function 1 routines. See Section 38.16.3 for details of operator class support functions.

Unlike the B-Tree checking functions which report corruption by raising errors, the heap checking function verify_heapam checks a table and attempts to return a set of rows, one row per corruption detected. Despite this, if facilities that verify_heapam relies upon are themselves corrupted, the function may be unable to continue and may instead raise an error.

Permission to execute amcheck functions may be granted to non-superusers, but before granting such permissions careful consideration should be given to data security and privacy concerns. Although the corruption reports generated by these functions do not focus on the contents of the corrupted data so much as on the structure of that data and the nature of the corruptions found, an attacker who gains permission to execute these functions, particularly if the attacker can also induce corruption, might be able to infer something of the data itself from such messages.

F.2.1. Functions

bt_index_check(index regclass, heapallindexed boolean) returns void

bt_index_check tests that its target, a B-Tree index, respects a variety of invariants. Example usage:

test=# SELECT bt_index_check(index => c.oid, heapallindexed => i.indisunique),
               c.relname,
               c.relpages
FROM pg_index i
JOIN pg_opclass op ON i.indclass[0] = op.oid
JOIN pg_am am ON op.opcmethod = am.oid
JOIN pg_class c ON i.indexrelid = c.oid
JOIN pg_namespace n ON c.relnamespace = n.oid
WHERE am.amname = 'btree' AND n.nspname = 'pg_catalog'
-- Don't check temp tables, which may be from another session:
AND c.relpersistence != 't'
-- Function may throw an error when this is omitted:
AND c.relkind = 'i' AND i.indisready AND i.indisvalid
ORDER BY c.relpages DESC LIMIT 10;
 bt_index_check |             relname             | relpages
----------------+---------------------------------+----------
                | pg_depend_reference_index       |       43
                | pg_depend_depender_index        |       40
                | pg_proc_proname_args_nsp_index  |       31
                | pg_description_o_c_o_index      |       21
                | pg_attribute_relid_attnam_index |       14
                | pg_proc_oid_index               |       10
                | pg_attribute_relid_attnum_index |        9
                | pg_amproc_fam_proc_index        |        5
                | pg_amop_opr_fam_index           |        5
                | pg_amop_fam_strat_index         |        5
(10 rows)

This example shows a session that performs verification of the 10 largest catalog indexes in the database “test”. Verification of the presence of heap tuples as index tuples is requested for the subset that are unique indexes. Since no error is raised, all indexes tested appear to be logically consistent. Naturally, this query could easily be changed to call bt_index_check for every index in the database where verification is supported.

bt_index_check acquires an AccessShareLock on the target index and the heap relation it belongs to. This lock mode is the same lock mode acquired on relations by simple SELECT statements. bt_index_check does not verify invariants that span child/parent relationships, but will verify the presence of all heap tuples as index tuples within the index when heapallindexed is true. When a routine, lightweight test for corruption is required in a live production environment, using bt_index_check often provides the best trade-off between thoroughness of verification and limiting the impact on application performance and availability.

bt_index_parent_check(index regclass, heapallindexed boolean, rootdescend boolean) returns void

bt_index_parent_check tests that its target, a B-Tree index, respects a variety of invariants. Optionally, when the heapallindexed argument is true, the function verifies the presence of all heap tuples that should be found within the index. When the optional rootdescend argument is true, verification re-finds tuples on the leaf level by performing a new search from the root page for each tuple. The checks that can be performed by bt_index_parent_check are a superset of the checks that can be performed by bt_index_check. bt_index_parent_check can be thought of as a more thorough variant of bt_index_check: unlike bt_index_check, bt_index_parent_check also checks invariants that span parent/child relationships, including checking that there are no missing downlinks in the index structure. bt_index_parent_check follows the general convention of raising an error if it finds a logical inconsistency or other problem.

A ShareLock is required on the target index by bt_index_parent_check (a ShareLock is also acquired on the heap relation). These locks prevent concurrent data modification from INSERT, UPDATE, and DELETE commands. The locks also prevent the underlying relation from being concurrently processed by VACUUM, as well as all other utility commands. Note that the function holds locks only while running, not for the entire transaction.

bt_index_parent_check's additional verification is more likely to detect various pathological cases. These cases may involve an incorrectly implemented B-Tree operator class used by the index that is checked, or, hypothetically, undiscovered bugs in the underlying B-Tree index access method code. Note that bt_index_parent_check cannot be used when Hot Standby mode is enabled (i.e., on read-only physical replicas), unlike bt_index_check.

Tip

bt_index_check and bt_index_parent_check both output log messages about the verification process at DEBUG1 and DEBUG2 severity levels. These messages provide detailed information about the verification process that may be of interest to PostgreSQL developers. Advanced users may also find this information helpful, since it provides additional context should verification actually detect an inconsistency. Running:

SET client_min_messages = DEBUG1;

in an interactive psql session before running a verification query will display messages about the progress of verification with a manageable level of detail.

verify_heapam(relation regclass, on_error_stop boolean, check_toast boolean, skip text, startblock bigint, endblock bigint, blkno OUT bigint, offnum OUT integer, attnum OUT integer, msg OUT text) returns setof record

Checks a table for structural corruption, where pages in the relation contain data that is invalidly formatted, and for logical corruption, where pages are structurally valid but inconsistent with the rest of the database cluster.

The following optional arguments are recognized:

on_error_stop

If true, corruption checking stops at the end of the first block in which any corruptions are found.

Defaults to false.

check_toast

If true, toasted values are checked against the target relation's TOAST table.

This option is known to be slow. Also, if the toast table or its index is corrupt, checking it against toast values could conceivably crash the server, although in many cases this would just produce an error.

Defaults to false.

skip

If not none, corruption checking skips blocks that are marked as all-visible or all-frozen, as specified. Valid options are all-visible, all-frozen and none.

Defaults to none.

startblock

If specified, corruption checking begins at the specified block, skipping all previous blocks. It is an error to specify a startblock outside the range of blocks in the target table.

By default, checking begins at the first block.

endblock

If specified, corruption checking ends at the specified block, skipping all remaining blocks. It is an error to specify an endblock outside the range of blocks in the target table.

By default, all blocks are checked.

For each corruption detected, verify_heapam returns a row with the following columns:

blkno

The number of the block containing the corrupt page.

offnum

The OffsetNumber of the corrupt tuple.

attnum

The attribute number of the corrupt column in the tuple, if the corruption is specific to a column and not the tuple as a whole.

msg

A message describing the problem detected.

F.2.2. Optional heapallindexed Verification

When the heapallindexed argument to B-Tree verification functions is true, an additional phase of verification is performed against the table associated with the target index relation. This consists of a “dummy” CREATE INDEX operation, which checks for the presence of all hypothetical new index tuples against a temporary, in-memory summarizing structure (this is built when needed during the basic first phase of verification). The summarizing structure “fingerprints” every tuple found within the target index. The high level principle behind heapallindexed verification is that a new index that is equivalent to the existing, target index must only have entries that can be found in the existing structure.

The additional heapallindexed phase adds significant overhead: verification will typically take several times longer. However, there is no change to the relation-level locks acquired when heapallindexed verification is performed.

The summarizing structure is bound in size by maintenance_work_mem. In order to ensure that there is no more than a 2% probability of failure to detect an inconsistency for each heap tuple that should be represented in the index, approximately 2 bytes of memory are needed per tuple. As less memory is made available per tuple, the probability of missing an inconsistency slowly increases. This approach limits the overhead of verification significantly, while only slightly reducing the probability of detecting a problem, especially for installations where verification is treated as a routine maintenance task. Any single absent or malformed tuple has a new opportunity to be detected with each new verification attempt.

F.2.3. Using amcheck Effectively

amcheck can be effective at detecting various types of failure modes that data checksums will fail to catch. These include:

  • Structural inconsistencies caused by incorrect operator class implementations.

    This includes issues caused by the comparison rules of operating system collations changing. Comparisons of datums of a collatable type like text must be immutable (just as all comparisons used for B-Tree index scans must be immutable), which implies that operating system collation rules must never change. Though rare, updates to operating system collation rules can cause these issues. More commonly, an inconsistency in the collation order between a primary server and a standby server is implicated, possibly because the major operating system version in use is inconsistent. Such inconsistencies will generally only arise on standby servers, and so can generally only be detected on standby servers.

    If a problem like this arises, it may not affect each individual index that is ordered using an affected collation, simply because indexed values might happen to have the same absolute ordering regardless of the behavioral inconsistency. See Section 24.1 and Section 24.2 for further details about how PostgreSQL uses operating system locales and collations.

  • Structural inconsistencies between indexes and the heap relations that are indexed (when heapallindexed verification is performed).

    There is no cross-checking of indexes against their heap relation during normal operation. Symptoms of heap corruption can be subtle.

  • Corruption caused by hypothetical undiscovered bugs in the underlying PostgreSQL access method code, sort code, or transaction management code.

    Automatic verification of the structural integrity of indexes plays a role in the general testing of new or proposed PostgreSQL features that could plausibly allow a logical inconsistency to be introduced. Verification of table structure and associated visibility and transaction status information plays a similar role. One obvious testing strategy is to call amcheck functions continuously when running the standard regression tests. See Section 33.1 for details on running the tests.

  • File system or storage subsystem faults where checksums happen to simply not be enabled.

    Note that amcheck examines a page as represented in some shared memory buffer at the time of verification if there is only a shared buffer hit when accessing the block. Consequently, amcheck does not necessarily examine data read from the file system at the time of verification. Note that when checksums are enabled, amcheck may raise an error due to a checksum failure when a corrupt block is read into a buffer.

  • Corruption caused by faulty RAM, or the broader memory subsystem.

    PostgreSQL does not protect against correctable memory errors and it is assumed you will operate using RAM that uses industry standard Error Correcting Codes (ECC) or better protection. However, ECC memory is typically only immune to single-bit errors, and should not be assumed to provide absolute protection against failures that result in memory corruption.

    When heapallindexed verification is performed, there is generally a greatly increased chance of detecting single-bit errors, since strict binary equality is tested, and the indexed attributes within the heap are tested.

Structural corruption can happen due to faulty storage hardware, or relation files being overwritten or modified by unrelated software. This kind of corruption can also be detected with data page checksums.

Relation pages which are correctly formatted, internally consistent, and correct relative to their own internal checksums may still contain logical corruption. As such, this kind of corruption cannot be detected with checksums. Examples include toasted values in the main table which lack a corresponding entry in the toast table, and tuples in the main table with a Transaction ID that is older than the oldest valid Transaction ID in the database or cluster.

Multiple causes of logical corruption have been observed in production systems, including bugs in the PostgreSQL server software, faulty and ill-conceived backup and restore tools, and user error.

Corrupt relations are most concerning in live production environments, precisely the same environments where high risk activities are least welcome. For this reason, verify_heapam has been designed to diagnose corruption without undue risk. It cannot guard against all causes of backend crashes, as even executing the calling query could be unsafe on a badly corrupted system. Access to catalog tables is performed and could be problematic if the catalogs themselves are corrupted.

In general, amcheck can only prove the presence of corruption; it cannot prove its absence.

F.2.4. Repairing Corruption

No error concerning corruption raised by amcheck should ever be a false positive. amcheck raises errors in the event of conditions that, by definition, should never happen, and so careful analysis of amcheck errors is often required.

There is no general method of repairing problems that amcheck detects. An explanation for the root cause of an invariant violation should be sought. pageinspect may play a useful role in diagnosing corruption that amcheck detects. A REINDEX may not be effective in repairing corruption.

Prev | Up | Next
F.1. adminpack | Home | F.3. auth_delay


F.3. auth_delay
Prev Up Appendix F. Additional Supplied Modules Home Next

F.3. auth_delay

F.3.1. Configuration Parameters
F.3.2. Author

auth_delay causes the server to pause briefly before reporting authentication failure, to make brute-force attacks on database passwords more difficult. Note that it does nothing to prevent denial-of-service attacks, and may even exacerbate them, since processes that are waiting before reporting authentication failure will still consume connection slots.

In order to function, this module must be loaded via shared_preload_libraries in postgresql.conf.

F.3.1. Configuration Parameters

auth_delay.milliseconds (int)

The number of milliseconds to wait before reporting an authentication failure. The default is 0.

These parameters must be set in postgresql.conf. Typical usage might be:

# postgresql.conf
shared_preload_libraries = 'auth_delay'

auth_delay.milliseconds = '500'

F.3.2. Author

KaiGai Kohei <kaigai@ak.jp.nec.com>

Prev | Up | Next
F.2. amcheck | Home | F.4. auto_explain


F.4. auto_explain
Prev Up Appendix F. Additional Supplied Modules Home Next

F.4. auto_explain

F.4.1. Configuration Parameters
F.4.2. Example
F.4.3. Author

The auto_explain module provides a means for logging execution plans of slow statements automatically, without having to run EXPLAIN by hand. This is especially helpful for tracking down un-optimized queries in large applications.

The module provides no SQL-accessible functions. To use it, simply load it into the server. You can load it into an individual session:

LOAD 'auto_explain';

(You must be superuser to do that.) More typical usage is to preload it into some or all sessions by including auto_explain in session_preload_libraries or shared_preload_libraries in postgresql.conf. Then you can track unexpectedly slow queries no matter when they happen. Of course there is a price in overhead for that.

F.4.1. Configuration Parameters

There are several configuration parameters that control the behavior of auto_explain. Note that the default behavior is to do nothing, so you must set at least auto_explain.log_min_duration if you want any results.

auto_explain.log_min_duration (integer)

auto_explain.log_min_duration is the minimum statement execution time, in milliseconds, that will cause the statement's plan to be logged. Setting this to 0 logs all plans. -1 (the default) disables logging of plans. For example, if you set it to 250ms then all statements that run 250ms or longer will be logged. Only superusers can change this setting.

auto_explain.log_analyze (boolean)

auto_explain.log_analyze causes EXPLAIN ANALYZE output, rather than just EXPLAIN output, to be printed when an execution plan is logged. This parameter is off by default. Only superusers can change this setting.

Note

When this parameter is on, per-plan-node timing occurs for all statements executed, whether or not they run long enough to actually get logged. This can have an extremely negative impact on performance. Turning off auto_explain.log_timing ameliorates the performance cost, at the price of obtaining less information.

auto_explain.log_buffers (boolean)

auto_explain.log_buffers controls whether buffer usage statistics are printed when an execution plan is logged; it's equivalent to the BUFFERS option of EXPLAIN. This parameter has no effect unless auto_explain.log_analyze is enabled. This parameter is off by default. Only superusers can change this setting.

auto_explain.log_wal (boolean)

auto_explain.log_wal controls whether WAL usage statistics are printed when an execution plan is logged; it's equivalent to the WAL option of EXPLAIN. This parameter has no effect unless auto_explain.log_analyze is enabled. This parameter is off by default. Only superusers can change this setting.

auto_explain.log_timing (boolean)

auto_explain.log_timing controls whether per-node timing information is printed when an execution plan is logged; it's equivalent to the TIMING option of EXPLAIN. The overhead of repeatedly reading the system clock can slow down queries significantly on some systems, so it may be useful to set this parameter to off when only actual row counts, and not exact times, are needed. This parameter has no effect unless auto_explain.log_analyze is enabled. This parameter is on by default. Only superusers can change this setting.

auto_explain.log_triggers (boolean)

auto_explain.log_triggers causes trigger execution statistics to be included when an execution plan is logged. This parameter has no effect unless auto_explain.log_analyze is enabled. This parameter is off by default. Only superusers can change this setting.

auto_explain.log_verbose (boolean)

auto_explain.log_verbose controls whether verbose details are printed when an execution plan is logged; it's equivalent to the VERBOSE option of EXPLAIN. This parameter is off by default. Only superusers can change this setting.

auto_explain.log_settings (boolean)

auto_explain.log_settings controls whether information about modified configuration options is printed when an execution plan is logged. Only options affecting query planning with value different from the built-in default value are included in the output. This parameter is off by default. Only superusers can change this setting.

auto_explain.log_format (enum)

auto_explain.log_format selects the EXPLAIN output format to be used. The allowed values are text, xml, json, and yaml. The default is text. Only superusers can change this setting.

auto_explain.log_level (enum)

auto_explain.log_level selects the log level at which auto_explain will log the query plan. Valid values are DEBUG5, DEBUG4, DEBUG3, DEBUG2, DEBUG1, INFO, NOTICE, WARNING, and LOG. The default is LOG. Only superusers can change this setting.

auto_explain.log_nested_statements (boolean)

auto_explain.log_nested_statements causes nested statements (statements executed inside a function) to be considered for logging. When it is off, only top-level query plans are logged. This parameter is off by default. Only superusers can change this setting.

auto_explain.sample_rate (real)

auto_explain.sample_rate causes auto_explain to only explain a fraction of the statements in each session. The default is 1, meaning explain all the queries. In case of nested statements, either all will be explained or none. Only superusers can change this setting.

In ordinary usage, these parameters are set in postgresql.conf, although superusers can alter them on-the-fly within their own sessions. Typical usage might be:

# postgresql.conf
session_preload_libraries = 'auto_explain'

auto_explain.log_min_duration = '3s'

F.4.2. Example

postgres=# LOAD 'auto_explain';
postgres=# SET auto_explain.log_min_duration = 0;
postgres=# SET auto_explain.log_analyze = true;
postgres=# SELECT count(*)
           FROM pg_class, pg_index
           WHERE oid = indrelid AND indisunique;

This might produce log output such as:

LOG:  duration: 3.651 ms  plan:
  Query Text: SELECT count(*)
              FROM pg_class, pg_index
              WHERE oid = indrelid AND indisunique;
  Aggregate  (cost=16.79..16.80 rows=1 width=0) (actual time=3.626..3.627 rows=1 loops=1)
    ->  Hash Join  (cost=4.17..16.55 rows=92 width=0) (actual time=3.349..3.594 rows=92 loops=1)
          Hash Cond: (pg_class.oid = pg_index.indrelid)
          ->  Seq Scan on pg_class  (cost=0.00..9.55 rows=255 width=4) (actual time=0.016..0.140 rows=255 loops=1)
          ->  Hash  (cost=3.02..3.02 rows=92 width=4) (actual time=3.238..3.238 rows=92 loops=1)
                Buckets: 1024  Batches: 1  Memory Usage: 4kB
                ->  Seq Scan on pg_index  (cost=0.00..3.02 rows=92 width=4) (actual time=0.008..3.187 rows=92 loops=1)
                      Filter: indisunique

F.4.3. Author

Takahiro Itagaki <itagaki.takahiro@oss.ntt.co.jp>

Prev | Up | Next
F.3. auth_delay | Home | F.5. bloom


F.5. bloom
Prev Up Appendix F. Additional Supplied Modules Home Next

F.5. bloom

F.5.1. Parameters
F.5.2. Examples
F.5.3. Operator Class Interface
F.5.4. Limitations
F.5.5. Authors

bloom provides an index access method based on Bloom filters.

A Bloom filter is a space-efficient data structure that is used to test whether an element is a member of a set. In the case of an index access method, it allows fast exclusion of non-matching tuples via signatures whose size is determined at index creation.

A signature is a lossy representation of the indexed attribute(s), and as such is prone to reporting false positives; that is, it may be reported that an element is in the set, when it is not. So index search results must always be rechecked using the actual attribute values from the heap entry. Larger signatures reduce the odds of a false positive and thus reduce the number of useless heap visits, but of course also make the index larger and hence slower to scan.

This type of index is most useful when a table has many attributes and queries test arbitrary combinations of them. A traditional btree index is faster than a bloom index, but it can require many btree indexes to support all possible queries where one needs only a single bloom index. Note however that bloom indexes only support equality queries, whereas btree indexes can also perform inequality and range searches.

F.5.1. Parameters

A bloom index accepts the following parameters in its WITH clause:

length

Length of each signature (index entry) in bits. It is rounded up to the nearest multiple of 16. The default is 80 bits and the maximum is 4096.

col1 — col32

Number of bits generated for each index column. Each parameter's name refers to the number of the index column that it controls. The default is 2 bits and the maximum is 4095. Parameters for index columns not actually used are ignored.

F.5.2. Examples

This is an example of creating a bloom index:

CREATE INDEX bloomidx ON tbloom USING bloom (i1,i2,i3)
       WITH (length=80, col1=2, col2=2, col3=4);

The index is created with a signature length of 80 bits, with attributes i1 and i2 mapped to 2 bits, and attribute i3 mapped to 4 bits. We could have omitted the length, col1, and col2 specifications since those have the default values.

Here is a more complete example of bloom index definition and usage, as well as a comparison with equivalent btree indexes. The bloom index is considerably smaller than the btree index, and can perform better.

=# CREATE TABLE tbloom AS
   SELECT
     (random() * 1000000)::int as i1,
     (random() * 1000000)::int as i2,
     (random() * 1000000)::int as i3,
     (random() * 1000000)::int as i4,
     (random() * 1000000)::int as i5,
     (random() * 1000000)::int as i6
   FROM
  generate_series(1,10000000);
SELECT 10000000

A sequential scan over this large table takes a long time:

=# EXPLAIN ANALYZE SELECT * FROM tbloom WHERE i2 = 898732 AND i5 = 123451;
                                              QUERY PLAN
-------------------------------------------------------------------​-----------------------------------
 Seq Scan on tbloom  (cost=0.00..2137.14 rows=3 width=24) (actual time=16.971..16.971 rows=0 loops=1)
   Filter: ((i2 = 898732) AND (i5 = 123451))
   Rows Removed by Filter: 100000
 Planning Time: 0.346 ms
 Execution Time: 16.988 ms
(5 rows)

Even with the btree index defined the result will still be a sequential scan:

=# CREATE INDEX btreeidx ON tbloom (i1, i2, i3, i4, i5, i6);
CREATE INDEX
=# SELECT pg_size_pretty(pg_relation_size('btreeidx'));
 pg_size_pretty
----------------
 3976 kB
(1 row)
=# EXPLAIN ANALYZE SELECT * FROM tbloom WHERE i2 = 898732 AND i5 = 123451;
                                              QUERY PLAN
-------------------------------------------------------------------​-----------------------------------
 Seq Scan on tbloom  (cost=0.00..2137.00 rows=2 width=24) (actual time=12.805..12.805 rows=0 loops=1)
   Filter: ((i2 = 898732) AND (i5 = 123451))
   Rows Removed by Filter: 100000
 Planning Time: 0.138 ms
 Execution Time: 12.817 ms
(5 rows)

Having the bloom index defined on the table is better than btree in handling this type of search:

=# CREATE INDEX bloomidx ON tbloom USING bloom (i1, i2, i3, i4, i5, i6);
CREATE INDEX
=# SELECT pg_size_pretty(pg_relation_size('bloomidx'));
 pg_size_pretty
----------------
 1584 kB
(1 row)
=# EXPLAIN ANALYZE SELECT * FROM tbloom WHERE i2 = 898732 AND i5 = 123451;
                                                     QUERY PLAN
-------------------------------------------------------------------​--------------------------------------------------
 Bitmap Heap Scan on tbloom  (cost=1792.00..1799.69 rows=2 width=24) (actual time=0.388..0.388 rows=0 loops=1)
   Recheck Cond: ((i2 = 898732) AND (i5 = 123451))
   Rows Removed by Index Recheck: 29
   Heap Blocks: exact=28
   ->  Bitmap Index Scan on bloomidx  (cost=0.00..1792.00 rows=2 width=0) (actual time=0.356..0.356 rows=29 loops=1)
         Index Cond: ((i2 = 898732) AND (i5 = 123451))
 Planning Time: 0.099 ms
 Execution Time: 0.408 ms
(8 rows)

Now, the main problem with the btree search is that btree is inefficient when the search conditions do not constrain the leading index column(s). A better strategy for btree is to create a separate index on each column. Then the planner will choose something like this:

=# CREATE INDEX btreeidx1 ON tbloom (i1);
CREATE INDEX
=# CREATE INDEX btreeidx2 ON tbloom (i2);
CREATE INDEX
=# CREATE INDEX btreeidx3 ON tbloom (i3);
CREATE INDEX
=# CREATE INDEX btreeidx4 ON tbloom (i4);
CREATE INDEX
=# CREATE INDEX btreeidx5 ON tbloom (i5);
CREATE INDEX
=# CREATE INDEX btreeidx6 ON tbloom (i6);
CREATE INDEX
=# EXPLAIN ANALYZE SELECT * FROM tbloom WHERE i2 = 898732 AND i5 = 123451;
                                                        QUERY PLAN
-------------------------------------------------------------------​--------------------------------------------------------
 Bitmap Heap Scan on tbloom  (cost=24.34..32.03 rows=2 width=24) (actual time=0.028..0.029 rows=0 loops=1)
   Recheck Cond: ((i5 = 123451) AND (i2 = 898732))
   ->  BitmapAnd  (cost=24.34..24.34 rows=2 width=0) (actual time=0.027..0.027 rows=0 loops=1)
         ->  Bitmap Index Scan on btreeidx5  (cost=0.00..12.04 rows=500 width=0) (actual time=0.026..0.026 rows=0 loops=1)
               Index Cond: (i5 = 123451)
         ->  Bitmap Index Scan on btreeidx2  (cost=0.00..12.04 rows=500 width=0) (never executed)
               Index Cond: (i2 = 898732)
 Planning Time: 0.491 ms
 Execution Time: 0.055 ms
(9 rows)

Although this query runs much faster than with either of the single indexes, we pay a penalty in index size. Each of the single-column btree indexes occupies 2 MB, so the total space needed is 12 MB, eight times the space used by the bloom index.

F.5.3. Operator Class Interface

An operator class for bloom indexes requires only a hash function for the indexed data type and an equality operator for searching. This example shows the operator class definition for the text data type:

CREATE OPERATOR CLASS text_ops
DEFAULT FOR TYPE text USING bloom AS
    OPERATOR    1   =(text, text),
    FUNCTION    1   hashtext(text);

F.5.4. Limitations

  • Only operator classes for int4 and text are included with the module.

  • Only the = operator is supported for search. But it is possible to add support for arrays with union and intersection operations in the future.

  • bloom access method doesn't support UNIQUE indexes.

  • bloom access method doesn't support searching for NULL values.

F.5.5. Authors

Teodor Sigaev <teodor@postgrespro.ru>, Postgres Professional, Moscow, Russia

Alexander Korotkov <a.korotkov@postgrespro.ru>, Postgres Professional, Moscow, Russia

Oleg Bartunov <obartunov@postgrespro.ru>, Postgres Professional, Moscow, Russia

Prev | Up | Next
F.4. auto_explain | Home | F.6. btree_gin


F.6. btree_gin
Prev Up Appendix F. Additional Supplied Modules Home Next

F.6. btree_gin

F.6.1. Example Usage
F.6.2. Authors

btree_gin provides sample GIN operator classes that implement B-tree equivalent behavior for the data types int2, int4, int8, float4, float8, timestamp with time zone, timestamp without time zone, time with time zone, time without time zone, date, interval, oid, money, "char", varchar, text, bytea, bit, varbit, macaddr, macaddr8, inet, cidr, uuid, name, bool, bpchar, and all enum types.

In general, these operator classes will not outperform the equivalent standard B-tree index methods, and they lack one major feature of the standard B-tree code: the ability to enforce uniqueness. However, they are useful for GIN testing and as a base for developing other GIN operator classes. Also, for queries that test both a GIN-indexable column and a B-tree-indexable column, it might be more efficient to create a multicolumn GIN index that uses one of these operator classes than to create two separate indexes that would have to be combined via bitmap ANDing.

This module is considered “trusted”, that is, it can be installed by non-superusers who have CREATE privilege on the current database.

F.6.1. Example Usage

CREATE TABLE test (a int4);
-- create index
CREATE INDEX testidx ON test USING GIN (a);
-- query
SELECT * FROM test WHERE a < 10;

F.6.2. Authors

Teodor Sigaev (<teodor@stack.net>) and Oleg Bartunov (<oleg@sai.msu.su>). See http://www.sai.msu.su/~megera/oddmuse/index.cgi/Gin for additional information.

Prev | Up | Next
F.5. bloom | Home | F.7. btree_gist


F.7. btree_gist
Prev Up Appendix F. Additional Supplied Modules Home Next

F.7. btree_gist

F.7.1. Example Usage
F.7.2. Authors

btree_gist provides GiST index operator classes that implement B-tree equivalent behavior for the data types int2, int4, int8, float4, float8, numeric, timestamp with time zone, timestamp without time zone, time with time zone, time without time zone, date, interval, oid, money, char, varchar, text, bytea, bit, varbit, macaddr, macaddr8, inet, cidr, uuid, and all enum types.

In general, these operator classes will not outperform the equivalent standard B-tree index methods, and they lack one major feature of the standard B-tree code: the ability to enforce uniqueness. However, they provide some other features that are not available with a B-tree index, as described below. Also, these operator classes are useful when a multicolumn GiST index is needed, wherein some of the columns are of data types that are only indexable with GiST but other columns are just simple data types. Lastly, these operator classes are useful for GiST testing and as a base for developing other GiST operator classes.

In addition to the typical B-tree search operators, btree_gist also provides index support for <> (“not equals”). This may be useful in combination with an exclusion constraint, as described below.

Also, for data types for which there is a natural distance metric, btree_gist defines a distance operator <->, and provides GiST index support for nearest-neighbor searches using this operator. Distance operators are provided for int2, int4, int8, float4, float8, timestamp with time zone, timestamp without time zone, time without time zone, date, interval, oid, and money.

This module is considered “trusted”, that is, it can be installed by non-superusers who have CREATE privilege on the current database.

F.7.1. Example Usage

Simple example using btree_gist instead of btree:

CREATE TABLE test (a int4);
-- create index
CREATE INDEX testidx ON test USING GIST (a);
-- query
SELECT * FROM test WHERE a < 10;
-- nearest-neighbor search: find the ten entries closest to "42"
SELECT *, a <-> 42 AS dist FROM test ORDER BY a <-> 42 LIMIT 10;

Use an exclusion constraint to enforce the rule that a cage at a zoo can contain only one kind of animal:

=> CREATE TABLE zoo (
  cage   INTEGER,
  animal TEXT,
  EXCLUDE USING GIST (cage WITH =, animal WITH <>)
);

=> INSERT INTO zoo VALUES(123, 'zebra');
INSERT 0 1
=> INSERT INTO zoo VALUES(123, 'zebra');
INSERT 0 1
=> INSERT INTO zoo VALUES(123, 'lion');
ERROR:  conflicting key value violates exclusion constraint "zoo_cage_animal_excl"
DETAIL:  Key (cage, animal)=(123, lion) conflicts with existing key (cage, animal)=(123, zebra).
=> INSERT INTO zoo VALUES(124, 'lion');
INSERT 0 1

F.7.2. Authors

Teodor Sigaev (<teodor@stack.net>), Oleg Bartunov (<oleg@sai.msu.su>), Janko Richter (<jankorichter@yahoo.de>), and Paul Jungwirth (<pj@illuminatedcomputing.com>). See http://www.sai.msu.su/~megera/postgres/gist/ for additional information.

Prev | Up | Next
F.6. btree_gin | Home | F.8. citext


F.8. citext
Prev Up Appendix F. Additional Supplied Modules Home Next

F.8. citext

F.8.1. Rationale
F.8.2. How to Use It
F.8.3. String Comparison Behavior
F.8.4. Limitations
F.8.5. Author

The citext module provides a case-insensitive character string type, citext. Essentially, it internally calls lower when comparing values. Otherwise, it behaves almost exactly like text.

Tip

Consider using nondeterministic collations (see Section 24.2.2.4) instead of this module. They can be used for case-insensitive comparisons, accent-insensitive comparisons, and other combinations, and they handle more Unicode special cases correctly.

This module is considered “trusted”, that is, it can be installed by non-superusers who have CREATE privilege on the current database.

F.8.1. Rationale

The standard approach to doing case-insensitive matches in PostgreSQL has been to use the lower function when comparing values, for example

SELECT * FROM tab WHERE lower(col) = LOWER(?);

This works reasonably well, but has a number of drawbacks:

  • It makes your SQL statements verbose, and you always have to remember to use lower on both the column and the query value.

  • It won't use an index, unless you create a functional index using lower.

  • If you declare a column as UNIQUE or PRIMARY KEY, the implicitly generated index is case-sensitive. So it's useless for case-insensitive searches, and it won't enforce uniqueness case-insensitively.

The citext data type allows you to eliminate calls to lower in SQL queries, and allows a primary key to be case-insensitive. citext is locale-aware, just like text, which means that the matching of upper case and lower case characters is dependent on the rules of the database's LC_CTYPE setting. Again, this behavior is identical to the use of lower in queries. But because it's done transparently by the data type, you don't have to remember to do anything special in your queries.

F.8.2. How to Use It

Here's a simple example of usage:

CREATE TABLE users (
    nick CITEXT PRIMARY KEY,
    pass TEXT   NOT NULL
);

INSERT INTO users VALUES ( 'larry',  sha256(random()::text::bytea) );
INSERT INTO users VALUES ( 'Tom',    sha256(random()::text::bytea) );
INSERT INTO users VALUES ( 'Damian', sha256(random()::text::bytea) );
INSERT INTO users VALUES ( 'NEAL',   sha256(random()::text::bytea) );
INSERT INTO users VALUES ( 'Bjørn',  sha256(random()::text::bytea) );

SELECT * FROM users WHERE nick = 'Larry';

The SELECT statement will return one tuple, even though the nick column was set to larry and the query was for Larry.

F.8.3. String Comparison Behavior

citext performs comparisons by converting each string to lower case (as though lower were called) and then comparing the results normally. Thus, for example, two strings are considered equal if lower would produce identical results for them.

In order to emulate a case-insensitive collation as closely as possible, there are citext-specific versions of a number of string-processing operators and functions. So, for example, the regular expression operators ~ and ~* exhibit the same behavior when applied to citext: they both match case-insensitively. The same is true for !~ and !~*, as well as for the LIKE operators ~~ and ~~*, and !~~ and !~~*. If you'd like to match case-sensitively, you can cast the operator's arguments to text.

Similarly, all of the following functions perform matching case-insensitively if their arguments are citext:

  • regexp_match()

  • regexp_matches()

  • regexp_replace()

  • regexp_split_to_array()

  • regexp_split_to_table()

  • replace()

  • split_part()

  • strpos()

  • translate()

For the regexp functions, if you want to match case-sensitively, you can specify the “c” flag to force a case-sensitive match. Otherwise, you must cast to text before using one of these functions if you want case-sensitive behavior.

F.8.4. Limitations

  • citext's case-folding behavior depends on the LC_CTYPE setting of your database. How it compares values is therefore determined when the database is created. It is not truly case-insensitive in the terms defined by the Unicode standard. Effectively, what this means is that, as long as you're happy with your collation, you should be happy with citext's comparisons. But if you have data in different languages stored in your database, users of one language may find their query results are not as expected if the collation is for another language.

  • As of PostgreSQL 9.1, you can attach a COLLATE specification to citext columns or data values. Currently, citext operators will honor a non-default COLLATE specification while comparing case-folded strings, but the initial folding to lower case is always done according to the database's LC_CTYPE setting (that is, as though COLLATE "default" were given). This may be changed in a future release so that both steps follow the input COLLATE specification.

  • citext is not as efficient as text because the operator functions and the B-tree comparison functions must make copies of the data and convert it to lower case for comparisons. Also, only text can support B-Tree deduplication. However, citext is slightly more efficient than using lower to get case-insensitive matching.

  • citext doesn't help much if you need data to compare case-sensitively in some contexts and case-insensitively in other contexts. The standard answer is to use the text type and manually use the lower function when you need to compare case-insensitively; this works all right if case-insensitive comparison is needed only infrequently. If you need case-insensitive behavior most of the time and case-sensitive infrequently, consider storing the data as citext and explicitly casting the column to text when you want case-sensitive comparison. In either situation, you will need two indexes if you want both types of searches to be fast.

  • The schema containing the citext operators must be in the current search_path (typically public); if it is not, the normal case-sensitive text operators will be invoked instead.

  • The approach of lower-casing strings for comparison does not handle some Unicode special cases correctly, for example when one upper-case letter has two lower-case letter equivalents. Unicode distinguishes between case mapping and case folding for this reason. Use nondeterministic collations instead of citext to handle that correctly.

F.8.5. Author

David E. Wheeler <david@kineticode.com>

Inspired by the original citext module by Donald Fraser.

Prev | Up | Next
F.7. btree_gist | Home | F.9. cube


F.9. cube
Prev Up Appendix F. Additional Supplied Modules Home Next

F.9. cube

F.9.1. Syntax
F.9.2. Precision
F.9.3. Usage
F.9.4. Defaults
F.9.5. Notes
F.9.6. Credits

This module implements a data type cube for representing multidimensional cubes.

This module is considered “trusted”, that is, it can be installed by non-superusers who have CREATE privilege on the current database.

F.9.1. Syntax

Table F.2 shows the valid external representations for the cube type. x, y, etc. denote floating-point numbers.

Table F.2. Cube External Representations

External Syntax | Meaning
x | A one-dimensional point (or, zero-length one-dimensional interval)
(x) | Same as above
x1,x2,...,xn | A point in n-dimensional space, represented internally as a zero-volume cube
(x1,x2,...,xn) | Same as above
(x),(y) | A one-dimensional interval starting at x and ending at y or vice versa; the order does not matter
[(x),(y)] | Same as above
(x1,...,xn),(y1,...,yn) | An n-dimensional cube represented by a pair of its diagonally opposite corners
[(x1,...,xn),(y1,...,yn)] | Same as above

It does not matter which order the opposite corners of a cube are entered in. The cube functions automatically swap values if needed to create a uniform “lower left — upper right” internal representation. When the corners coincide, cube stores only one corner along with an “is point” flag to avoid wasting space.

White space is ignored on input, so [(x),(y)] is the same as [ ( x ), ( y ) ].

F.9.2. Precision

Values are stored internally as 64-bit floating point numbers. This means that numbers with more than about 16 significant digits will be truncated.

F.9.3. Usage

Table F.3 shows the specialized operators provided for type cube.

Table F.3. Cube Operators

Operator

Description

cube && cube → boolean

Do the cubes overlap?

cube @> cube → boolean

Does the first cube contain the second?

cube <@ cube → boolean

Is the first cube contained in the second?

cube -> integer → float8

Extracts the n-th coordinate of the cube (counting from 1).

cube ~> integer → float8

Extracts the n-th coordinate of the cube, counting in the following way: n = 2 * k - 1 means lower bound of k-th dimension, n = 2 * k means upper bound of k-th dimension. Negative n denotes the inverse value of the corresponding positive coordinate. This operator is designed for KNN-GiST support.

cube <-> cube → float8

Computes the Euclidean distance between the two cubes.

cube <#> cube → float8

Computes the taxicab (L-1 metric) distance between the two cubes.

cube <=> cube → float8

Computes the Chebyshev (L-inf metric) distance between the two cubes.

In addition to the above operators, the usual comparison operators shown in Table 9.1 are available for type cube. These operators first compare the first coordinates, and if those are equal, compare the second coordinates, etc. They exist mainly to support the b-tree index operator class for cube, which can be useful for example if you would like a UNIQUE constraint on a cube column. Otherwise, this ordering is not of much practical use.

The cube module also provides a GiST index operator class for cube values. A cube GiST index can be used to search for values using the =, &&, @>, and <@ operators in WHERE clauses.

In addition, a cube GiST index can be used to find nearest neighbors using the metric operators <->, <#>, and <=> in ORDER BY clauses. For example, the nearest neighbor of the 3-D point (0.5, 0.5, 0.5) could be found efficiently with:

SELECT c FROM test ORDER BY c <-> cube(array[0.5,0.5,0.5]) LIMIT 1;

The ~> operator can also be used in this way to efficiently retrieve the first few values sorted by a selected coordinate. For example, to get the first few cubes ordered by the first coordinate (lower left corner) ascending one could use the following query:

SELECT c FROM test ORDER BY c ~> 1 LIMIT 5;

And to get 2-D cubes ordered by the first coordinate of the upper right corner descending:

SELECT c FROM test ORDER BY c ~> 3 DESC LIMIT 5;

Table F.4 shows the available functions.

Table F.4. Cube Functions

Function

Description

Example(s)

cube ( float8 ) → cube

Makes a one dimensional cube with both coordinates the same.

cube(1) → (1)

cube ( float8, float8 ) → cube

Makes a one dimensional cube.

cube(1, 2) → (1),(2)

cube ( float8[] ) → cube

Makes a zero-volume cube using the coordinates defined by the array.

cube(ARRAY[1,2,3]) → (1, 2, 3)

cube ( float8[], float8[] ) → cube

Makes a cube with upper right and lower left coordinates as defined by the two arrays, which must be of the same length.

cube(ARRAY[1,2], ARRAY[3,4]) → (1, 2),(3, 4)

cube ( cube, float8 ) → cube

Makes a new cube by adding a dimension on to an existing cube, with the same values for both endpoints of the new coordinate. This is useful for building cubes piece by piece from calculated values.

cube('(1,2),(3,4)'::cube, 5) → (1, 2, 5),(3, 4, 5)

cube ( cube, float8, float8 ) → cube

Makes a new cube by adding a dimension on to an existing cube. This is useful for building cubes piece by piece from calculated values.

cube('(1,2),(3,4)'::cube, 5, 6) → (1, 2, 5),(3, 4, 6)

cube_dim ( cube ) → integer

Returns the number of dimensions of the cube.

cube_dim('(1,2),(3,4)') → 2

cube_ll_coord ( cube, integer ) → float8

Returns the n-th coordinate value for the lower left corner of the cube.

cube_ll_coord('(1,2),(3,4)', 2) → 2

cube_ur_coord ( cube, integer ) → float8

Returns the n-th coordinate value for the upper right corner of the cube.

cube_ur_coord('(1,2),(3,4)', 2) → 4

cube_is_point ( cube ) → boolean

Returns true if the cube is a point, that is, the two defining corners are the same.

cube_is_point(cube(1,1)) → t

cube_distance ( cube, cube ) → float8

Returns the distance between two cubes. If both cubes are points, this is the normal distance function.

cube_distance('(1,2)', '(3,4)') → 2.8284271247461903

cube_subset ( cube, integer[] ) → cube

Makes a new cube from an existing cube, using a list of dimension indexes from an array. Can be used to extract the endpoints of a single dimension, or to drop dimensions, or to reorder them as desired.

cube_subset(cube('(1,3,5),(6,7,8)'), ARRAY[2]) → (3),(7)

cube_subset(cube('(1,3,5),(6,7,8)'), ARRAY[3,2,1,1]) → (5, 3, 1, 1),(8, 7, 6, 6)

cube_union ( cube, cube ) → cube

Produces the union of two cubes.

cube_union('(1,2)', '(3,4)') → (1, 2),(3, 4)

cube_inter ( cube, cube ) → cube

Produces the intersection of two cubes.

cube_inter('(1,2)', '(3,4)') → (3, 4),(1, 2)

cube_enlarge ( c cube, r double, n integer ) → cube

Increases the size of the cube by the specified radius r in at least n dimensions. If the radius is negative the cube is shrunk instead. All defined dimensions are changed by the radius r. Lower-left coordinates are decreased by r and upper-right coordinates are increased by r. If a lower-left coordinate is increased to more than the corresponding upper-right coordinate (this can only happen when r < 0) than both coordinates are set to their average. If n is greater than the number of defined dimensions and the cube is being enlarged (r > 0), then extra dimensions are added to make n altogether; 0 is used as the initial value for the extra coordinates. This function is useful for creating bounding boxes around a point for searching for nearby points.

cube_enlarge('(1,2),(3,4)', 0.5, 3) → (0.5, 1.5, -0.5),(3.5, 4.5, 0.5)

F.9.4. Defaults

I believe this union:

select cube_union('(0,5,2),(2,3,1)', '0');
cube_union
-------------------
(0, 0, 0),(2, 5, 2)
(1 row)

does not contradict common sense, neither does the intersection

select cube_inter('(0,-1),(1,1)', '(-2),(2)');
cube_inter
-------------
(0, 0),(1, 0)
(1 row)

In all binary operations on differently-dimensioned cubes, I assume the lower-dimensional one to be a Cartesian projection, i. e., having zeroes in place of coordinates omitted in the string representation. The above examples are equivalent to:

cube_union('(0,5,2),(2,3,1)','(0,0,0),(0,0,0)');
cube_inter('(0,-1),(1,1)','(-2,0),(2,0)');

The following containment predicate uses the point syntax, while in fact the second argument is internally represented by a box. This syntax makes it unnecessary to define a separate point type and functions for (box,point) predicates.

select cube_contains('(0,0),(1,1)', '0.5,0.5');
cube_contains
--------------
t
(1 row)

F.9.5. Notes

For examples of usage, see the regression test sql/cube.sql.

To make it harder for people to break things, there is a limit of 100 on the number of dimensions of cubes. This is set in cubedata.h if you need something bigger.

F.9.6. Credits

Original author: Gene Selkov, Jr. <selkovjr@mcs.anl.gov>, Mathematics and Computer Science Division, Argonne National Laboratory.

My thanks are primarily to Prof. Joe Hellerstein (https://dsf.berkeley.edu/jmh/) for elucidating the gist of the GiST (http://gist.cs.berkeley.edu/), and to his former student Andy Dong for his example written for Illustra. I am also grateful to all Postgres developers, present and past, for enabling myself to create my own world and live undisturbed in it. And I would like to acknowledge my gratitude to Argonne Lab and to the U.S. Department of Energy for the years of faithful support of my database research.

Minor updates to this package were made by Bruno Wolff III <bruno@wolff.to> in August/September of 2002. These include changing the precision from single precision to double precision and adding some new functions.

Additional updates were made by Joshua Reich <josh@root.net> in July 2006. These include cube(float8[], float8[]) and cleaning up the code to use the V1 call protocol instead of the deprecated V0 protocol.

Prev | Up | Next
F.8. citext | Home | F.10. dblink


F.10. dblink
Prev Up Appendix F. Additional Supplied Modules Home Next

F.10. dblink

dblink_connect — opens a persistent connection to a remote database
dblink_connect_u — opens a persistent connection to a remote database, insecurely
dblink_disconnect — closes a persistent connection to a remote database
dblink — executes a query in a remote database
dblink_exec — executes a command in a remote database
dblink_open — opens a cursor in a remote database
dblink_fetch — returns rows from an open cursor in a remote database
dblink_close — closes a cursor in a remote database
dblink_get_connections — returns the names of all open named dblink connections
dblink_error_message — gets last error message on the named connection
dblink_send_query — sends an async query to a remote database
dblink_is_busy — checks if connection is busy with an async query
dblink_get_notify — retrieve async notifications on a connection
dblink_get_result — gets an async query result
dblink_cancel_query — cancels any active query on the named connection
dblink_get_pkey — returns the positions and field names of a relation's primary key fields
dblink_build_sql_insert — builds an INSERT statement using a local tuple, replacing the primary key field values with alternative supplied values
dblink_build_sql_delete — builds a DELETE statement using supplied values for primary key field values
dblink_build_sql_update — builds an UPDATE statement using a local tuple, replacing the primary key field values with alternative supplied values

dblink is a module that supports connections to other PostgreSQL databases from within a database session.

See also postgres_fdw, which provides roughly the same functionality using a more modern and standards-compliant infrastructure.

Prev | Up | Next
F.9. cube | Home | dblink_connect


dblink_connect
Prev Up F.10. dblink Home Next

dblink_connect

dblink_connect — opens a persistent connection to a remote database

Synopsis

dblink_connect(text connstr) returns text
dblink_connect(text connname, text connstr) returns text

Description

dblink_connect() establishes a connection to a remote PostgreSQL database. The server and database to be contacted are identified through a standard libpq connection string. Optionally, a name can be assigned to the connection. Multiple named connections can be open at once, but only one unnamed connection is permitted at a time. The connection will persist until closed or until the database session is ended.

The connection string may also be the name of an existing foreign server. It is recommended to use the foreign-data wrapper dblink_fdw when defining the foreign server. See the example below, as well as CREATE SERVER and CREATE USER MAPPING.

Arguments

connname

The name to use for this connection; if omitted, an unnamed connection is opened, replacing any existing unnamed connection.

connstr

libpq-style connection info string, for example hostaddr=127.0.0.1 port=5432 dbname=mydb user=postgres password=mypasswd options=-csearch_path=. For details see Section 34.1.1. Alternatively, the name of a foreign server.

Return Value

Returns status, which is always OK (since any error causes the function to throw an error instead of returning).

Notes

If untrusted users have access to a database that has not adopted a secure schema usage pattern, begin each session by removing publicly-writable schemas from search_path. One could, for example, add options=-csearch_path= to connstr. This consideration is not specific to dblink; it applies to every interface for executing arbitrary SQL commands.

Only superusers may use dblink_connect to create non-password-authenticated connections. If non-superusers need this capability, use dblink_connect_u instead.

It is unwise to choose connection names that contain equal signs, as this opens a risk of confusion with connection info strings in other dblink functions.

Examples

SELECT dblink_connect('dbname=postgres options=-csearch_path=');
 dblink_connect
----------------
 OK
(1 row)

SELECT dblink_connect('myconn', 'dbname=postgres options=-csearch_path=');
 dblink_connect
----------------
 OK
(1 row)

-- FOREIGN DATA WRAPPER functionality
-- Note: local connection must require password authentication for this to work properly
--       Otherwise, you will receive the following error from dblink_connect():
--       ERROR:  password is required
--       DETAIL:  Non-superuser cannot connect if the server does not request a password.
--       HINT:  Target server's authentication method must be changed.

CREATE SERVER fdtest FOREIGN DATA WRAPPER dblink_fdw OPTIONS (hostaddr '127.0.0.1', dbname 'contrib_regression');

CREATE USER regress_dblink_user WITH PASSWORD 'secret';
CREATE USER MAPPING FOR regress_dblink_user SERVER fdtest OPTIONS (user 'regress_dblink_user', password 'secret');
GRANT USAGE ON FOREIGN SERVER fdtest TO regress_dblink_user;
GRANT SELECT ON TABLE foo TO regress_dblink_user;

\set ORIGINAL_USER :USER
\c - regress_dblink_user
SELECT dblink_connect('myconn', 'fdtest');
 dblink_connect
----------------
 OK
(1 row)

SELECT * FROM dblink('myconn', 'SELECT * FROM foo') AS t(a int, b text, c text[]);
 a  | b |       c
----+---+---------------
  0 | a | {a0,b0,c0}
  1 | b | {a1,b1,c1}
  2 | c | {a2,b2,c2}
  3 | d | {a3,b3,c3}
  4 | e | {a4,b4,c4}
  5 | f | {a5,b5,c5}
  6 | g | {a6,b6,c6}
  7 | h | {a7,b7,c7}
  8 | i | {a8,b8,c8}
  9 | j | {a9,b9,c9}
 10 | k | {a10,b10,c10}
(11 rows)

\c - :ORIGINAL_USER
REVOKE USAGE ON FOREIGN SERVER fdtest FROM regress_dblink_user;
REVOKE SELECT ON TABLE foo FROM regress_dblink_user;
DROP USER MAPPING FOR regress_dblink_user SERVER fdtest;
DROP USER regress_dblink_user;
DROP SERVER fdtest;

Prev | Up | Next
F.10. dblink | Home | dblink_connect_u


dblink_connect_u
Prev Up F.10. dblink Home Next

dblink_connect_u

dblink_connect_u — opens a persistent connection to a remote database, insecurely

Synopsis

dblink_connect_u(text connstr) returns text
dblink_connect_u(text connname, text connstr) returns text

Description

dblink_connect_u() is identical to dblink_connect(), except that it will allow non-superusers to connect using any authentication method.

If the remote server selects an authentication method that does not involve a password, then impersonation and subsequent escalation of privileges can occur, because the session will appear to have originated from the user as which the local PostgreSQL server runs. Also, even if the remote server does demand a password, it is possible for the password to be supplied from the server environment, such as a ~/.pgpass file belonging to the server's user. This opens not only a risk of impersonation, but the possibility of exposing a password to an untrustworthy remote server. Therefore, dblink_connect_u() is initially installed with all privileges revoked from PUBLIC, making it un-callable except by superusers. In some situations it may be appropriate to grant EXECUTE permission for dblink_connect_u() to specific users who are considered trustworthy, but this should be done with care. It is also recommended that any ~/.pgpass file belonging to the server's user not contain any records specifying a wildcard host name.

For further details see dblink_connect().

Prev | Up | Next
dblink_connect | Home | dblink_disconnect


dblink_disconnect
Prev Up F.10. dblink Home Next

dblink_disconnect

dblink_disconnect — closes a persistent connection to a remote database

Synopsis

dblink_disconnect() returns text
dblink_disconnect(text connname) returns text

Description

dblink_disconnect() closes a connection previously opened by dblink_connect(). The form with no arguments closes an unnamed connection.

Arguments

connname

The name of a named connection to be closed.

Return Value

Returns status, which is always OK (since any error causes the function to throw an error instead of returning).

Examples

SELECT dblink_disconnect();
 dblink_disconnect
-------------------
 OK
(1 row)

SELECT dblink_disconnect('myconn');
 dblink_disconnect
-------------------
 OK
(1 row)

Prev | Up | Next
dblink_connect_u | Home | dblink


dblink
Prev Up F.10. dblink Home Next

dblink

dblink — executes a query in a remote database

Synopsis

dblink(text connname, text sql [, bool fail_on_error]) returns setof record
dblink(text connstr, text sql [, bool fail_on_error]) returns setof record
dblink(text sql [, bool fail_on_error]) returns setof record

Description

dblink executes a query (usually a SELECT, but it can be any SQL statement that returns rows) in a remote database.

When two text arguments are given, the first one is first looked up as a persistent connection's name; if found, the command is executed on that connection. If not found, the first argument is treated as a connection info string as for dblink_connect, and the indicated connection is made just for the duration of this command.

Arguments

connname

Name of the connection to use; omit this parameter to use the unnamed connection.

connstr

A connection info string, as previously described for dblink_connect.

sql

The SQL query that you wish to execute in the remote database, for example select * from foo.

fail_on_error

If true (the default when omitted) then an error thrown on the remote side of the connection causes an error to also be thrown locally. If false, the remote error is locally reported as a NOTICE, and the function returns no rows.

Return Value

The function returns the row(s) produced by the query. Since dblink can be used with any query, it is declared to return record, rather than specifying any particular set of columns. This means that you must specify the expected set of columns in the calling query — otherwise PostgreSQL would not know what to expect. Here is an example:

SELECT *
    FROM dblink('dbname=mydb options=-csearch_path=',
                'select proname, prosrc from pg_proc')
      AS t1(proname name, prosrc text)
    WHERE proname LIKE 'bytea%';

The “alias” part of the FROM clause must specify the column names and types that the function will return. (Specifying column names in an alias is actually standard SQL syntax, but specifying column types is a PostgreSQL extension.) This allows the system to understand what * should expand to, and what proname in the WHERE clause refers to, in advance of trying to execute the function. At run time, an error will be thrown if the actual query result from the remote database does not have the same number of columns shown in the FROM clause. The column names need not match, however, and dblink does not insist on exact type matches either. It will succeed so long as the returned data strings are valid input for the column type declared in the FROM clause.

Notes

A convenient way to use dblink with predetermined queries is to create a view. This allows the column type information to be buried in the view, instead of having to spell it out in every query. For example,

CREATE VIEW myremote_pg_proc AS
  SELECT *
    FROM dblink('dbname=postgres options=-csearch_path=',
                'select proname, prosrc from pg_proc')
    AS t1(proname name, prosrc text);

SELECT * FROM myremote_pg_proc WHERE proname LIKE 'bytea%';

Examples

SELECT * FROM dblink('dbname=postgres options=-csearch_path=',
                     'select proname, prosrc from pg_proc')
  AS t1(proname name, prosrc text) WHERE proname LIKE 'bytea%';
  proname   |   prosrc
------------+------------
 byteacat   | byteacat
 byteaeq    | byteaeq
 bytealt    | bytealt
 byteale    | byteale
 byteagt    | byteagt
 byteage    | byteage
 byteane    | byteane
 byteacmp   | byteacmp
 bytealike  | bytealike
 byteanlike | byteanlike
 byteain    | byteain
 byteaout   | byteaout
(12 rows)

SELECT dblink_connect('dbname=postgres options=-csearch_path=');
 dblink_connect
----------------
 OK
(1 row)

SELECT * FROM dblink('select proname, prosrc from pg_proc')
  AS t1(proname name, prosrc text) WHERE proname LIKE 'bytea%';
  proname   |   prosrc
------------+------------
 byteacat   | byteacat
 byteaeq    | byteaeq
 bytealt    | bytealt
 byteale    | byteale
 byteagt    | byteagt
 byteage    | byteage
 byteane    | byteane
 byteacmp   | byteacmp
 bytealike  | bytealike
 byteanlike | byteanlike
 byteain    | byteain
 byteaout   | byteaout
(12 rows)

SELECT dblink_connect('myconn', 'dbname=regression options=-csearch_path=');
 dblink_connect
----------------
 OK
(1 row)

SELECT * FROM dblink('myconn', 'select proname, prosrc from pg_proc')
  AS t1(proname name, prosrc text) WHERE proname LIKE 'bytea%';
  proname   |   prosrc
------------+------------
 bytearecv  | bytearecv
 byteasend  | byteasend
 byteale    | byteale
 byteagt    | byteagt
 byteage    | byteage
 byteane    | byteane
 byteacmp   | byteacmp
 bytealike  | bytealike
 byteanlike | byteanlike
 byteacat   | byteacat
 byteaeq    | byteaeq
 bytealt    | bytealt
 byteain    | byteain
 byteaout   | byteaout
(14 rows)

Prev | Up | Next
dblink_disconnect | Home | dblink_exec


dblink_exec
Prev Up F.10. dblink Home Next

dblink_exec

dblink_exec — executes a command in a remote database

Synopsis

dblink_exec(text connname, text sql [, bool fail_on_error]) returns text
dblink_exec(text connstr, text sql [, bool fail_on_error]) returns text
dblink_exec(text sql [, bool fail_on_error]) returns text

Description

dblink_exec executes a command (that is, any SQL statement that doesn't return rows) in a remote database.

When two text arguments are given, the first one is first looked up as a persistent connection's name; if found, the command is executed on that connection. If not found, the first argument is treated as a connection info string as for dblink_connect, and the indicated connection is made just for the duration of this command.

Arguments

connname

Name of the connection to use; omit this parameter to use the unnamed connection.

connstr

A connection info string, as previously described for dblink_connect.

sql

The SQL command that you wish to execute in the remote database, for example insert into foo values(0, 'a', '{"a0","b0","c0"}').

fail_on_error

If true (the default when omitted) then an error thrown on the remote side of the connection causes an error to also be thrown locally. If false, the remote error is locally reported as a NOTICE, and the function's return value is set to ERROR.

Return Value

Returns status, either the command's status string or ERROR.

Examples

SELECT dblink_connect('dbname=dblink_test_standby');
 dblink_connect
----------------
 OK
(1 row)

SELECT dblink_exec('insert into foo values(21, ''z'', ''{"a0","b0","c0"}'');');
   dblink_exec
-----------------
 INSERT 943366 1
(1 row)

SELECT dblink_connect('myconn', 'dbname=regression');
 dblink_connect
----------------
 OK
(1 row)

SELECT dblink_exec('myconn', 'insert into foo values(21, ''z'', ''{"a0","b0","c0"}'');');
   dblink_exec
------------------
 INSERT 6432584 1
(1 row)

SELECT dblink_exec('myconn', 'insert into pg_class values (''foo'')',false);
NOTICE:  sql error
DETAIL:  ERROR:  null value in column "relnamespace" violates not-null constraint

 dblink_exec
-------------
 ERROR
(1 row)

Prev | Up | Next
dblink | Home | dblink_open


dblink_open
Prev Up F.10. dblink Home Next

dblink_open

dblink_open — opens a cursor in a remote database

Synopsis

dblink_open(text cursorname, text sql [, bool fail_on_error]) returns text
dblink_open(text connname, text cursorname, text sql [, bool fail_on_error]) returns text

Description

dblink_open() opens a cursor in a remote database. The cursor can subsequently be manipulated with dblink_fetch() and dblink_close().

Arguments

connname

Name of the connection to use; omit this parameter to use the unnamed connection.

cursorname

The name to assign to this cursor.

sql

The SELECT statement that you wish to execute in the remote database, for example select * from pg_class.

fail_on_error

If true (the default when omitted) then an error thrown on the remote side of the connection causes an error to also be thrown locally. If false, the remote error is locally reported as a NOTICE, and the function's return value is set to ERROR.

Return Value

Returns status, either OK or ERROR.

Notes

Since a cursor can only persist within a transaction, dblink_open starts an explicit transaction block (BEGIN) on the remote side, if the remote side was not already within a transaction. This transaction will be closed again when the matching dblink_close is executed. Note that if you use dblink_exec to change data between dblink_open and dblink_close, and then an error occurs or you use dblink_disconnect before dblink_close, your change will be lost because the transaction will be aborted.

Examples

SELECT dblink_connect('dbname=postgres options=-csearch_path=');
 dblink_connect
----------------
 OK
(1 row)

SELECT dblink_open('foo', 'select proname, prosrc from pg_proc');
 dblink_open
-------------
 OK
(1 row)

Prev | Up | Next
dblink_exec | Home | dblink_fetch


dblink_fetch
Prev Up F.10. dblink Home Next

dblink_fetch

dblink_fetch — returns rows from an open cursor in a remote database

Synopsis

dblink_fetch(text cursorname, int howmany [, bool fail_on_error]) returns setof record
dblink_fetch(text connname, text cursorname, int howmany [, bool fail_on_error]) returns setof record

Description

dblink_fetch fetches rows from a cursor previously established by dblink_open.

Arguments

connname

Name of the connection to use; omit this parameter to use the unnamed connection.

cursorname

The name of the cursor to fetch from.

howmany

The maximum number of rows to retrieve. The next howmany rows are fetched, starting at the current cursor position, moving forward. Once the cursor has reached its end, no more rows are produced.

fail_on_error

If true (the default when omitted) then an error thrown on the remote side of the connection causes an error to also be thrown locally. If false, the remote error is locally reported as a NOTICE, and the function returns no rows.

Return Value

The function returns the row(s) fetched from the cursor. To use this function, you will need to specify the expected set of columns, as previously discussed for dblink.

Notes

On a mismatch between the number of return columns specified in the FROM clause, and the actual number of columns returned by the remote cursor, an error will be thrown. In this event, the remote cursor is still advanced by as many rows as it would have been if the error had not occurred. The same is true for any other error occurring in the local query after the remote FETCH has been done.

Examples

SELECT dblink_connect('dbname=postgres options=-csearch_path=');
 dblink_connect
----------------
 OK
(1 row)

SELECT dblink_open('foo', 'select proname, prosrc from pg_proc where proname like ''bytea%''');
 dblink_open
-------------
 OK
(1 row)

SELECT * FROM dblink_fetch('foo', 5) AS (funcname name, source text);
 funcname |  source
----------+----------
 byteacat | byteacat
 byteacmp | byteacmp
 byteaeq  | byteaeq
 byteage  | byteage
 byteagt  | byteagt
(5 rows)

SELECT * FROM dblink_fetch('foo', 5) AS (funcname name, source text);
 funcname  |  source
-----------+-----------
 byteain   | byteain
 byteale   | byteale
 bytealike | bytealike
 bytealt   | bytealt
 byteane   | byteane
(5 rows)

SELECT * FROM dblink_fetch('foo', 5) AS (funcname name, source text);
  funcname  |   source
------------+------------
 byteanlike | byteanlike
 byteaout   | byteaout
(2 rows)

SELECT * FROM dblink_fetch('foo', 5) AS (funcname name, source text);
 funcname | source
----------+--------
(0 rows)

Prev | Up | Next
dblink_open | Home | dblink_close


dblink_close
Prev Up F.10. dblink Home Next

dblink_close

dblink_close — closes a cursor in a remote database

Synopsis

dblink_close(text cursorname [, bool fail_on_error]) returns text
dblink_close(text connname, text cursorname [, bool fail_on_error]) returns text

Description

dblink_close closes a cursor previously opened with dblink_open.

Arguments

connname

Name of the connection to use; omit this parameter to use the unnamed connection.

cursorname

The name of the cursor to close.

fail_on_error

If true (the default when omitted) then an error thrown on the remote side of the connection causes an error to also be thrown locally. If false, the remote error is locally reported as a NOTICE, and the function's return value is set to ERROR.

Return Value

Returns status, either OK or ERROR.

Notes

If dblink_open started an explicit transaction block, and this is the last remaining open cursor in this connection, dblink_close will issue the matching COMMIT.

Examples

SELECT dblink_connect('dbname=postgres options=-csearch_path=');
 dblink_connect
----------------
 OK
(1 row)

SELECT dblink_open('foo', 'select proname, prosrc from pg_proc');
 dblink_open
-------------
 OK
(1 row)

SELECT dblink_close('foo');
 dblink_close
--------------
 OK
(1 row)

Prev | Up | Next
dblink_fetch | Home | dblink_get_connections


dblink_get_connections
Prev Up F.10. dblink Home Next

dblink_get_connections

dblink_get_connections — returns the names of all open named dblink connections

Synopsis

dblink_get_connections() returns text[]

Description

dblink_get_connections returns an array of the names of all open named dblink connections.

Return Value

Returns a text array of connection names, or NULL if none.

Examples

SELECT dblink_get_connections();

Prev | Up | Next
dblink_close | Home | dblink_error_message


dblink_error_message
Prev Up F.10. dblink Home Next

dblink_error_message

dblink_error_message — gets last error message on the named connection

Synopsis

dblink_error_message(text connname) returns text

Description

dblink_error_message fetches the most recent remote error message for a given connection.

Arguments

connname

Name of the connection to use.

Return Value

Returns last error message, or OK if there has been no error in this connection.

Notes

When asynchronous queries are initiated by dblink_send_query, the error message associated with the connection might not get updated until the server's response message is consumed. This typically means that dblink_is_busy or dblink_get_result should be called prior to dblink_error_message, so that any error generated by the asynchronous query will be visible.

Examples

SELECT dblink_error_message('dtest1');

Prev | Up | Next
dblink_get_connections | Home | dblink_send_query


dblink_send_query
Prev Up F.10. dblink Home Next

dblink_send_query

dblink_send_query — sends an async query to a remote database

Synopsis

dblink_send_query(text connname, text sql) returns int

Description

dblink_send_query sends a query to be executed asynchronously, that is, without immediately waiting for the result. There must not be an async query already in progress on the connection.

After successfully dispatching an async query, completion status can be checked with dblink_is_busy, and the results are ultimately collected with dblink_get_result. It is also possible to attempt to cancel an active async query using dblink_cancel_query.

Arguments

connname

Name of the connection to use.

sql

The SQL statement that you wish to execute in the remote database, for example select * from pg_class.

Return Value

Returns 1 if the query was successfully dispatched, 0 otherwise.

Examples

SELECT dblink_send_query('dtest1', 'SELECT * FROM foo WHERE f1 < 3');

Prev | Up | Next
dblink_error_message | Home | dblink_is_busy


dblink_is_busy
Prev Up F.10. dblink Home Next

dblink_is_busy

dblink_is_busy — checks if connection is busy with an async query

Synopsis

dblink_is_busy(text connname) returns int

Description

dblink_is_busy tests whether an async query is in progress.

Arguments

connname

Name of the connection to check.

Return Value

Returns 1 if connection is busy, 0 if it is not busy. If this function returns 0, it is guaranteed that dblink_get_result will not block.

Examples

SELECT dblink_is_busy('dtest1');

Prev | Up | Next
dblink_send_query | Home | dblink_get_notify


dblink_get_notify
Prev Up F.10. dblink Home Next

dblink_get_notify

dblink_get_notify — retrieve async notifications on a connection

Synopsis

dblink_get_notify() returns setof (notify_name text, be_pid int, extra text)
dblink_get_notify(text connname) returns setof (notify_name text, be_pid int, extra text)

Description

dblink_get_notify retrieves notifications on either the unnamed connection, or on a named connection if specified. To receive notifications via dblink, LISTEN must first be issued, using dblink_exec. For details see LISTEN and NOTIFY.

Arguments

connname

The name of a named connection to get notifications on.

Return Value

Returns setof (notify_name text, be_pid int, extra text), or an empty set if none.

Examples

SELECT dblink_exec('LISTEN virtual');
 dblink_exec
-------------
 LISTEN
(1 row)

SELECT * FROM dblink_get_notify();
 notify_name | be_pid | extra
-------------+--------+-------
(0 rows)

NOTIFY virtual;
NOTIFY

SELECT * FROM dblink_get_notify();
 notify_name | be_pid | extra
-------------+--------+-------
 virtual     |   1229 |
(1 row)

Prev | Up | Next
dblink_is_busy | Home | dblink_get_result


dblink_get_result
Prev Up F.10. dblink Home Next

dblink_get_result

dblink_get_result — gets an async query result

Synopsis

dblink_get_result(text connname [, bool fail_on_error]) returns setof record

Description

dblink_get_result collects the results of an asynchronous query previously sent with dblink_send_query. If the query is not already completed, dblink_get_result will wait until it is.

Arguments

connname

Name of the connection to use.

fail_on_error

If true (the default when omitted) then an error thrown on the remote side of the connection causes an error to also be thrown locally. If false, the remote error is locally reported as a NOTICE, and the function returns no rows.

Return Value

For an async query (that is, an SQL statement returning rows), the function returns the row(s) produced by the query. To use this function, you will need to specify the expected set of columns, as previously discussed for dblink.

For an async command (that is, an SQL statement not returning rows), the function returns a single row with a single text column containing the command's status string. It is still necessary to specify that the result will have a single text column in the calling FROM clause.

Notes

This function must be called if dblink_send_query returned 1. It must be called once for each query sent, and one additional time to obtain an empty set result, before the connection can be used again.

When using dblink_send_query and dblink_get_result, dblink fetches the entire remote query result before returning any of it to the local query processor. If the query returns a large number of rows, this can result in transient memory bloat in the local session. It may be better to open such a query as a cursor with dblink_open and then fetch a manageable number of rows at a time. Alternatively, use plain dblink(), which avoids memory bloat by spooling large result sets to disk.

Examples

contrib_regression=# SELECT dblink_connect('dtest1', 'dbname=contrib_regression');
 dblink_connect
----------------
 OK
(1 row)

contrib_regression=# SELECT * FROM
contrib_regression-# dblink_send_query('dtest1', 'select * from foo where f1 < 3') AS t1;
 t1
----
  1
(1 row)

contrib_regression=# SELECT * FROM dblink_get_result('dtest1') AS t1(f1 int, f2 text, f3 text[]);
 f1 | f2 |     f3
----+----+------------
  0 | a  | {a0,b0,c0}
  1 | b  | {a1,b1,c1}
  2 | c  | {a2,b2,c2}
(3 rows)

contrib_regression=# SELECT * FROM dblink_get_result('dtest1') AS t1(f1 int, f2 text, f3 text[]);
 f1 | f2 | f3
----+----+----
(0 rows)

contrib_regression=# SELECT * FROM
contrib_regression-# dblink_send_query('dtest1', 'select * from foo where f1 < 3; select * from foo where f1 > 6') AS t1;
 t1
----
  1
(1 row)

contrib_regression=# SELECT * FROM dblink_get_result('dtest1') AS t1(f1 int, f2 text, f3 text[]);
 f1 | f2 |     f3
----+----+------------
  0 | a  | {a0,b0,c0}
  1 | b  | {a1,b1,c1}
  2 | c  | {a2,b2,c2}
(3 rows)

contrib_regression=# SELECT * FROM dblink_get_result('dtest1') AS t1(f1 int, f2 text, f3 text[]);
 f1 | f2 |      f3
----+----+---------------
  7 | h  | {a7,b7,c7}
  8 | i  | {a8,b8,c8}
  9 | j  | {a9,b9,c9}
 10 | k  | {a10,b10,c10}
(4 rows)

contrib_regression=# SELECT * FROM dblink_get_result('dtest1') AS t1(f1 int, f2 text, f3 text[]);
 f1 | f2 | f3
----+----+----
(0 rows)

Prev | Up | Next
dblink_get_notify | Home | dblink_cancel_query


dblink_cancel_query
Prev Up F.10. dblink Home Next

dblink_cancel_query

dblink_cancel_query — cancels any active query on the named connection

Synopsis

dblink_cancel_query(text connname) returns text

Description

dblink_cancel_query attempts to cancel any query that is in progress on the named connection. Note that this is not certain to succeed (since, for example, the remote query might already have finished). A cancel request simply improves the odds that the query will fail soon. You must still complete the normal query protocol, for example by calling dblink_get_result.

Arguments

connname

Name of the connection to use.

Return Value

Returns OK if the cancel request has been sent, or the text of an error message on failure.

Examples

SELECT dblink_cancel_query('dtest1');

Prev | Up | Next
dblink_get_result | Home | dblink_get_pkey


dblink_get_pkey
Prev Up F.10. dblink Home Next

dblink_get_pkey

dblink_get_pkey — returns the positions and field names of a relation's primary key fields

Synopsis

dblink_get_pkey(text relname) returns setof dblink_pkey_results

Description

dblink_get_pkey provides information about the primary key of a relation in the local database. This is sometimes useful in generating queries to be sent to remote databases.

Arguments

relname

Name of a local relation, for example foo or myschema.mytab. Include double quotes if the name is mixed-case or contains special characters, for example "FooBar"; without quotes, the string will be folded to lower case.

Return Value

Returns one row for each primary key field, or no rows if the relation has no primary key. The result row type is defined as

CREATE TYPE dblink_pkey_results AS (position int, colname text);

The position column simply runs from 1 to N; it is the number of the field within the primary key, not the number within the table's columns.

Examples

CREATE TABLE foobar (
    f1 int,
    f2 int,
    f3 int,
    PRIMARY KEY (f1, f2, f3)
);
CREATE TABLE

SELECT * FROM dblink_get_pkey('foobar');
 position | colname
----------+---------
        1 | f1
        2 | f2
        3 | f3
(3 rows)

Prev | Up | Next
dblink_cancel_query | Home | dblink_build_sql_insert


dblink_build_sql_insert
Prev Up F.10. dblink Home Next

dblink_build_sql_insert

dblink_build_sql_insert — builds an INSERT statement using a local tuple, replacing the primary key field values with alternative supplied values

Synopsis

dblink_build_sql_insert(text relname,
                        int2vector primary_key_attnums,
                        integer num_primary_key_atts,
                        text[] src_pk_att_vals_array,
                        text[] tgt_pk_att_vals_array) returns text

Description

dblink_build_sql_insert can be useful in doing selective replication of a local table to a remote database. It selects a row from the local table based on primary key, and then builds an SQL INSERT command that will duplicate that row, but with the primary key values replaced by the values in the last argument. (To make an exact copy of the row, just specify the same values for the last two arguments.)

Arguments

relname

Name of a local relation, for example foo or myschema.mytab. Include double quotes if the name is mixed-case or contains special characters, for example "FooBar"; without quotes, the string will be folded to lower case.

primary_key_attnums

Attribute numbers (1-based) of the primary key fields, for example 1 2.

num_primary_key_atts

The number of primary key fields.

src_pk_att_vals_array

Values of the primary key fields to be used to look up the local tuple. Each field is represented in text form. An error is thrown if there is no local row with these primary key values.

tgt_pk_att_vals_array

Values of the primary key fields to be placed in the resulting INSERT command. Each field is represented in text form.

Return Value

Returns the requested SQL statement as text.

Notes

As of PostgreSQL 9.0, the attribute numbers in primary_key_attnums are interpreted as logical column numbers, corresponding to the column's position in SELECT * FROM relname. Previous versions interpreted the numbers as physical column positions. There is a difference if any column(s) to the left of the indicated column have been dropped during the lifetime of the table.

Examples

SELECT dblink_build_sql_insert('foo', '1 2', 2, '{"1", "a"}', '{"1", "b''a"}');
             dblink_build_sql_insert
--------------------------------------------------
 INSERT INTO foo(f1,f2,f3) VALUES('1','b''a','1')
(1 row)

Prev | Up | Next
dblink_get_pkey | Home | dblink_build_sql_delete


dblink_build_sql_delete
Prev Up F.10. dblink Home Next

dblink_build_sql_delete

dblink_build_sql_delete — builds a DELETE statement using supplied values for primary key field values

Synopsis

dblink_build_sql_delete(text relname,
                        int2vector primary_key_attnums,
                        integer num_primary_key_atts,
                        text[] tgt_pk_att_vals_array) returns text

Description

dblink_build_sql_delete can be useful in doing selective replication of a local table to a remote database. It builds an SQL DELETE command that will delete the row with the given primary key values.

Arguments

relname

Name of a local relation, for example foo or myschema.mytab. Include double quotes if the name is mixed-case or contains special characters, for example "FooBar"; without quotes, the string will be folded to lower case.

primary_key_attnums

Attribute numbers (1-based) of the primary key fields, for example 1 2.

num_primary_key_atts

The number of primary key fields.

tgt_pk_att_vals_array

Values of the primary key fields to be used in the resulting DELETE command. Each field is represented in text form.

Return Value

Returns the requested SQL statement as text.

Notes

As of PostgreSQL 9.0, the attribute numbers in primary_key_attnums are interpreted as logical column numbers, corresponding to the column's position in SELECT * FROM relname. Previous versions interpreted the numbers as physical column positions. There is a difference if any column(s) to the left of the indicated column have been dropped during the lifetime of the table.

Examples

SELECT dblink_build_sql_delete('"MyFoo"', '1 2', 2, '{"1", "b"}');
           dblink_build_sql_delete
---------------------------------------------
 DELETE FROM "MyFoo" WHERE f1='1' AND f2='b'
(1 row)

Prev | Up | Next
dblink_build_sql_insert | Home | dblink_build_sql_update


dblink_build_sql_update
Prev Up F.10. dblink Home Next

dblink_build_sql_update

dblink_build_sql_update — builds an UPDATE statement using a local tuple, replacing the primary key field values with alternative supplied values

Synopsis

dblink_build_sql_update(text relname,
                        int2vector primary_key_attnums,
                        integer num_primary_key_atts,
                        text[] src_pk_att_vals_array,
                        text[] tgt_pk_att_vals_array) returns text

Description

dblink_build_sql_update can be useful in doing selective replication of a local table to a remote database. It selects a row from the local table based on primary key, and then builds an SQL UPDATE command that will duplicate that row, but with the primary key values replaced by the values in the last argument. (To make an exact copy of the row, just specify the same values for the last two arguments.) The UPDATE command always assigns all fields of the row — the main difference between this and dblink_build_sql_insert is that it's assumed that the target row already exists in the remote table.

Arguments

relname

Name of a local relation, for example foo or myschema.mytab. Include double quotes if the name is mixed-case or contains special characters, for example "FooBar"; without quotes, the string will be folded to lower case.

primary_key_attnums

Attribute numbers (1-based) of the primary key fields, for example 1 2.

num_primary_key_atts

The number of primary key fields.

src_pk_att_vals_array

Values of the primary key fields to be used to look up the local tuple. Each field is represented in text form. An error is thrown if there is no local row with these primary key values.

tgt_pk_att_vals_array

Values of the primary key fields to be placed in the resulting UPDATE command. Each field is represented in text form.

Return Value

Returns the requested SQL statement as text.

Notes

As of PostgreSQL 9.0, the attribute numbers in primary_key_attnums are interpreted as logical column numbers, corresponding to the column's position in SELECT * FROM relname. Previous versions interpreted the numbers as physical column positions. There is a difference if any column(s) to the left of the indicated column have been dropped during the lifetime of the table.

Examples

SELECT dblink_build_sql_update('foo', '1 2', 2, '{"1", "a"}', '{"1", "b"}');
                   dblink_build_sql_update
-------------------------------------------------------------
 UPDATE foo SET f1='1',f2='b',f3='1' WHERE f1='1' AND f2='b'
(1 row)

Prev | Up | Next
dblink_build_sql_delete | Home | F.11. dict_int


F.11. dict_int
Prev Up Appendix F. Additional Supplied Modules Home Next

F.11. dict_int

F.11.1. Configuration
F.11.2. Usage

dict_int is an example of an add-on dictionary template for full-text search. The motivation for this example dictionary is to control the indexing of integers (signed and unsigned), allowing such numbers to be indexed while preventing excessive growth in the number of unique words, which greatly affects the performance of searching.

This module is considered “trusted”, that is, it can be installed by non-superusers who have CREATE privilege on the current database.

F.11.1. Configuration

The dictionary accepts three options:

  • The maxlen parameter specifies the maximum number of digits allowed in an integer word. The default value is 6.

  • The rejectlong parameter specifies whether an overlength integer should be truncated or ignored. If rejectlong is false (the default), the dictionary returns the first maxlen digits of the integer. If rejectlong is true, the dictionary treats an overlength integer as a stop word, so that it will not be indexed. Note that this also means that such an integer cannot be searched for.

  • The absval parameter specifies whether leading “+” or “-” signs should be removed from integer words. The default is false. When true, the sign is removed before maxlen is applied.

F.11.2. Usage

Installing the dict_int extension creates a text search template intdict_template and a dictionary intdict based on it, with the default parameters. You can alter the parameters, for example

mydb# ALTER TEXT SEARCH DICTIONARY intdict (MAXLEN = 4, REJECTLONG = true);
ALTER TEXT SEARCH DICTIONARY

or create new dictionaries based on the template.

To test the dictionary, you can try

mydb# select ts_lexize('intdict', '12345678');
 ts_lexize
-----------
 {123456}

but real-world usage will involve including it in a text search configuration as described in Chapter 12. That might look like this:

ALTER TEXT SEARCH CONFIGURATION english
    ALTER MAPPING FOR int, uint WITH intdict;

Prev | Up | Next
dblink_build_sql_update | Home | F.12. dict_xsyn


F.12. dict_xsyn
Prev Up Appendix F. Additional Supplied Modules Home Next

F.12. dict_xsyn

F.12.1. Configuration
F.12.2. Usage

dict_xsyn (Extended Synonym Dictionary) is an example of an add-on dictionary template for full-text search. This dictionary type replaces words with groups of their synonyms, and so makes it possible to search for a word using any of its synonyms.

F.12.1. Configuration

A dict_xsyn dictionary accepts the following options:

  • matchorig controls whether the original word is accepted by the dictionary. Default is true.

  • matchsynonyms controls whether the synonyms are accepted by the dictionary. Default is false.

  • keeporig controls whether the original word is included in the dictionary's output. Default is true.

  • keepsynonyms controls whether the synonyms are included in the dictionary's output. Default is true.

  • rules is the base name of the file containing the list of synonyms. This file must be stored in $SHAREDIR/tsearch_data/ (where $SHAREDIR means the PostgreSQL installation's shared-data directory). Its name must end in .rules (which is not to be included in the rules parameter).

The rules file has the following format:

  • Each line represents a group of synonyms for a single word, which is given first on the line. Synonyms are separated by whitespace, thus:

    word syn1 syn2 syn3

  • The sharp (#) sign is a comment delimiter. It may appear at any position in a line. The rest of the line will be skipped.

Look at xsyn_sample.rules, which is installed in $SHAREDIR/tsearch_data/, for an example.

F.12.2. Usage

Installing the dict_xsyn extension creates a text search template xsyn_template and a dictionary xsyn based on it, with default parameters. You can alter the parameters, for example

mydb# ALTER TEXT SEARCH DICTIONARY xsyn (RULES='my_rules', KEEPORIG=false);
ALTER TEXT SEARCH DICTIONARY

or create new dictionaries based on the template.

To test the dictionary, you can try

mydb=# SELECT ts_lexize('xsyn', 'word');
      ts_lexize
-----------------------
 {syn1,syn2,syn3}

mydb# ALTER TEXT SEARCH DICTIONARY xsyn (RULES='my_rules', KEEPORIG=true);
ALTER TEXT SEARCH DICTIONARY

mydb=# SELECT ts_lexize('xsyn', 'word');
      ts_lexize
-----------------------
 {word,syn1,syn2,syn3}

mydb# ALTER TEXT SEARCH DICTIONARY xsyn (RULES='my_rules', KEEPORIG=false, MATCHSYNONYMS=true);
ALTER TEXT SEARCH DICTIONARY

mydb=# SELECT ts_lexize('xsyn', 'syn1');
      ts_lexize
-----------------------
 {syn1,syn2,syn3}

mydb# ALTER TEXT SEARCH DICTIONARY xsyn (RULES='my_rules', KEEPORIG=true, MATCHORIG=false, KEEPSYNONYMS=false);
ALTER TEXT SEARCH DICTIONARY

mydb=# SELECT ts_lexize('xsyn', 'syn1');
      ts_lexize
-----------------------
 {word}

Real-world usage will involve including it in a text search configuration as described in Chapter 12. That might look like this:

ALTER TEXT SEARCH CONFIGURATION english
    ALTER MAPPING FOR word, asciiword WITH xsyn, english_stem;

Prev | Up | Next
F.11. dict_int | Home | F.13. earthdistance


F.13. earthdistance
Prev Up Appendix F. Additional Supplied Modules Home Next

F.13. earthdistance

F.13.1. Cube-Based Earth Distances
F.13.2. Point-Based Earth Distances

The earthdistance module provides two different approaches to calculating great circle distances on the surface of the Earth. The one described first depends on the cube module. The second one is based on the built-in point data type, using longitude and latitude for the coordinates.

In this module, the Earth is assumed to be perfectly spherical. (If that's too inaccurate for you, you might want to look at the PostGIS project.)

The cube module must be installed before earthdistance can be installed (although you can use the CASCADE option of CREATE EXTENSION to install both in one command).

Caution

It is strongly recommended that earthdistance and cube be installed in the same schema, and that that schema be one for which CREATE privilege has not been and will not be granted to any untrusted users. Otherwise there are installation-time security hazards if earthdistance's schema contains objects defined by a hostile user. Furthermore, when using earthdistance's functions after installation, the entire search path should contain only trusted schemas.

F.13.1. Cube-Based Earth Distances

Data is stored in cubes that are points (both corners are the same) using 3 coordinates representing the x, y, and z distance from the center of the Earth. A domain earth over cube is provided, which includes constraint checks that the value meets these restrictions and is reasonably close to the actual surface of the Earth.

The radius of the Earth is obtained from the earth() function. It is given in meters. But by changing this one function you can change the module to use some other units, or to use a different value of the radius that you feel is more appropriate.

This package has applications to astronomical databases as well. Astronomers will probably want to change earth() to return a radius of 180/pi() so that distances are in degrees.

Functions are provided to support input in latitude and longitude (in degrees), to support output of latitude and longitude, to calculate the great circle distance between two points and to easily specify a bounding box usable for index searches.

The provided functions are shown in Table F.5.

Table F.5. Cube-Based Earthdistance Functions

Function

Description

earth () → float8

Returns the assumed radius of the Earth.

sec_to_gc ( float8 ) → float8

Converts the normal straight line (secant) distance between two points on the surface of the Earth to the great circle distance between them.

gc_to_sec ( float8 ) → float8

Converts the great circle distance between two points on the surface of the Earth to the normal straight line (secant) distance between them.

ll_to_earth ( float8, float8 ) → earth

Returns the location of a point on the surface of the Earth given its latitude (argument 1) and longitude (argument 2) in degrees.

latitude ( earth ) → float8

Returns the latitude in degrees of a point on the surface of the Earth.

longitude ( earth ) → float8

Returns the longitude in degrees of a point on the surface of the Earth.

earth_distance ( earth, earth ) → float8

Returns the great circle distance between two points on the surface of the Earth.

earth_box ( earth, float8 ) → cube

Returns a box suitable for an indexed search using the cube @> operator for points within a given great circle distance of a location. Some points in this box are further than the specified great circle distance from the location, so a second check using earth_distance should be included in the query.

F.13.2. Point-Based Earth Distances

The second part of the module relies on representing Earth locations as values of type point, in which the first component is taken to represent longitude in degrees, and the second component is taken to represent latitude in degrees. Points are taken as (longitude, latitude) and not vice versa because longitude is closer to the intuitive idea of x-axis and latitude to y-axis.

A single operator is provided, shown in Table F.6.

Table F.6. Point-Based Earthdistance Operators

Operator

Description

point <@> point → float8

Computes the distance in statute miles between two points on the Earth's surface.

Note that unlike the cube-based part of the module, units are hardwired here: changing the earth() function will not affect the results of this operator.

One disadvantage of the longitude/latitude representation is that you need to be careful about the edge conditions near the poles and near +/- 180 degrees of longitude. The cube-based representation avoids these discontinuities.

Prev | Up | Next
F.12. dict_xsyn | Home | F.14. file_fdw


F.14. file_fdw
Prev Up Appendix F. Additional Supplied Modules Home Next

F.14. file_fdw

The file_fdw module provides the foreign-data wrapper file_fdw, which can be used to access data files in the server's file system, or to execute programs on the server and read their output. The data file or program output must be in a format that can be read by COPY FROM; see COPY for details. Access to data files is currently read-only.

A foreign table created using this wrapper can have the following options:

filename

Specifies the file to be read. Relative paths are relative to the data directory. Either filename or program must be specified, but not both.

program

Specifies the command to be executed. The standard output of this command will be read as though COPY FROM PROGRAM were used. Either program or filename must be specified, but not both.

format

Specifies the data format, the same as COPY's FORMAT option.

header

Specifies whether the data has a header line, the same as COPY's HEADER option.

delimiter

Specifies the data delimiter character, the same as COPY's DELIMITER option.

quote

Specifies the data quote character, the same as COPY's QUOTE option.

escape

Specifies the data escape character, the same as COPY's ESCAPE option.

null

Specifies the data null string, the same as COPY's NULL option.

encoding

Specifies the data encoding, the same as COPY's ENCODING option.

Note that while COPY allows options such as HEADER to be specified without a corresponding value, the foreign table option syntax requires a value to be present in all cases. To activate COPY options typically written without a value, you can pass the value TRUE, since all such options are Booleans.

A column of a foreign table created using this wrapper can have the following options:

force_not_null

This is a Boolean option. If true, it specifies that values of the column should not be matched against the null string (that is, the table-level null option). This has the same effect as listing the column in COPY's FORCE_NOT_NULL option.

force_null

This is a Boolean option. If true, it specifies that values of the column which match the null string are returned as NULL even if the value is quoted. Without this option, only unquoted values matching the null string are returned as NULL. This has the same effect as listing the column in COPY's FORCE_NULL option.

COPY's FORCE_QUOTE option is currently not supported by file_fdw.

These options can only be specified for a foreign table or its columns, not in the options of the file_fdw foreign-data wrapper, nor in the options of a server or user mapping using the wrapper.

Changing table-level options requires being a superuser or having the privileges of the role pg_read_server_files (to use a filename) or the role pg_execute_server_program (to use a program), for security reasons: only certain users should be able to control which file is read or which program is run. In principle regular users could be allowed to change the other options, but that's not supported at present.

When specifying the program option, keep in mind that the option string is executed by the shell. If you need to pass any arguments to the command that come from an untrusted source, you must be careful to strip or escape any characters that might have special meaning to the shell. For security reasons, it is best to use a fixed command string, or at least avoid passing any user input in it.

For a foreign table using file_fdw, EXPLAIN shows the name of the file to be read or program to be run. For a file, unless COSTS OFF is specified, the file size (in bytes) is shown as well.

Example F.1. Create a Foreign Table for PostgreSQL CSV Logs

One of the obvious uses for file_fdw is to make the PostgreSQL activity log available as a table for querying. To do this, first you must be logging to a CSV file, which here we will call pglog.csv. First, install file_fdw as an extension:

CREATE EXTENSION file_fdw;

Then create a foreign server:

CREATE SERVER pglog FOREIGN DATA WRAPPER file_fdw;

Now you are ready to create the foreign data table. Using the CREATE FOREIGN TABLE command, you will need to define the columns for the table, the CSV file name, and its format:

CREATE FOREIGN TABLE pglog (
  log_time timestamp(3) with time zone,
  user_name text,
  database_name text,
  process_id integer,
  connection_from text,
  session_id text,
  session_line_num bigint,
  command_tag text,
  session_start_time timestamp with time zone,
  virtual_transaction_id text,
  transaction_id bigint,
  error_severity text,
  sql_state_code text,
  message text,
  detail text,
  hint text,
  internal_query text,
  internal_query_pos integer,
  context text,
  query text,
  query_pos integer,
  location text,
  application_name text,
  backend_type text,
  leader_pid integer,
  query_id bigint
) SERVER pglog
OPTIONS ( filename 'log/pglog.csv', format 'csv' );

That's it — now you can query your log directly. In production, of course, you would need to define some way to deal with log rotation.

Prev | Up | Next
F.13. earthdistance | Home | F.15. fuzzystrmatch


F.15. fuzzystrmatch
Prev Up Appendix F. Additional Supplied Modules Home Next

F.15. fuzzystrmatch

F.15.1. Soundex
F.15.2. Levenshtein
F.15.3. Metaphone
F.15.4. Double Metaphone

The fuzzystrmatch module provides several functions to determine similarities and distance between strings.

Caution

At present, the soundex, metaphone, dmetaphone, and dmetaphone_alt functions do not work well with multibyte encodings (such as UTF-8).

This module is considered “trusted”, that is, it can be installed by non-superusers who have CREATE privilege on the current database.

F.15.1. Soundex

The Soundex system is a method of matching similar-sounding names by converting them to the same code. It was initially used by the United States Census in 1880, 1900, and 1910. Note that Soundex is not very useful for non-English names.

The fuzzystrmatch module provides two functions for working with Soundex codes:

soundex(text) returns text
difference(text, text) returns int

The soundex function converts a string to its Soundex code. The difference function converts two strings to their Soundex codes and then reports the number of matching code positions. Since Soundex codes have four characters, the result ranges from zero to four, with zero being no match and four being an exact match. (Thus, the function is misnamed — similarity would have been a better name.)

Here are some usage examples:

SELECT soundex('hello world!');

SELECT soundex('Anne'), soundex('Ann'), difference('Anne', 'Ann');
SELECT soundex('Anne'), soundex('Andrew'), difference('Anne', 'Andrew');
SELECT soundex('Anne'), soundex('Margaret'), difference('Anne', 'Margaret');

CREATE TABLE s (nm text);

INSERT INTO s VALUES ('john');
INSERT INTO s VALUES ('joan');
INSERT INTO s VALUES ('wobbly');
INSERT INTO s VALUES ('jack');

SELECT * FROM s WHERE soundex(nm) = soundex('john');

SELECT * FROM s WHERE difference(s.nm, 'john') > 2;

F.15.2. Levenshtein

This function calculates the Levenshtein distance between two strings:

levenshtein(text source, text target, int ins_cost, int del_cost, int sub_cost) returns int
levenshtein(text source, text target) returns int
levenshtein_less_equal(text source, text target, int ins_cost, int del_cost, int sub_cost, int max_d) returns int
levenshtein_less_equal(text source, text target, int max_d) returns int

Both source and target can be any non-null string, with a maximum of 255 characters. The cost parameters specify how much to charge for a character insertion, deletion, or substitution, respectively. You can omit the cost parameters, as in the second version of the function; in that case they all default to 1.

levenshtein_less_equal is an accelerated version of the Levenshtein function for use when only small distances are of interest. If the actual distance is less than or equal to max_d, then levenshtein_less_equal returns the correct distance; otherwise it returns some value greater than max_d. If max_d is negative then the behavior is the same as levenshtein.

Examples:

test=# SELECT levenshtein('GUMBO', 'GAMBOL');
 levenshtein
-------------
           2
(1 row)

test=# SELECT levenshtein('GUMBO', 'GAMBOL', 2, 1, 1);
 levenshtein
-------------
           3
(1 row)

test=# SELECT levenshtein_less_equal('extensive', 'exhaustive', 2);
 levenshtein_less_equal
------------------------
                      3
(1 row)

test=# SELECT levenshtein_less_equal('extensive', 'exhaustive', 4);
 levenshtein_less_equal
------------------------
                      4
(1 row)

F.15.3. Metaphone

Metaphone, like Soundex, is based on the idea of constructing a representative code for an input string. Two strings are then deemed similar if they have the same codes.

This function calculates the metaphone code of an input string:

metaphone(text source, int max_output_length) returns text

source has to be a non-null string with a maximum of 255 characters. max_output_length sets the maximum length of the output metaphone code; if longer, the output is truncated to this length.

Example:

test=# SELECT metaphone('GUMBO', 4);
 metaphone
-----------
 KM
(1 row)

F.15.4. Double Metaphone

The Double Metaphone system computes two “sounds like” strings for a given input string — a “primary” and an “alternate”. In most cases they are the same, but for non-English names especially they can be a bit different, depending on pronunciation. These functions compute the primary and alternate codes:

dmetaphone(text source) returns text
dmetaphone_alt(text source) returns text

There is no length limit on the input strings.

Example:

test=# SELECT dmetaphone('gumbo');
 dmetaphone
------------
 KMP
(1 row)

Prev | Up | Next
F.14. file_fdw | Home | F.16. hstore


F.16. hstore
Prev Up Appendix F. Additional Supplied Modules Home Next

F.16. hstore

F.16.1. hstore External Representation
F.16.2. hstore Operators and Functions
F.16.3. Indexes
F.16.4. Examples
F.16.5. Statistics
F.16.6. Compatibility
F.16.7. Transforms
F.16.8. Authors

This module implements the hstore data type for storing sets of key/value pairs within a single PostgreSQL value. This can be useful in various scenarios, such as rows with many attributes that are rarely examined, or semi-structured data. Keys and values are simply text strings.

This module is considered “trusted”, that is, it can be installed by non-superusers who have CREATE privilege on the current database.

F.16.1. hstore External Representation

The text representation of an hstore, used for input and output, includes zero or more key => value pairs separated by commas. Some examples:

k => v
foo => bar, baz => whatever
"1-a" => "anything at all"

The order of the pairs is not significant (and may not be reproduced on output). Whitespace between pairs or around the => sign is ignored. Double-quote keys and values that include whitespace, commas, =s or >s. To include a double quote or a backslash in a key or value, escape it with a backslash.

Each key in an hstore is unique. If you declare an hstore with duplicate keys, only one will be stored in the hstore and there is no guarantee as to which will be kept:

SELECT 'a=>1,a=>2'::hstore;
  hstore
----------
 "a"=>"1"

A value (but not a key) can be an SQL NULL. For example:

key => NULL

The NULL keyword is case-insensitive. Double-quote the NULL to treat it as the ordinary string “NULL”.

Note

Keep in mind that the hstore text format, when used for input, applies before any required quoting or escaping. If you are passing an hstore literal via a parameter, then no additional processing is needed. But if you're passing it as a quoted literal constant, then any single-quote characters and (depending on the setting of the standard_conforming_strings configuration parameter) backslash characters need to be escaped correctly. See Section 4.1.2.1 for more on the handling of string constants.

On output, double quotes always surround keys and values, even when it's not strictly necessary.

F.16.2. hstore Operators and Functions

The operators provided by the hstore module are shown in Table F.7, the functions in Table F.8.

Table F.7. hstore Operators

Operator

Description

Example(s)

hstore -> text → text

Returns value associated with given key, or NULL if not present.

'a=>x, b=>y'::hstore -> 'a' → x

hstore -> text[] → text[]

Returns values associated with given keys, or NULL if not present.

'a=>x, b=>y, c=>z'::hstore -> ARRAY['c','a'] → {"z","x"}

hstore || hstore → hstore

Concatenates two hstores.

'a=>b, c=>d'::hstore || 'c=>x, d=>q'::hstore → "a"=>"b", "c"=>"x", "d"=>"q"

hstore ? text → boolean

Does hstore contain key?

'a=>1'::hstore ? 'a' → t

hstore ?& text[] → boolean

Does hstore contain all the specified keys?

'a=>1,b=>2'::hstore ?& ARRAY['a','b'] → t

hstore ?| text[] → boolean

Does hstore contain any of the specified keys?

'a=>1,b=>2'::hstore ?| ARRAY['b','c'] → t

hstore @> hstore → boolean

Does left operand contain right?

'a=>b, b=>1, c=>NULL'::hstore @> 'b=>1' → t

hstore <@ hstore → boolean

Is left operand contained in right?

'a=>c'::hstore <@ 'a=>b, b=>1, c=>NULL' → f

hstore - text → hstore

Deletes key from left operand.

'a=>1, b=>2, c=>3'::hstore - 'b'::text → "a"=>"1", "c"=>"3"

hstore - text[] → hstore

Deletes keys from left operand.

'a=>1, b=>2, c=>3'::hstore - ARRAY['a','b'] → "c"=>"3"

hstore - hstore → hstore

Deletes pairs from left operand that match pairs in the right operand.

'a=>1, b=>2, c=>3'::hstore - 'a=>4, b=>2'::hstore → "a"=>"1", "c"=>"3"

anyelement #= hstore → anyelement

Replaces fields in the left operand (which must be a composite type) with matching values from hstore.

ROW(1,3) #= 'f1=>11'::hstore → (11,3)

%% hstore → text[]

Converts hstore to an array of alternating keys and values.

%% 'a=>foo, b=>bar'::hstore → {a,foo,b,bar}

%# hstore → text[]

Converts hstore to a two-dimensional key/value array.

%# 'a=>foo, b=>bar'::hstore → {{a,foo},{b,bar}}

Table F.8. hstore Functions

Function

Description

Example(s)

hstore ( record ) → hstore

Constructs an hstore from a record or row.

hstore(ROW(1,2)) → "f1"=>"1", "f2"=>"2"

hstore ( text[] ) → hstore

Constructs an hstore from an array, which may be either a key/value array, or a two-dimensional array.

hstore(ARRAY['a','1','b','2']) → "a"=>"1", "b"=>"2"

hstore(ARRAY[['c','3'],['d','4']]) → "c"=>"3", "d"=>"4"

hstore ( text[], text[] ) → hstore

Constructs an hstore from separate key and value arrays.

hstore(ARRAY['a','b'], ARRAY['1','2']) → "a"=>"1", "b"=>"2"

hstore ( text, text ) → hstore

Makes a single-item hstore.

hstore('a', 'b') → "a"=>"b"

akeys ( hstore ) → text[]

Extracts an hstore's keys as an array.

akeys('a=>1,b=>2') → {a,b}

skeys ( hstore ) → setof text

Extracts an hstore's keys as a set.

skeys('a=>1,b=>2') →

a
b

avals ( hstore ) → text[]

Extracts an hstore's values as an array.

avals('a=>1,b=>2') → {1,2}

svals ( hstore ) → setof text

Extracts an hstore's values as a set.

svals('a=>1,b=>2') →

1
2

hstore_to_array ( hstore ) → text[]

Extracts an hstore's keys and values as an array of alternating keys and values.

hstore_to_array('a=>1,b=>2') → {a,1,b,2}

hstore_to_matrix ( hstore ) → text[]

Extracts an hstore's keys and values as a two-dimensional array.

hstore_to_matrix('a=>1,b=>2') → {{a,1},{b,2}}

hstore_to_json ( hstore ) → json

Converts an hstore to a json value, converting all non-null values to JSON strings.

This function is used implicitly when an hstore value is cast to json.

hstore_to_json('"a key"=>1, b=>t, c=>null, d=>12345, e=>012345, f=>1.234, g=>2.345e+4') → {"a key": "1", "b": "t", "c": null, "d": "12345", "e": "012345", "f": "1.234", "g": "2.345e+4"}

hstore_to_jsonb ( hstore ) → jsonb

Converts an hstore to a jsonb value, converting all non-null values to JSON strings.

This function is used implicitly when an hstore value is cast to jsonb.

hstore_to_jsonb('"a key"=>1, b=>t, c=>null, d=>12345, e=>012345, f=>1.234, g=>2.345e+4') → {"a key": "1", "b": "t", "c": null, "d": "12345", "e": "012345", "f": "1.234", "g": "2.345e+4"}

hstore_to_json_loose ( hstore ) → json

Converts an hstore to a json value, but attempts to distinguish numerical and Boolean values so they are unquoted in the JSON.

hstore_to_json_loose('"a key"=>1, b=>t, c=>null, d=>12345, e=>012345, f=>1.234, g=>2.345e+4') → {"a key": 1, "b": true, "c": null, "d": 12345, "e": "012345", "f": 1.234, "g": 2.345e+4}

hstore_to_jsonb_loose ( hstore ) → jsonb

Converts an hstore to a jsonb value, but attempts to distinguish numerical and Boolean values so they are unquoted in the JSON.

hstore_to_jsonb_loose('"a key"=>1, b=>t, c=>null, d=>12345, e=>012345, f=>1.234, g=>2.345e+4') → {"a key": 1, "b": true, "c": null, "d": 12345, "e": "012345", "f": 1.234, "g": 2.345e+4}

slice ( hstore, text[] ) → hstore

Extracts a subset of an hstore containing only the specified keys.

slice('a=>1,b=>2,c=>3'::hstore, ARRAY['b','c','x']) → "b"=>"2", "c"=>"3"

each ( hstore ) → setof record ( key text, value text )

Extracts an hstore's keys and values as a set of records.

select * from each('a=>1,b=>2') →

 key | value
-----+-------
 a   | 1
 b   | 2

exist ( hstore, text ) → boolean

Does hstore contain key?

exist('a=>1', 'a') → t

defined ( hstore, text ) → boolean

Does hstore contain a non-NULL value for key?

defined('a=>NULL', 'a') → f

delete ( hstore, text ) → hstore

Deletes pair with matching key.

delete('a=>1,b=>2', 'b') → "a"=>"1"

delete ( hstore, text[] ) → hstore

Deletes pairs with matching keys.

delete('a=>1,b=>2,c=>3', ARRAY['a','b']) → "c"=>"3"

delete ( hstore, hstore ) → hstore

Deletes pairs matching those in the second argument.

delete('a=>1,b=>2', 'a=>4,b=>2'::hstore) → "a"=>"1"

populate_record ( anyelement, hstore ) → anyelement

Replaces fields in the left operand (which must be a composite type) with matching values from hstore.

populate_record(ROW(1,2), 'f1=>42'::hstore) → (42,2)

In addition to these operators and functions, values of the hstore type can be subscripted, allowing them to act like associative arrays. Only a single subscript of type text can be specified; it is interpreted as a key and the corresponding value is fetched or stored. For example,

CREATE TABLE mytable (h hstore);
INSERT INTO mytable VALUES ('a=>b, c=>d');
SELECT h['a'] FROM mytable;
 h
---
 b
(1 row)

UPDATE mytable SET h['c'] = 'new';
SELECT h FROM mytable;
          h
----------------------
 "a"=>"b", "c"=>"new"
(1 row)

A subscripted fetch returns NULL if the subscript is NULL or that key does not exist in the hstore. (Thus, a subscripted fetch is not greatly different from the -> operator.) A subscripted update fails if the subscript is NULL; otherwise, it replaces the value for that key, adding an entry to the hstore if the key does not already exist.

F.16.3. Indexes

hstore has GiST and GIN index support for the @>, ?, ?& and ?| operators. For example:

CREATE INDEX hidx ON testhstore USING GIST (h);

CREATE INDEX hidx ON testhstore USING GIN (h);

gist_hstore_ops GiST opclass approximates a set of key/value pairs as a bitmap signature. Its optional integer parameter siglen determines the signature length in bytes. The default length is 16 bytes. Valid values of signature length are between 1 and 2024 bytes. Longer signatures lead to a more precise search (scanning a smaller fraction of the index and fewer heap pages), at the cost of a larger index.

Example of creating such an index with a signature length of 32 bytes:

CREATE INDEX hidx ON testhstore USING GIST (h gist_hstore_ops(siglen=32));

hstore also supports btree or hash indexes for the = operator. This allows hstore columns to be declared UNIQUE, or to be used in GROUP BY, ORDER BY or DISTINCT expressions. The sort ordering for hstore values is not particularly useful, but these indexes may be useful for equivalence lookups. Create indexes for = comparisons as follows:

CREATE INDEX hidx ON testhstore USING BTREE (h);

CREATE INDEX hidx ON testhstore USING HASH (h);

F.16.4. Examples

Add a key, or update an existing key with a new value:

UPDATE tab SET h['c'] = '3';

Another way to do the same thing is:

UPDATE tab SET h = h || hstore('c', '3');

If multiple keys are to be added or changed in one operation, the concatenation approach is more efficient than subscripting:

UPDATE tab SET h = h || hstore(array['q', 'w'], array['11', '12']);

Delete a key:

UPDATE tab SET h = delete(h, 'k1');

Convert a record to an hstore:

CREATE TABLE test (col1 integer, col2 text, col3 text);
INSERT INTO test VALUES (123, 'foo', 'bar');

SELECT hstore(t) FROM test AS t;
                   hstore
---------------------------------------------
 "col1"=>"123", "col2"=>"foo", "col3"=>"bar"
(1 row)

Convert an hstore to a predefined record type:

CREATE TABLE test (col1 integer, col2 text, col3 text);

SELECT * FROM populate_record(null::test,
                              '"col1"=>"456", "col2"=>"zzz"');
 col1 | col2 | col3
------+------+------
  456 | zzz  |
(1 row)

Modify an existing record using the values from an hstore:

CREATE TABLE test (col1 integer, col2 text, col3 text);
INSERT INTO test VALUES (123, 'foo', 'bar');

SELECT (r).* FROM (SELECT t #= '"col3"=>"baz"' AS r FROM test t) s;
 col1 | col2 | col3
------+------+------
  123 | foo  | baz
(1 row)

F.16.5. Statistics

The hstore type, because of its intrinsic liberality, could contain a lot of different keys. Checking for valid keys is the task of the application. The following examples demonstrate several techniques for checking keys and obtaining statistics.

Simple example:

SELECT * FROM each('aaa=>bq, b=>NULL, ""=>1');

Using a table:

CREATE TABLE stat AS SELECT (each(h)).key, (each(h)).value FROM testhstore;

Online statistics:

SELECT key, count(*) FROM
  (SELECT (each(h)).key FROM testhstore) AS stat
  GROUP BY key
  ORDER BY count DESC, key;
    key    | count
-----------+-------
 line      |   883
 query     |   207
 pos       |   203
 node      |   202
 space     |   197
 status    |   195
 public    |   194
 title     |   190
 org       |   189
...................

F.16.6. Compatibility

As of PostgreSQL 9.0, hstore uses a different internal representation than previous versions. This presents no obstacle for dump/restore upgrades since the text representation (used in the dump) is unchanged.

In the event of a binary upgrade, upward compatibility is maintained by having the new code recognize old-format data. This will entail a slight performance penalty when processing data that has not yet been modified by the new code. It is possible to force an upgrade of all values in a table column by doing an UPDATE statement as follows:

UPDATE tablename SET hstorecol = hstorecol || '';

Another way to do it is:

ALTER TABLE tablename ALTER hstorecol TYPE hstore USING hstorecol || '';

The ALTER TABLE method requires an ACCESS EXCLUSIVE lock on the table, but does not result in bloating the table with old row versions.

F.16.7. Transforms

Additional extensions are available that implement transforms for the hstore type for the languages PL/Perl and PL/Python. The extensions for PL/Perl are called hstore_plperl and hstore_plperlu, for trusted and untrusted PL/Perl. If you install these transforms and specify them when creating a function, hstore values are mapped to Perl hashes. The extensions for PL/Python are called hstore_plpythonu, hstore_plpython2u, and hstore_plpython3u (see Section 46.1 for the PL/Python naming convention). If you use them, hstore values are mapped to Python dictionaries.

Caution

It is strongly recommended that the transform extensions be installed in the same schema as hstore. Otherwise there are installation-time security hazards if a transform extension's schema contains objects defined by a hostile user.

F.16.8. Authors

Oleg Bartunov <oleg@sai.msu.su>, Moscow, Moscow University, Russia

Teodor Sigaev <teodor@sigaev.ru>, Moscow, Delta-Soft Ltd., Russia

Additional enhancements by Andrew Gierth <andrew@tao11.riddles.org.uk>, United Kingdom

Prev | Up | Next
F.15. fuzzystrmatch | Home | F.17. intagg


F.17. intagg
Prev Up Appendix F. Additional Supplied Modules Home Next

F.17. intagg

F.17.1. Functions
F.17.2. Sample Uses

The intagg module provides an integer aggregator and an enumerator. intagg is now obsolete, because there are built-in functions that provide a superset of its capabilities. However, the module is still provided as a compatibility wrapper around the built-in functions.

F.17.1. Functions

The aggregator is an aggregate function int_array_aggregate(integer) that produces an integer array containing exactly the integers it is fed. This is a wrapper around array_agg, which does the same thing for any array type.

The enumerator is a function int_array_enum(integer[]) that returns setof integer. It is essentially the reverse operation of the aggregator: given an array of integers, expand it into a set of rows. This is a wrapper around unnest, which does the same thing for any array type.

F.17.2. Sample Uses

Many database systems have the notion of a one to many table. Such a table usually sits between two indexed tables, for example:

CREATE TABLE left (id INT PRIMARY KEY, ...);
CREATE TABLE right (id INT PRIMARY KEY, ...);
CREATE TABLE one_to_many(left INT REFERENCES left, right INT REFERENCES right);

It is typically used like this:

SELECT right.* from right JOIN one_to_many ON (right.id = one_to_many.right)
  WHERE one_to_many.left = item;

This will return all the items in the right hand table for an entry in the left hand table. This is a very common construct in SQL.

Now, this methodology can be cumbersome with a very large number of entries in the one_to_many table. Often, a join like this would result in an index scan and a fetch for each right hand entry in the table for a particular left hand entry. If you have a very dynamic system, there is not much you can do. However, if you have some data which is fairly static, you can create a summary table with the aggregator.

CREATE TABLE summary AS
  SELECT left, int_array_aggregate(right) AS right
  FROM one_to_many
  GROUP BY left;

This will create a table with one row per left item, and an array of right items. Now this is pretty useless without some way of using the array; that's why there is an array enumerator. You can do

SELECT left, int_array_enum(right) FROM summary WHERE left = item;

The above query using int_array_enum produces the same results as

SELECT left, right FROM one_to_many WHERE left = item;

The difference is that the query against the summary table has to get only one row from the table, whereas the direct query against one_to_many must index scan and fetch a row for each entry.

On one system, an EXPLAIN showed a query with a cost of 8488 was reduced to a cost of 329. The original query was a join involving the one_to_many table, which was replaced by:

SELECT right, count(right) FROM
  ( SELECT left, int_array_enum(right) AS right
    FROM summary JOIN (SELECT left FROM left_table WHERE left = item) AS lefts
         ON (summary.left = lefts.left)
  ) AS list
  GROUP BY right
  ORDER BY count DESC;

Prev | Up | Next
F.16. hstore | Home | F.18. intarray


F.18. intarray
Prev Up Appendix F. Additional Supplied Modules Home Next

F.18. intarray

F.18.1. intarray Functions and Operators
F.18.2. Index Support
F.18.3. Example
F.18.4. Benchmark
F.18.5. Authors

The intarray module provides a number of useful functions and operators for manipulating null-free arrays of integers. There is also support for indexed searches using some of the operators.

All of these operations will throw an error if a supplied array contains any NULL elements.

Many of these operations are only sensible for one-dimensional arrays. Although they will accept input arrays of more dimensions, the data is treated as though it were a linear array in storage order.

This module is considered “trusted”, that is, it can be installed by non-superusers who have CREATE privilege on the current database.

F.18.1. intarray Functions and Operators

The functions provided by the intarray module are shown in Table F.9, the operators in Table F.10.

Table F.9. intarray Functions

Function

Description

Example(s)

icount ( integer[] ) → integer

Returns the number of elements in the array.

icount('{1,2,3}'::integer[]) → 3

sort ( integer[], dir text ) → integer[]

Sorts the array in either ascending or descending order. dir must be asc or desc.

sort('{1,3,2}'::integer[], 'desc') → {3,2,1}

sort ( integer[] ) → integer[]

sort_asc ( integer[] ) → integer[]

Sorts in ascending order.

sort(array[11,77,44]) → {11,44,77}

sort_desc ( integer[] ) → integer[]

Sorts in descending order.

sort_desc(array[11,77,44]) → {77,44,11}

uniq ( integer[] ) → integer[]

Removes adjacent duplicates. Often used with sort to remove all duplicates.

uniq('{1,2,2,3,1,1}'::integer[]) → {1,2,3,1}

uniq(sort('{1,2,3,2,1}'::integer[])) → {1,2,3}

idx ( integer[], item integer ) → integer

Returns index of the first array element matching item, or 0 if no match.

idx(array[11,22,33,22,11], 22) → 2

subarray ( integer[], start integer, len integer ) → integer[]

Extracts the portion of the array starting at position start, with len elements.

subarray('{1,2,3,2,1}'::integer[], 2, 3) → {2,3,2}

subarray ( integer[], start integer ) → integer[]

Extracts the portion of the array starting at position start.

subarray('{1,2,3,2,1}'::integer[], 2) → {2,3,2,1}

intset ( integer ) → integer[]

Makes a single-element array.

intset(42) → {42}

Table F.10. intarray Operators

Operator

Description

integer[] && integer[] → boolean

Do arrays overlap (have at least one element in common)?

integer[] @> integer[] → boolean

Does left array contain right array?

integer[] <@ integer[] → boolean

Is left array contained in right array?

# integer[] → integer

Returns the number of elements in the array.

integer[] # integer → integer

Returns index of the first array element matching the right argument, or 0 if no match. (Same as idx function.)

integer[] + integer → integer[]

Adds element to end of array.

integer[] + integer[] → integer[]

Concatenates the arrays.

integer[] - integer → integer[]

Removes entries matching the right argument from the array.

integer[] - integer[] → integer[]

Removes elements of the right array from the left array.

integer[] | integer → integer[]

Computes the union of the arguments.

integer[] | integer[] → integer[]

Computes the union of the arguments.

integer[] & integer[] → integer[]

Computes the intersection of the arguments.

integer[] @@ query_int → boolean

Does array satisfy query? (see below)

query_int ~~ integer[] → boolean

Does array satisfy query? (commutator of @@)

The operators &&, @> and <@ are equivalent to PostgreSQL's built-in operators of the same names, except that they work only on integer arrays that do not contain nulls, while the built-in operators work for any array type. This restriction makes them faster than the built-in operators in many cases.

The @@ and ~~ operators test whether an array satisfies a query, which is expressed as a value of a specialized data type query_int. A query consists of integer values that are checked against the elements of the array, possibly combined using the operators & (AND), | (OR), and ! (NOT). Parentheses can be used as needed. For example, the query 1&(2|3) matches arrays that contain 1 and also contain either 2 or 3.

F.18.2. Index Support

intarray provides index support for the &&, @>, and @@ operators, as well as regular array equality.

Two parameterized GiST index operator classes are provided: gist__int_ops (used by default) is suitable for small- to medium-size data sets, while gist__intbig_ops uses a larger signature and is more suitable for indexing large data sets (i.e., columns containing a large number of distinct array values). The implementation uses an RD-tree data structure with built-in lossy compression.

gist__int_ops approximates an integer set as an array of integer ranges. Its optional integer parameter numranges determines the maximum number of ranges in one index key. The default value of numranges is 100. Valid values are between 1 and 253. Using larger arrays as GiST index keys leads to a more precise search (scanning a smaller fraction of the index and fewer heap pages), at the cost of a larger index.

gist__intbig_ops approximates an integer set as a bitmap signature. Its optional integer parameter siglen determines the signature length in bytes. The default signature length is 16 bytes. Valid values of signature length are between 1 and 2024 bytes. Longer signatures lead to a more precise search (scanning a smaller fraction of the index and fewer heap pages), at the cost of a larger index.

There is also a non-default GIN operator class gin__int_ops, which supports these operators as well as <@.

The choice between GiST and GIN indexing depends on the relative performance characteristics of GiST and GIN, which are discussed elsewhere.

F.18.3. Example

-- a message can be in one or more “sections”
CREATE TABLE message (mid INT PRIMARY KEY, sections INT[], ...);

-- create specialized index with signature length of 32 bytes
CREATE INDEX message_rdtree_idx ON message USING GIST (sections gist__intbig_ops (siglen = 32));

-- select messages in section 1 OR 2 - OVERLAP operator
SELECT message.mid FROM message WHERE message.sections && '{1,2}';

-- select messages in sections 1 AND 2 - CONTAINS operator
SELECT message.mid FROM message WHERE message.sections @> '{1,2}';

-- the same, using QUERY operator
SELECT message.mid FROM message WHERE message.sections @@ '1&2'::query_int;

F.18.4. Benchmark

The source directory contrib/intarray/bench contains a benchmark test suite, which can be run against an installed PostgreSQL server. (It also requires DBD::Pg to be installed.) To run:

cd .../contrib/intarray/bench
createdb TEST
psql -c "CREATE EXTENSION intarray" TEST
./create_test.pl | psql TEST
./bench.pl

The bench.pl script has numerous options, which are displayed when it is run without any arguments.

F.18.5. Authors

All work was done by Teodor Sigaev (<teodor@sigaev.ru>) and Oleg Bartunov (<oleg@sai.msu.su>). See http://www.sai.msu.su/~megera/postgres/gist/ for additional information. Andrey Oktyabrski did a great work on adding new functions and operations.

Prev | Up | Next
F.17. intagg | Home | F.19. isn


F.19. isn
Prev Up Appendix F. Additional Supplied Modules Home Next

F.19. isn

F.19.1. Data Types
F.19.2. Casts
F.19.3. Functions and Operators
F.19.4. Examples
F.19.5. Bibliography
F.19.6. Author

The isn module provides data types for the following international product numbering standards: EAN13, UPC, ISBN (books), ISMN (music), and ISSN (serials). Numbers are validated on input according to a hard-coded list of prefixes; this list of prefixes is also used to hyphenate numbers on output. Since new prefixes are assigned from time to time, the list of prefixes may be out of date. It is hoped that a future version of this module will obtain the prefix list from one or more tables that can be easily updated by users as needed; however, at present, the list can only be updated by modifying the source code and recompiling. Alternatively, prefix validation and hyphenation support may be dropped from a future version of this module.

This module is considered “trusted”, that is, it can be installed by non-superusers who have CREATE privilege on the current database.

F.19.1. Data Types

Table F.11 shows the data types provided by the isn module.

Table F.11. isn Data Types

Data Type | Description
EAN13 | European Article Numbers, always displayed in the EAN13 display format
ISBN13 | International Standard Book Numbers to be displayed in the new EAN13 display format
ISMN13 | International Standard Music Numbers to be displayed in the new EAN13 display format
ISSN13 | International Standard Serial Numbers to be displayed in the new EAN13 display format
ISBN | International Standard Book Numbers to be displayed in the old short display format
ISMN | International Standard Music Numbers to be displayed in the old short display format
ISSN | International Standard Serial Numbers to be displayed in the old short display format
UPC | Universal Product Codes

Some notes:

  1. ISBN13, ISMN13, ISSN13 numbers are all EAN13 numbers.

  2. EAN13 numbers aren't always ISBN13, ISMN13 or ISSN13 (some are).

  3. Some ISBN13 numbers can be displayed as ISBN.

  4. Some ISMN13 numbers can be displayed as ISMN.

  5. Some ISSN13 numbers can be displayed as ISSN.

  6. UPC numbers are a subset of the EAN13 numbers (they are basically EAN13 without the first 0 digit).

  7. All UPC, ISBN, ISMN and ISSN numbers can be represented as EAN13 numbers.

Internally, all these types use the same representation (a 64-bit integer), and all are interchangeable. Multiple types are provided to control display formatting and to permit tighter validity checking of input that is supposed to denote one particular type of number.

The ISBN, ISMN, and ISSN types will display the short version of the number (ISxN 10) whenever it's possible, and will show ISxN 13 format for numbers that do not fit in the short version. The EAN13, ISBN13, ISMN13 and ISSN13 types will always display the long version of the ISxN (EAN13).

F.19.2. Casts

The isn module provides the following pairs of type casts:

  • ISBN13 <=> EAN13

  • ISMN13 <=> EAN13

  • ISSN13 <=> EAN13

  • ISBN <=> EAN13

  • ISMN <=> EAN13

  • ISSN <=> EAN13

  • UPC <=> EAN13

  • ISBN <=> ISBN13

  • ISMN <=> ISMN13

  • ISSN <=> ISSN13

When casting from EAN13 to another type, there is a run-time check that the value is within the domain of the other type, and an error is thrown if not. The other casts are simply relabelings that will always succeed.

F.19.3. Functions and Operators

The isn module provides the standard comparison operators, plus B-tree and hash indexing support for all these data types. In addition there are several specialized functions; shown in Table F.12. In this table, isn means any one of the module's data types.

Table F.12. isn Functions

Function

Description

isn_weak ( boolean ) → boolean

Sets the weak input mode, and returns new setting.

isn_weak () → boolean

Returns the current status of the weak mode.

make_valid ( isn ) → isn

Validates an invalid number (clears the invalid flag).

is_valid ( isn ) → boolean

Checks for the presence of the invalid flag.

Weak mode is used to be able to insert invalid data into a table. Invalid means the check digit is wrong, not that there are missing numbers.

Why would you want to use the weak mode? Well, it could be that you have a huge collection of ISBN numbers, and that there are so many of them that for weird reasons some have the wrong check digit (perhaps the numbers were scanned from a printed list and the OCR got the numbers wrong, perhaps the numbers were manually captured... who knows). Anyway, the point is you might want to clean the mess up, but you still want to be able to have all the numbers in your database and maybe use an external tool to locate the invalid numbers in the database so you can verify the information and validate it more easily; so for example you'd want to select all the invalid numbers in the table.

When you insert invalid numbers in a table using the weak mode, the number will be inserted with the corrected check digit, but it will be displayed with an exclamation mark (!) at the end, for example 0-11-000322-5!. This invalid marker can be checked with the is_valid function and cleared with the make_valid function.

You can also force the insertion of invalid numbers even when not in the weak mode, by appending the ! character at the end of the number.

Another special feature is that during input, you can write ? in place of the check digit, and the correct check digit will be inserted automatically.

F.19.4. Examples

--Using the types directly:
SELECT isbn('978-0-393-04002-9');
SELECT isbn13('0901690546');
SELECT issn('1436-4522');

--Casting types:
-- note that you can only cast from ean13 to another type when the
-- number would be valid in the realm of the target type;
-- thus, the following will NOT work: select isbn(ean13('0220356483481'));
-- but these will:
SELECT upc(ean13('0220356483481'));
SELECT ean13(upc('220356483481'));

--Create a table with a single column to hold ISBN numbers:
CREATE TABLE test (id isbn);
INSERT INTO test VALUES('9780393040029');

--Automatically calculate check digits (observe the '?'):
INSERT INTO test VALUES('220500896?');
INSERT INTO test VALUES('978055215372?');

SELECT issn('3251231?');
SELECT ismn('979047213542?');

--Using the weak mode:
SELECT isn_weak(true);
INSERT INTO test VALUES('978-0-11-000533-4');
INSERT INTO test VALUES('9780141219307');
INSERT INTO test VALUES('2-205-00876-X');
SELECT isn_weak(false);

SELECT id FROM test WHERE NOT is_valid(id);
UPDATE test SET id = make_valid(id) WHERE id = '2-205-00876-X!';

SELECT * FROM test;

SELECT isbn13(id) FROM test;

F.19.5. Bibliography

The information to implement this module was collected from several sites, including:

  • https://www.isbn-international.org/

  • https://www.issn.org/

  • https://www.ismn-international.org/

  • https://www.wikipedia.org/

The prefixes used for hyphenation were also compiled from:

  • https://www.gs1.org/standards/id-keys

  • https://en.wikipedia.org/wiki/List_of_ISBN_identifier_groups

  • https://www.isbn-international.org/content/isbn-users-manual

  • https://en.wikipedia.org/wiki/International_Standard_Music_Number

  • https://www.ismn-international.org/ranges.html

Care was taken during the creation of the algorithms and they were meticulously verified against the suggested algorithms in the official ISBN, ISMN, ISSN User Manuals.

F.19.6. Author

Germán Méndez Bravo (Kronuz), 2004–2006

This module was inspired by Garrett A. Wollman's isbn_issn code.

Prev | Up | Next
F.18. intarray | Home | F.20. lo


F.20. lo
Prev Up Appendix F. Additional Supplied Modules Home Next

F.20. lo

F.20.1. Rationale
F.20.2. How to Use It
F.20.3. Limitations
F.20.4. Author

The lo module provides support for managing Large Objects (also called LOs or BLOBs). This includes a data type lo and a trigger lo_manage.

This module is considered “trusted”, that is, it can be installed by non-superusers who have CREATE privilege on the current database.

F.20.1. Rationale

One of the problems with the JDBC driver (and this affects the ODBC driver also), is that the specification assumes that references to BLOBs (Binary Large OBjects) are stored within a table, and if that entry is changed, the associated BLOB is deleted from the database.

As PostgreSQL stands, this doesn't occur. Large objects are treated as objects in their own right; a table entry can reference a large object by OID, but there can be multiple table entries referencing the same large object OID, so the system doesn't delete the large object just because you change or remove one such entry.

Now this is fine for PostgreSQL-specific applications, but standard code using JDBC or ODBC won't delete the objects, resulting in orphan objects — objects that are not referenced by anything, and simply occupy disk space.

The lo module allows fixing this by attaching a trigger to tables that contain LO reference columns. The trigger essentially just does a lo_unlink whenever you delete or modify a value referencing a large object. When you use this trigger, you are assuming that there is only one database reference to any large object that is referenced in a trigger-controlled column!

The module also provides a data type lo, which is really just a domain of the oid type. This is useful for differentiating database columns that hold large object references from those that are OIDs of other things. You don't have to use the lo type to use the trigger, but it may be convenient to use it to keep track of which columns in your database represent large objects that you are managing with the trigger. It is also rumored that the ODBC driver gets confused if you don't use lo for BLOB columns.

F.20.2. How to Use It

Here's a simple example of usage:

CREATE TABLE image (title text, raster lo);

CREATE TRIGGER t_raster BEFORE UPDATE OR DELETE ON image
    FOR EACH ROW EXECUTE FUNCTION lo_manage(raster);

For each column that will contain unique references to large objects, create a BEFORE UPDATE OR DELETE trigger, and give the column name as the sole trigger argument. You can also restrict the trigger to only execute on updates to the column by using BEFORE UPDATE OF column_name. If you need multiple lo columns in the same table, create a separate trigger for each one, remembering to give a different name to each trigger on the same table.

F.20.3. Limitations

  • Dropping a table will still orphan any objects it contains, as the trigger is not executed. You can avoid this by preceding the DROP TABLE with DELETE FROM table.

    TRUNCATE has the same hazard.

    If you already have, or suspect you have, orphaned large objects, see the vacuumlo module to help you clean them up. It's a good idea to run vacuumlo occasionally as a back-stop to the lo_manage trigger.

  • Some frontends may create their own tables, and will not create the associated trigger(s). Also, users may not remember (or know) to create the triggers.

F.20.4. Author

Peter Mount <peter@retep.org.uk>

Prev | Up | Next
F.19. isn | Home | F.21. ltree


F.21. ltree
Prev Up Appendix F. Additional Supplied Modules Home Next

F.21. ltree

F.21.1. Definitions
F.21.2. Operators and Functions
F.21.3. Indexes
F.21.4. Example
F.21.5. Transforms
F.21.6. Authors

This module implements a data type ltree for representing labels of data stored in a hierarchical tree-like structure. Extensive facilities for searching through label trees are provided.

This module is considered “trusted”, that is, it can be installed by non-superusers who have CREATE privilege on the current database.

F.21.1. Definitions

A label is a sequence of alphanumeric characters and underscores (for example, in C locale the characters A-Za-z0-9_ are allowed). Labels must be less than 256 characters long.

Examples: 42, Personal_Services

A label path is a sequence of zero or more labels separated by dots, for example L1.L2.L3, representing a path from the root of a hierarchical tree to a particular node. The length of a label path cannot exceed 65535 labels.

Example: Top.Countries.Europe.Russia

The ltree module provides several data types:

  • ltree stores a label path.

  • lquery represents a regular-expression-like pattern for matching ltree values. A simple word matches that label within a path. A star symbol (*) matches zero or more labels. These can be joined with dots to form a pattern that must match the whole label path. For example:

    foo         Match the exact label path foo
*.foo.*     Match any label path containing the label foo
*.foo       Match any label path whose last label is foo

    Both star symbols and simple words can be quantified to restrict how many labels they can match:

    *{n}        Match exactly n labels
*{n,}       Match at least n labels
*{n,m}      Match at least n but not more than m labels
*{,m}       Match at most m labels — same as *{0,m}
foo{n,m}    Match at least n but not more than m occurrences of foo
foo{,}      Match any number of occurrences of foo, including zero

    In the absence of any explicit quantifier, the default for a star symbol is to match any number of labels (that is, {,}) while the default for a non-star item is to match exactly once (that is, {1}).

    There are several modifiers that can be put at the end of a non-star lquery item to make it match more than just the exact match:

    @           Match case-insensitively, for example a@ matches A
*           Match any label with this prefix, for example foo* matches foobar
%           Match initial underscore-separated words

    The behavior of % is a bit complicated. It tries to match words rather than the entire label. For example foo_bar% matches foo_bar_baz but not foo_barbaz. If combined with *, prefix matching applies to each word separately, for example foo_bar%* matches foo1_bar2_baz but not foo1_br2_baz.

    Also, you can write several possibly-modified non-star items separated with | (OR) to match any of those items, and you can put ! (NOT) at the start of a non-star group to match any label that doesn't match any of the alternatives. A quantifier, if any, goes at the end of the group; it means some number of matches for the group as a whole (that is, some number of labels matching or not matching any of the alternatives).

    Here's an annotated example of lquery:

    Top.*{0,2}.sport*@.!football|tennis{1,}.Russ*|Spain
a.  b.     c.      d.                   e.

    This query will match any label path that:

    1. begins with the label Top

    2. and next has zero to two labels before

    3. a label beginning with the case-insensitive prefix sport

    4. then has one or more labels, none of which match football nor tennis

    5. and then ends with a label beginning with Russ or exactly matching Spain.

  • ltxtquery represents a full-text-search-like pattern for matching ltree values. An ltxtquery value contains words, possibly with the modifiers @, *, % at the end; the modifiers have the same meanings as in lquery. Words can be combined with & (AND), | (OR), ! (NOT), and parentheses. The key difference from lquery is that ltxtquery matches words without regard to their position in the label path.

    Here's an example ltxtquery:

    Europe & Russia*@ & !Transportation

    This will match paths that contain the label Europe and any label beginning with Russia (case-insensitive), but not paths containing the label Transportation. The location of these words within the path is not important. Also, when % is used, the word can be matched to any underscore-separated word within a label, regardless of position.

Note: ltxtquery allows whitespace between symbols, but ltree and lquery do not.

F.21.2. Operators and Functions

Type ltree has the usual comparison operators =, <>, <, >, <=, >=. Comparison sorts in the order of a tree traversal, with the children of a node sorted by label text. In addition, the specialized operators shown in Table F.13 are available.

Table F.13. ltree Operators

Operator

Description

ltree @> ltree → boolean

Is left argument an ancestor of right (or equal)?

ltree <@ ltree → boolean

Is left argument a descendant of right (or equal)?

ltree ~ lquery → boolean

lquery ~ ltree → boolean

Does ltree match lquery?

ltree ? lquery[] → boolean

lquery[] ? ltree → boolean

Does ltree match any lquery in array?

ltree @ ltxtquery → boolean

ltxtquery @ ltree → boolean

Does ltree match ltxtquery?

ltree || ltree → ltree

Concatenates ltree paths.

ltree || text → ltree

text || ltree → ltree

Converts text to ltree and concatenates.

ltree[] @> ltree → boolean

ltree <@ ltree[] → boolean

Does array contain an ancestor of ltree?

ltree[] <@ ltree → boolean

ltree @> ltree[] → boolean

Does array contain a descendant of ltree?

ltree[] ~ lquery → boolean

lquery ~ ltree[] → boolean

Does array contain any path matching lquery?

ltree[] ? lquery[] → boolean

lquery[] ? ltree[] → boolean

Does ltree array contain any path matching any lquery?

ltree[] @ ltxtquery → boolean

ltxtquery @ ltree[] → boolean

Does array contain any path matching ltxtquery?

ltree[] ?@> ltree → ltree

Returns first array entry that is an ancestor of ltree, or NULL if none.

ltree[] ?<@ ltree → ltree

Returns first array entry that is a descendant of ltree, or NULL if none.

ltree[] ?~ lquery → ltree

Returns first array entry that matches lquery, or NULL if none.

ltree[] ?@ ltxtquery → ltree

Returns first array entry that matches ltxtquery, or NULL if none.

The operators <@, @>, @ and ~ have analogues ^<@, ^@>, ^@, ^~, which are the same except they do not use indexes. These are useful only for testing purposes.

The available functions are shown in Table F.14.

Table F.14. ltree Functions

Function

Description

Example(s)

subltree ( ltree, start integer, end integer ) → ltree

Returns subpath of ltree from position start to position end-1 (counting from 0).

subltree('Top.Child1.Child2', 1, 2) → Child1

subpath ( ltree, offset integer, len integer ) → ltree

Returns subpath of ltree starting at position offset, with length len. If offset is negative, subpath starts that far from the end of the path. If len is negative, leaves that many labels off the end of the path.

subpath('Top.Child1.Child2', 0, 2) → Top.Child1

subpath ( ltree, offset integer ) → ltree

Returns subpath of ltree starting at position offset, extending to end of path. If offset is negative, subpath starts that far from the end of the path.

subpath('Top.Child1.Child2', 1) → Child1.Child2

nlevel ( ltree ) → integer

Returns number of labels in path.

nlevel('Top.Child1.Child2') → 3

index ( a ltree, b ltree ) → integer

Returns position of first occurrence of b in a, or -1 if not found.

index('0.1.2.3.5.4.5.6.8.5.6.8', '5.6') → 6

index ( a ltree, b ltree, offset integer ) → integer

Returns position of first occurrence of b in a, or -1 if not found. The search starts at position offset; negative offset means start -offset labels from the end of the path.

index('0.1.2.3.5.4.5.6.8.5.6.8', '5.6', -4) → 9

text2ltree ( text ) → ltree

Casts text to ltree.

ltree2text ( ltree ) → text

Casts ltree to text.

lca ( ltree [, ltree [, ... ]] ) → ltree

Computes longest common ancestor of paths (up to 8 arguments are supported).

lca('1.2.3', '1.2.3.4.5.6') → 1.2

lca ( ltree[] ) → ltree

Computes longest common ancestor of paths in array.

lca(array['1.2.3'::ltree,'1.2.3.4']) → 1.2

F.21.3. Indexes

ltree supports several types of indexes that can speed up the indicated operators:

  • B-tree index over ltree: <, <=, =, >=, >

  • GiST index over ltree (gist_ltree_ops opclass): <, <=, =, >=, >, @>, <@, @, ~, ?

    gist_ltree_ops GiST opclass approximates a set of path labels as a bitmap signature. Its optional integer parameter siglen determines the signature length in bytes. The default signature length is 8 bytes. Valid values of signature length are between 1 and 2024 bytes. Longer signatures lead to a more precise search (scanning a smaller fraction of the index and fewer heap pages), at the cost of a larger index.

    Example of creating such an index with the default signature length of 8 bytes:

    CREATE INDEX path_gist_idx ON test USING GIST (path);

    Example of creating such an index with a signature length of 100 bytes:

    CREATE INDEX path_gist_idx ON test USING GIST (path gist_ltree_ops(siglen=100));

  • GiST index over ltree[] (gist__ltree_ops opclass): ltree[] <@ ltree, ltree @> ltree[], @, ~, ?

    gist__ltree_ops GiST opclass works similarly to gist_ltree_ops and also takes signature length as a parameter. The default value of siglen in gist__ltree_ops is 28 bytes.

    Example of creating such an index with the default signature length of 28 bytes:

    CREATE INDEX path_gist_idx ON test USING GIST (array_path);

    Example of creating such an index with a signature length of 100 bytes:

    CREATE INDEX path_gist_idx ON test USING GIST (array_path gist__ltree_ops(siglen=100));

    Note: This index type is lossy.

F.21.4. Example

This example uses the following data (also available in file contrib/ltree/ltreetest.sql in the source distribution):

CREATE TABLE test (path ltree);
INSERT INTO test VALUES ('Top');
INSERT INTO test VALUES ('Top.Science');
INSERT INTO test VALUES ('Top.Science.Astronomy');
INSERT INTO test VALUES ('Top.Science.Astronomy.Astrophysics');
INSERT INTO test VALUES ('Top.Science.Astronomy.Cosmology');
INSERT INTO test VALUES ('Top.Hobbies');
INSERT INTO test VALUES ('Top.Hobbies.Amateurs_Astronomy');
INSERT INTO test VALUES ('Top.Collections');
INSERT INTO test VALUES ('Top.Collections.Pictures');
INSERT INTO test VALUES ('Top.Collections.Pictures.Astronomy');
INSERT INTO test VALUES ('Top.Collections.Pictures.Astronomy.Stars');
INSERT INTO test VALUES ('Top.Collections.Pictures.Astronomy.Galaxies');
INSERT INTO test VALUES ('Top.Collections.Pictures.Astronomy.Astronauts');
CREATE INDEX path_gist_idx ON test USING GIST (path);
CREATE INDEX path_idx ON test USING BTREE (path);

Now, we have a table test populated with data describing the hierarchy shown below:

                        Top
                     /   |  \
             Science Hobbies Collections
                 /       |              \
        Astronomy   Amateurs_Astronomy Pictures
           /  \                            |
Astrophysics  Cosmology                Astronomy
                                        /  |    \
                                 Galaxies Stars Astronauts

We can do inheritance:

ltreetest=> SELECT path FROM test WHERE path <@ 'Top.Science';
                path
------------------------------------
 Top.Science
 Top.Science.Astronomy
 Top.Science.Astronomy.Astrophysics
 Top.Science.Astronomy.Cosmology
(4 rows)

Here are some examples of path matching:

ltreetest=> SELECT path FROM test WHERE path ~ '*.Astronomy.*';
                     path
-----------------------------------------------
 Top.Science.Astronomy
 Top.Science.Astronomy.Astrophysics
 Top.Science.Astronomy.Cosmology
 Top.Collections.Pictures.Astronomy
 Top.Collections.Pictures.Astronomy.Stars
 Top.Collections.Pictures.Astronomy.Galaxies
 Top.Collections.Pictures.Astronomy.Astronauts
(7 rows)

ltreetest=> SELECT path FROM test WHERE path ~ '*.!pictures@.Astronomy.*';
                path
------------------------------------
 Top.Science.Astronomy
 Top.Science.Astronomy.Astrophysics
 Top.Science.Astronomy.Cosmology
(3 rows)

Here are some examples of full text search:

ltreetest=> SELECT path FROM test WHERE path @ 'Astro*% & !pictures@';
                path
------------------------------------
 Top.Science.Astronomy
 Top.Science.Astronomy.Astrophysics
 Top.Science.Astronomy.Cosmology
 Top.Hobbies.Amateurs_Astronomy
(4 rows)

ltreetest=> SELECT path FROM test WHERE path @ 'Astro* & !pictures@';
                path
------------------------------------
 Top.Science.Astronomy
 Top.Science.Astronomy.Astrophysics
 Top.Science.Astronomy.Cosmology
(3 rows)

Path construction using functions:

ltreetest=> SELECT subpath(path,0,2)||'Space'||subpath(path,2) FROM test WHERE path <@ 'Top.Science.Astronomy';
                 ?column?
------------------------------------------
 Top.Science.Space.Astronomy
 Top.Science.Space.Astronomy.Astrophysics
 Top.Science.Space.Astronomy.Cosmology
(3 rows)

We could simplify this by creating an SQL function that inserts a label at a specified position in a path:

CREATE FUNCTION ins_label(ltree, int, text) RETURNS ltree
    AS 'select subpath($1,0,$2) || $3 || subpath($1,$2);'
    LANGUAGE SQL IMMUTABLE;

ltreetest=> SELECT ins_label(path,2,'Space') FROM test WHERE path <@ 'Top.Science.Astronomy';
                ins_label
------------------------------------------
 Top.Science.Space.Astronomy
 Top.Science.Space.Astronomy.Astrophysics
 Top.Science.Space.Astronomy.Cosmology
(3 rows)

F.21.5. Transforms

Additional extensions are available that implement transforms for the ltree type for PL/Python. The extensions are called ltree_plpythonu, ltree_plpython2u, and ltree_plpython3u (see Section 46.1 for the PL/Python naming convention). If you install these transforms and specify them when creating a function, ltree values are mapped to Python lists. (The reverse is currently not supported, however.)

Caution

It is strongly recommended that the transform extensions be installed in the same schema as ltree. Otherwise there are installation-time security hazards if a transform extension's schema contains objects defined by a hostile user.

F.21.6. Authors

All work was done by Teodor Sigaev (<teodor@stack.net>) and Oleg Bartunov (<oleg@sai.msu.su>). See http://www.sai.msu.su/~megera/postgres/gist/ for additional information. Authors would like to thank Eugeny Rodichev for helpful discussions. Comments and bug reports are welcome.

Prev | Up | Next
F.20. lo | Home | F.22. old_snapshot


F.22. old_snapshot
Prev Up Appendix F. Additional Supplied Modules Home Next

F.22. old_snapshot

F.22.1. Functions

The old_snapshot module allows inspection of the server state that is used to implement old_snapshot_threshold.

F.22.1. Functions

pg_old_snapshot_time_mapping(array_offset OUT int4, end_timestamp OUT timestamptz, newest_xmin OUT xid) returns setof record

Returns all of the entries in the server's timestamp to XID mapping. Each entry represents the newest xmin of any snapshot taken in the corresponding minute.

Prev | Up | Next
F.21. ltree | Home | F.23. pageinspect


F.23. pageinspect
Prev Up Appendix F. Additional Supplied Modules Home Next

F.23. pageinspect

F.23.1. General Functions
F.23.2. Heap Functions
F.23.3. B-Tree Functions
F.23.4. BRIN Functions
F.23.5. GIN Functions
F.23.6. GiST Functions
F.23.7. Hash Functions

The pageinspect module provides functions that allow you to inspect the contents of database pages at a low level, which is useful for debugging purposes. All of these functions may be used only by superusers.

F.23.1. General Functions

get_raw_page(relname text, fork text, blkno bigint) returns bytea

get_raw_page reads the specified block of the named relation and returns a copy as a bytea value. This allows a single time-consistent copy of the block to be obtained. fork should be 'main' for the main data fork, 'fsm' for the free space map, 'vm' for the visibility map, or 'init' for the initialization fork.

get_raw_page(relname text, blkno bigint) returns bytea

A shorthand version of get_raw_page, for reading from the main fork. Equivalent to get_raw_page(relname, 'main', blkno)

page_header(page bytea) returns record

page_header shows fields that are common to all PostgreSQL heap and index pages.

A page image obtained with get_raw_page should be passed as argument. For example:

test=# SELECT * FROM page_header(get_raw_page('pg_class', 0));
    lsn    | checksum | flags  | lower | upper | special | pagesize | version | prune_xid
-----------+----------+--------+-------+-------+---------+----------+---------+-----------
 0/24A1B50 |        0 |      1 |   232 |   368 |    8192 |     8192 |       4 |         0

The returned columns correspond to the fields in the PageHeaderData struct. See src/include/storage/bufpage.h for details.

The checksum field is the checksum stored in the page, which might be incorrect if the page is somehow corrupted. If data checksums are not enabled for this instance, then the value stored is meaningless.

page_checksum(page bytea, blkno bigint) returns smallint

page_checksum computes the checksum for the page, as if it was located at the given block.

A page image obtained with get_raw_page should be passed as argument. For example:

test=# SELECT page_checksum(get_raw_page('pg_class', 0), 0);
 page_checksum
---------------
         13443

Note that the checksum depends on the block number, so matching block numbers should be passed (except when doing esoteric debugging).

The checksum computed with this function can be compared with the checksum result field of the function page_header. If data checksums are enabled for this instance, then the two values should be equal.

fsm_page_contents(page bytea) returns text

fsm_page_contents shows the internal node structure of an FSM page. For example:

test=# SELECT fsm_page_contents(get_raw_page('pg_class', 'fsm', 0));

The output is a multiline string, with one line per node in the binary tree within the page. Only those nodes that are not zero are printed. The so-called "next" pointer, which points to the next slot to be returned from the page, is also printed.

See src/backend/storage/freespace/README for more information on the structure of an FSM page.

F.23.2. Heap Functions

heap_page_items(page bytea) returns setof record

heap_page_items shows all line pointers on a heap page. For those line pointers that are in use, tuple headers as well as tuple raw data are also shown. All tuples are shown, whether or not the tuples were visible to an MVCC snapshot at the time the raw page was copied.

A heap page image obtained with get_raw_page should be passed as argument. For example:

test=# SELECT * FROM heap_page_items(get_raw_page('pg_class', 0));

See src/include/storage/itemid.h and src/include/access/htup_details.h for explanations of the fields returned.

The heap_tuple_infomask_flags function can be used to unpack the flag bits of t_infomask and t_infomask2 for heap tuples.

tuple_data_split(rel_oid oid, t_data bytea, t_infomask integer, t_infomask2 integer, t_bits text [, do_detoast bool]) returns bytea[]

tuple_data_split splits tuple data into attributes in the same way as backend internals.

test=# SELECT tuple_data_split('pg_class'::regclass, t_data, t_infomask, t_infomask2, t_bits) FROM heap_page_items(get_raw_page('pg_class', 0));

This function should be called with the same arguments as the return attributes of heap_page_items.

If do_detoast is true, attributes will be detoasted as needed. Default value is false.

heap_page_item_attrs(page bytea, rel_oid regclass [, do_detoast bool]) returns setof record

heap_page_item_attrs is equivalent to heap_page_items except that it returns tuple raw data as an array of attributes that can optionally be detoasted by do_detoast which is false by default.

A heap page image obtained with get_raw_page should be passed as argument. For example:

test=# SELECT * FROM heap_page_item_attrs(get_raw_page('pg_class', 0), 'pg_class'::regclass);

heap_tuple_infomask_flags(t_infomask integer, t_infomask2 integer) returns record

heap_tuple_infomask_flags decodes the t_infomask and t_infomask2 returned by heap_page_items into a human-readable set of arrays made of flag names, with one column for all the flags and one column for combined flags. For example:

test=# SELECT t_ctid, raw_flags, combined_flags
         FROM heap_page_items(get_raw_page('pg_class', 0)),
           LATERAL heap_tuple_infomask_flags(t_infomask, t_infomask2)
         WHERE t_infomask IS NOT NULL OR t_infomask2 IS NOT NULL;

This function should be called with the same arguments as the return attributes of heap_page_items.

Combined flags are displayed for source-level macros that take into account the value of more than one raw bit, such as HEAP_XMIN_FROZEN.

See src/include/access/htup_details.h for explanations of the flag names returned.

F.23.3. B-Tree Functions

bt_metap(relname text) returns record

bt_metap returns information about a B-tree index's metapage. For example:

test=# SELECT * FROM bt_metap('pg_cast_oid_index');
-[ RECORD 1 ]-------------+-------
magic                     | 340322
version                   | 4
root                      | 1
level                     | 0
fastroot                  | 1
fastlevel                 | 0
last_cleanup_num_delpages | 0
last_cleanup_num_tuples   | 230
allequalimage             | f

bt_page_stats(relname text, blkno bigint) returns record

bt_page_stats returns summary information about single pages of B-tree indexes. For example:

test=# SELECT * FROM bt_page_stats('pg_cast_oid_index', 1);
-[ RECORD 1 ]-+-----
blkno         | 1
type          | l
live_items    | 224
dead_items    | 0
avg_item_size | 16
page_size     | 8192
free_size     | 3668
btpo_prev     | 0
btpo_next     | 0
btpo_level    | 0
btpo_flags    | 3

bt_page_items(relname text, blkno bigint) returns setof record

bt_page_items returns detailed information about all of the items on a B-tree index page. For example:

test=# SELECT itemoffset, ctid, itemlen, nulls, vars, data, dead, htid, tids[0:2] AS some_tids
        FROM bt_page_items('tenk2_hundred', 5);
 itemoffset |   ctid    | itemlen | nulls | vars |          data           | dead |  htid  |      some_tids
------------+-----------+---------+-------+------+-------------------------+------+--------+---------------------
          1 | (16,1)    |      16 | f     | f    | 30 00 00 00 00 00 00 00 |      |        |
          2 | (16,8292) |     616 | f     | f    | 24 00 00 00 00 00 00 00 | f    | (1,6)  | {"(1,6)","(10,22)"}
          3 | (16,8292) |     616 | f     | f    | 25 00 00 00 00 00 00 00 | f    | (1,18) | {"(1,18)","(4,22)"}
          4 | (16,8292) |     616 | f     | f    | 26 00 00 00 00 00 00 00 | f    | (4,18) | {"(4,18)","(6,17)"}
          5 | (16,8292) |     616 | f     | f    | 27 00 00 00 00 00 00 00 | f    | (1,2)  | {"(1,2)","(1,19)"}
          6 | (16,8292) |     616 | f     | f    | 28 00 00 00 00 00 00 00 | f    | (2,24) | {"(2,24)","(4,11)"}
          7 | (16,8292) |     616 | f     | f    | 29 00 00 00 00 00 00 00 | f    | (2,17) | {"(2,17)","(11,2)"}
          8 | (16,8292) |     616 | f     | f    | 2a 00 00 00 00 00 00 00 | f    | (0,25) | {"(0,25)","(3,20)"}
          9 | (16,8292) |     616 | f     | f    | 2b 00 00 00 00 00 00 00 | f    | (0,10) | {"(0,10)","(0,14)"}
         10 | (16,8292) |     616 | f     | f    | 2c 00 00 00 00 00 00 00 | f    | (1,3)  | {"(1,3)","(3,9)"}
         11 | (16,8292) |     616 | f     | f    | 2d 00 00 00 00 00 00 00 | f    | (6,28) | {"(6,28)","(11,1)"}
         12 | (16,8292) |     616 | f     | f    | 2e 00 00 00 00 00 00 00 | f    | (0,27) | {"(0,27)","(1,13)"}
         13 | (16,8292) |     616 | f     | f    | 2f 00 00 00 00 00 00 00 | f    | (4,17) | {"(4,17)","(4,21)"}
(13 rows)

This is a B-tree leaf page. All tuples that point to the table happen to be posting list tuples (all of which store a total of 100 6 byte TIDs). There is also a “high key” tuple at itemoffset number 1. ctid is used to store encoded information about each tuple in this example, though leaf page tuples often store a heap TID directly in the ctid field instead. tids is the list of TIDs stored as a posting list.

In an internal page (not shown), the block number part of ctid is a “downlink”, which is a block number of another page in the index itself. The offset part (the second number) of ctid stores encoded information about the tuple, such as the number of columns present (suffix truncation may have removed unneeded suffix columns). Truncated columns are treated as having the value “minus infinity”.

htid shows a heap TID for the tuple, regardless of the underlying tuple representation. This value may match ctid, or may be decoded from the alternative representations used by posting list tuples and tuples from internal pages. Tuples in internal pages usually have the implementation level heap TID column truncated away, which is represented as a NULL htid value.

Note that the first item on any non-rightmost page (any page with a non-zero value in the btpo_next field) is the page's “high key”, meaning its data serves as an upper bound on all items appearing on the page, while its ctid field does not point to another block. Also, on internal pages, the first real data item (the first item that is not a high key) reliably has every column truncated away, leaving no actual value in its data field. Such an item does have a valid downlink in its ctid field, however.

For more details about the structure of B-tree indexes, see Section 64.4.1. For more details about deduplication and posting lists, see Section 64.4.3.

bt_page_items(page bytea) returns setof record

It is also possible to pass a page to bt_page_items as a bytea value. A page image obtained with get_raw_page should be passed as argument. So the last example could also be rewritten like this:

test=# SELECT itemoffset, ctid, itemlen, nulls, vars, data, dead, htid, tids[0:2] AS some_tids
        FROM bt_page_items(get_raw_page('tenk2_hundred', 5));
 itemoffset |   ctid    | itemlen | nulls | vars |          data           | dead |  htid  |      some_tids
------------+-----------+---------+-------+------+-------------------------+------+--------+---------------------
          1 | (16,1)    |      16 | f     | f    | 30 00 00 00 00 00 00 00 |      |        |
          2 | (16,8292) |     616 | f     | f    | 24 00 00 00 00 00 00 00 | f    | (1,6)  | {"(1,6)","(10,22)"}
          3 | (16,8292) |     616 | f     | f    | 25 00 00 00 00 00 00 00 | f    | (1,18) | {"(1,18)","(4,22)"}
          4 | (16,8292) |     616 | f     | f    | 26 00 00 00 00 00 00 00 | f    | (4,18) | {"(4,18)","(6,17)"}
          5 | (16,8292) |     616 | f     | f    | 27 00 00 00 00 00 00 00 | f    | (1,2)  | {"(1,2)","(1,19)"}
          6 | (16,8292) |     616 | f     | f    | 28 00 00 00 00 00 00 00 | f    | (2,24) | {"(2,24)","(4,11)"}
          7 | (16,8292) |     616 | f     | f    | 29 00 00 00 00 00 00 00 | f    | (2,17) | {"(2,17)","(11,2)"}
          8 | (16,8292) |     616 | f     | f    | 2a 00 00 00 00 00 00 00 | f    | (0,25) | {"(0,25)","(3,20)"}
          9 | (16,8292) |     616 | f     | f    | 2b 00 00 00 00 00 00 00 | f    | (0,10) | {"(0,10)","(0,14)"}
         10 | (16,8292) |     616 | f     | f    | 2c 00 00 00 00 00 00 00 | f    | (1,3)  | {"(1,3)","(3,9)"}
         11 | (16,8292) |     616 | f     | f    | 2d 00 00 00 00 00 00 00 | f    | (6,28) | {"(6,28)","(11,1)"}
         12 | (16,8292) |     616 | f     | f    | 2e 00 00 00 00 00 00 00 | f    | (0,27) | {"(0,27)","(1,13)"}
         13 | (16,8292) |     616 | f     | f    | 2f 00 00 00 00 00 00 00 | f    | (4,17) | {"(4,17)","(4,21)"}
(13 rows)

All the other details are the same as explained in the previous item.

F.23.4. BRIN Functions

brin_page_type(page bytea) returns text

brin_page_type returns the page type of the given BRIN index page, or throws an error if the page is not a valid BRIN page. For example:

test=# SELECT brin_page_type(get_raw_page('brinidx', 0));
 brin_page_type
----------------
 meta

brin_metapage_info(page bytea) returns record

brin_metapage_info returns assorted information about a BRIN index metapage. For example:

test=# SELECT * FROM brin_metapage_info(get_raw_page('brinidx', 0));
   magic    | version | pagesperrange | lastrevmappage
------------+---------+---------------+----------------
 0xA8109CFA |       1 |             4 |              2

brin_revmap_data(page bytea) returns setof tid

brin_revmap_data returns the list of tuple identifiers in a BRIN index range map page. For example:

test=# SELECT * FROM brin_revmap_data(get_raw_page('brinidx', 2)) LIMIT 5;
  pages
---------
 (6,137)
 (6,138)
 (6,139)
 (6,140)
 (6,141)

brin_page_items(page bytea, index oid) returns setof record

brin_page_items returns the data stored in the BRIN data page. For example:

test=# SELECT * FROM brin_page_items(get_raw_page('brinidx', 5),
                                     'brinidx')
       ORDER BY blknum, attnum LIMIT 6;
 itemoffset | blknum | attnum | allnulls | hasnulls | placeholder |    value
------------+--------+--------+----------+----------+-------------+--------------
        137 |      0 |      1 | t        | f        | f           |
        137 |      0 |      2 | f        | f        | f           | {1 .. 88}
        138 |      4 |      1 | t        | f        | f           |
        138 |      4 |      2 | f        | f        | f           | {89 .. 176}
        139 |      8 |      1 | t        | f        | f           |
        139 |      8 |      2 | f        | f        | f           | {177 .. 264}

The returned columns correspond to the fields in the BrinMemTuple and BrinValues structs. See src/include/access/brin_tuple.h for details.

F.23.5. GIN Functions

gin_metapage_info(page bytea) returns record

gin_metapage_info returns information about a GIN index metapage. For example:

test=# SELECT * FROM gin_metapage_info(get_raw_page('gin_index', 0));
-[ RECORD 1 ]----+-----------
pending_head     | 4294967295
pending_tail     | 4294967295
tail_free_size   | 0
n_pending_pages  | 0
n_pending_tuples | 0
n_total_pages    | 7
n_entry_pages    | 6
n_data_pages     | 0
n_entries        | 693
version          | 2

gin_page_opaque_info(page bytea) returns record

gin_page_opaque_info returns information about a GIN index opaque area, like the page type. For example:

test=# SELECT * FROM gin_page_opaque_info(get_raw_page('gin_index', 2));
 rightlink | maxoff |         flags
-----------+--------+------------------------
         5 |      0 | {data,leaf,compressed}
(1 row)

gin_leafpage_items(page bytea) returns setof record

gin_leafpage_items returns information about the data stored in a GIN leaf page. For example:

test=# SELECT first_tid, nbytes, tids[0:5] AS some_tids
        FROM gin_leafpage_items(get_raw_page('gin_test_idx', 2));
 first_tid | nbytes |                        some_tids
-----------+--------+----------------------------------------------------------
 (8,41)    |    244 | {"(8,41)","(8,43)","(8,44)","(8,45)","(8,46)"}
 (10,45)   |    248 | {"(10,45)","(10,46)","(10,47)","(10,48)","(10,49)"}
 (12,52)   |    248 | {"(12,52)","(12,53)","(12,54)","(12,55)","(12,56)"}
 (14,59)   |    320 | {"(14,59)","(14,60)","(14,61)","(14,62)","(14,63)"}
 (167,16)  |    376 | {"(167,16)","(167,17)","(167,18)","(167,19)","(167,20)"}
 (170,30)  |    376 | {"(170,30)","(170,31)","(170,32)","(170,33)","(170,34)"}
 (173,44)  |    197 | {"(173,44)","(173,45)","(173,46)","(173,47)","(173,48)"}
(7 rows)

F.23.6. GiST Functions

gist_page_opaque_info(page bytea) returns record

gist_page_opaque_info returns information from a GiST index page's opaque area, such as the NSN, rightlink and page type. For example:

test=# SELECT * FROM gist_page_opaque_info(get_raw_page('test_gist_idx', 2));
 lsn | nsn | rightlink | flags
-----+-----+-----------+--------
 0/1 | 0/0 |         1 | {leaf}
(1 row)

gist_page_items(page bytea, index_oid regclass) returns setof record

gist_page_items returns information about the data stored in a page of a GiST index. For example:

test=# SELECT * FROM gist_page_items(get_raw_page('test_gist_idx', 0), 'test_gist_idx');
 itemoffset |   ctid    | itemlen | dead |       keys
------------+-----------+---------+------+-------------------
          1 | (1,65535) |      40 | f    | (p)=((166,166))
          2 | (2,65535) |      40 | f    | (p)=((332,332))
          3 | (3,65535) |      40 | f    | (p)=((498,498))
          4 | (4,65535) |      40 | f    | (p)=((664,664))
          5 | (5,65535) |      40 | f    | (p)=((830,830))
          6 | (6,65535) |      40 | f    | (p)=((996,996))
          7 | (7,65535) |      40 | f    | (p)=((1000,1000))
(7 rows)

gist_page_items_bytea(page bytea) returns setof record

Same as gist_page_items, but returns the key data as a raw bytea blob. Since it does not attempt to decode the key, it does not need to know which index is involved. For example:

test=# SELECT * FROM gist_page_items_bytea(get_raw_page('test_gist_idx', 0));
 itemoffset |   ctid    | itemlen | dead |                                      key_data
------------+-----------+---------+------+-----------------------------------------​-------------------------------------------
          1 | (1,65535) |      40 | f    | \x00000100ffff28000000000000c0644000000000​00c06440000000000000f03f000000000000f03f
          2 | (2,65535) |      40 | f    | \x00000200ffff28000000000000c0744000000000​00c074400000000000e064400000000000e06440
          3 | (3,65535) |      40 | f    | \x00000300ffff28000000000000207f4000000000​00207f400000000000d074400000000000d07440
          4 | (4,65535) |      40 | f    | \x00000400ffff28000000000000c0844000000000​00c084400000000000307f400000000000307f40
          5 | (5,65535) |      40 | f    | \x00000500ffff28000000000000f0894000000000​00f089400000000000c884400000000000c88440
          6 | (6,65535) |      40 | f    | \x00000600ffff28000000000000208f4000000000​00208f400000000000f889400000000000f88940
          7 | (7,65535) |      40 | f    | \x00000700ffff28000000000000408f4000000000​00408f400000000000288f400000000000288f40
(7 rows)

F.23.7. Hash Functions

hash_page_type(page bytea) returns text

hash_page_type returns page type of the given HASH index page. For example:

test=# SELECT hash_page_type(get_raw_page('con_hash_index', 0));
 hash_page_type
----------------
 metapage

hash_page_stats(page bytea) returns setof record

hash_page_stats returns information about a bucket or overflow page of a HASH index. For example:

test=# SELECT * FROM hash_page_stats(get_raw_page('con_hash_index', 1));
-[ RECORD 1 ]---+-----------
live_items      | 407
dead_items      | 0
page_size       | 8192
free_size       | 8
hasho_prevblkno | 4096
hasho_nextblkno | 8474
hasho_bucket    | 0
hasho_flag      | 66
hasho_page_id   | 65408

hash_page_items(page bytea) returns setof record

hash_page_items returns information about the data stored in a bucket or overflow page of a HASH index page. For example:

test=# SELECT * FROM hash_page_items(get_raw_page('con_hash_index', 1)) LIMIT 5;
 itemoffset |   ctid    |    data
------------+-----------+------------
          1 | (899,77)  | 1053474816
          2 | (897,29)  | 1053474816
          3 | (894,207) | 1053474816
          4 | (892,159) | 1053474816
          5 | (890,111) | 1053474816

hash_bitmap_info(index oid, blkno bigint) returns record

hash_bitmap_info shows the status of a bit in the bitmap page for a particular overflow page of HASH index. For example:

test=# SELECT * FROM hash_bitmap_info('con_hash_index', 2052);
 bitmapblkno | bitmapbit | bitstatus
-------------+-----------+-----------
          65 |         3 | t

hash_metapage_info(page bytea) returns record

hash_metapage_info returns information stored in the meta page of a HASH index. For example:

test=# SELECT magic, version, ntuples, ffactor, bsize, bmsize, bmshift,
test-#     maxbucket, highmask, lowmask, ovflpoint, firstfree, nmaps, procid,
test-#     regexp_replace(spares::text, '(,0)*}', '}') as spares,
test-#     regexp_replace(mapp::text, '(,0)*}', '}') as mapp
test-# FROM hash_metapage_info(get_raw_page('con_hash_index', 0));
-[ RECORD 1 ]-------------------------------------------------​------------------------------
magic     | 105121344
version   | 4
ntuples   | 500500
ffactor   | 40
bsize     | 8152
bmsize    | 4096
bmshift   | 15
maxbucket | 12512
highmask  | 16383
lowmask   | 8191
ovflpoint | 28
firstfree | 1204
nmaps     | 1
procid    | 450
spares    | {0,0,0,0,0,0,1,1,1,1,1,1,1,1,3,4,4,4,45,55,58,59,​508,567,628,704,1193,1202,1204}
mapp      | {65}

Prev | Up | Next
F.22. old_snapshot | Home | F.24. passwordcheck


F.24. passwordcheck
Prev Up Appendix F. Additional Supplied Modules Home Next

F.24. passwordcheck

The passwordcheck module checks users' passwords whenever they are set with CREATE ROLE or ALTER ROLE. If a password is considered too weak, it will be rejected and the command will terminate with an error.

To enable this module, add '$libdir/passwordcheck' to shared_preload_libraries in postgresql.conf, then restart the server.

You can adapt this module to your needs by changing the source code. For example, you can use CrackLib to check passwords — this only requires uncommenting two lines in the Makefile and rebuilding the module. (We cannot include CrackLib by default for license reasons.) Without CrackLib, the module enforces a few simple rules for password strength, which you can modify or extend as you see fit.

Caution

To prevent unencrypted passwords from being sent across the network, written to the server log or otherwise stolen by a database administrator, PostgreSQL allows the user to supply pre-encrypted passwords. Many client programs make use of this functionality and encrypt the password before sending it to the server.

This limits the usefulness of the passwordcheck module, because in that case it can only try to guess the password. For this reason, passwordcheck is not recommended if your security requirements are high. It is more secure to use an external authentication method such as GSSAPI (see Chapter 21) than to rely on passwords within the database.

Alternatively, you could modify passwordcheck to reject pre-encrypted passwords, but forcing users to set their passwords in clear text carries its own security risks.

Prev | Up | Next
F.23. pageinspect | Home | F.25. pg_buffercache


F.25. pg_buffercache
Prev Up Appendix F. Additional Supplied Modules Home Next

F.25. pg_buffercache

F.25.1. The pg_buffercache View
F.25.2. Sample Output
F.25.3. Authors

The pg_buffercache module provides a means for examining what's happening in the shared buffer cache in real time.

The module provides a C function pg_buffercache_pages that returns a set of records, plus a view pg_buffercache that wraps the function for convenient use.

By default, use is restricted to superusers and members of the pg_monitor role. Access may be granted to others using GRANT.

F.25.1. The pg_buffercache View

The definitions of the columns exposed by the view are shown in Table F.15.

Table F.15. pg_buffercache Columns

Column Type

Description

bufferid integer

ID, in the range 1..shared_buffers

relfilenode oid (references pg_class.relfilenode)

Filenode number of the relation

reltablespace oid (references pg_tablespace.oid)

Tablespace OID of the relation

reldatabase oid (references pg_database.oid)

Database OID of the relation

relforknumber smallint

Fork number within the relation; see common/relpath.h

relblocknumber bigint

Page number within the relation

isdirty boolean

Is the page dirty?

usagecount smallint

Clock-sweep access count

pinning_backends integer

Number of backends pinning this buffer

There is one row for each buffer in the shared cache. Unused buffers are shown with all fields null except bufferid. Shared system catalogs are shown as belonging to database zero.

Because the cache is shared by all the databases, there will normally be pages from relations not belonging to the current database. This means that there may not be matching join rows in pg_class for some rows, or that there could even be incorrect joins. If you are trying to join against pg_class, it's a good idea to restrict the join to rows having reldatabase equal to the current database's OID or zero.

Since buffer manager locks are not taken to copy the buffer state data that the view will display, accessing pg_buffercache view has less impact on normal buffer activity but it doesn't provide a consistent set of results across all buffers. However, we ensure that the information of each buffer is self-consistent.

F.25.2. Sample Output

regression=# SELECT n.nspname, c.relname, count(*) AS buffers
             FROM pg_buffercache b JOIN pg_class c
             ON b.relfilenode = pg_relation_filenode(c.oid) AND
                b.reldatabase IN (0, (SELECT oid FROM pg_database
                                      WHERE datname = current_database()))
             JOIN pg_namespace n ON n.oid = c.relnamespace
             GROUP BY n.nspname, c.relname
             ORDER BY 3 DESC
             LIMIT 10;

  nspname   |        relname         | buffers
------------+------------------------+---------
 public     | delete_test_table      |     593
 public     | delete_test_table_pkey |     494
 pg_catalog | pg_attribute           |     472
 public     | quad_poly_tbl          |     353
 public     | tenk2                  |     349
 public     | tenk1                  |     349
 public     | gin_test_idx           |     306
 pg_catalog | pg_largeobject         |     206
 public     | gin_test_tbl           |     188
 public     | spgist_text_tbl        |     182
(10 rows)

F.25.3. Authors

Mark Kirkwood <markir@paradise.net.nz>

Design suggestions: Neil Conway <neilc@samurai.com>

Debugging advice: Tom Lane <tgl@sss.pgh.pa.us>

Prev | Up | Next
F.24. passwordcheck | Home | F.26. pgcrypto


F.26. pgcrypto
Prev Up Appendix F. Additional Supplied Modules Home Next

F.26. pgcrypto

F.26.1. General Hashing Functions
F.26.2. Password Hashing Functions
F.26.3. PGP Encryption Functions
F.26.4. Raw Encryption Functions
F.26.5. Random-Data Functions
F.26.6. Notes
F.26.7. Author

The pgcrypto module provides cryptographic functions for PostgreSQL.

This module is considered “trusted”, that is, it can be installed by non-superusers who have CREATE privilege on the current database.

F.26.1. General Hashing Functions

F.26.1.1. digest()

digest(data text, type text) returns bytea
digest(data bytea, type text) returns bytea

Computes a binary hash of the given data. type is the algorithm to use. Standard algorithms are md5, sha1, sha224, sha256, sha384 and sha512. If pgcrypto was built with OpenSSL, more algorithms are available, as detailed in Table F.19.

If you want the digest as a hexadecimal string, use encode() on the result. For example:

CREATE OR REPLACE FUNCTION sha1(bytea) returns text AS $$
    SELECT encode(digest($1, 'sha1'), 'hex')
$$ LANGUAGE SQL STRICT IMMUTABLE;

F.26.1.2. hmac()

hmac(data text, key text, type text) returns bytea
hmac(data bytea, key bytea, type text) returns bytea

Calculates hashed MAC for data with key key. type is the same as in digest().

This is similar to digest() but the hash can only be recalculated knowing the key. This prevents the scenario of someone altering data and also changing the hash to match.

If the key is larger than the hash block size it will first be hashed and the result will be used as key.

F.26.2. Password Hashing Functions

The functions crypt() and gen_salt() are specifically designed for hashing passwords. crypt() does the hashing and gen_salt() prepares algorithm parameters for it.

The algorithms in crypt() differ from the usual MD5 or SHA1 hashing algorithms in the following respects:

  1. They are slow. As the amount of data is so small, this is the only way to make brute-forcing passwords hard.

  2. They use a random value, called the salt, so that users having the same password will have different encrypted passwords. This is also an additional defense against reversing the algorithm.

  3. They include the algorithm type in the result, so passwords hashed with different algorithms can co-exist.

  4. Some of them are adaptive — that means when computers get faster, you can tune the algorithm to be slower, without introducing incompatibility with existing passwords.

Table F.16 lists the algorithms supported by the crypt() function.

Table F.16. Supported Algorithms for crypt()

Algorithm | Max Password Length | Adaptive? | Salt Bits | Output Length | Description
bf | 72 | yes | 128 | 60 | Blowfish-based, variant 2a
md5 | unlimited | no | 48 | 34 | MD5-based crypt
xdes | 8 | yes | 24 | 20 | Extended DES
des | 8 | no | 12 | 13 | Original UNIX crypt

F.26.2.1. crypt()

crypt(password text, salt text) returns text

Calculates a crypt(3)-style hash of password. When storing a new password, you need to use gen_salt() to generate a new salt value. To check a password, pass the stored hash value as salt, and test whether the result matches the stored value.

Example of setting a new password:

UPDATE ... SET pswhash = crypt('new password', gen_salt('md5'));

Example of authentication:

SELECT (pswhash = crypt('entered password', pswhash)) AS pswmatch FROM ... ;

This returns true if the entered password is correct.

F.26.2.2. gen_salt()

gen_salt(type text [, iter_count integer ]) returns text

Generates a new random salt string for use in crypt(). The salt string also tells crypt() which algorithm to use.

The type parameter specifies the hashing algorithm. The accepted types are: des, xdes, md5 and bf.

The iter_count parameter lets the user specify the iteration count, for algorithms that have one. The higher the count, the more time it takes to hash the password and therefore the more time to break it. Although with too high a count the time to calculate a hash may be several years — which is somewhat impractical. If the iter_count parameter is omitted, the default iteration count is used. Allowed values for iter_count depend on the algorithm and are shown in Table F.17.

Table F.17. Iteration Counts for crypt()

Algorithm | Default | Min | Max
xdes | 725 | 1 | 16777215
bf | 6 | 4 | 31

For xdes there is an additional limitation that the iteration count must be an odd number.

To pick an appropriate iteration count, consider that the original DES crypt was designed to have the speed of 4 hashes per second on the hardware of that time. Slower than 4 hashes per second would probably dampen usability. Faster than 100 hashes per second is probably too fast.

Table F.18 gives an overview of the relative slowness of different hashing algorithms. The table shows how much time it would take to try all combinations of characters in an 8-character password, assuming that the password contains either only lower case letters, or upper- and lower-case letters and numbers. In the crypt-bf entries, the number after a slash is the iter_count parameter of gen_salt.

Table F.18. Hash Algorithm Speeds

Algorithm | Hashes/sec | For [a-z] | For [A-Za-z0-9] | Duration relative to md5 hash
crypt-bf/8 | 1792 | 4 years | 3927 years | 100k
crypt-bf/7 | 3648 | 2 years | 1929 years | 50k
crypt-bf/6 | 7168 | 1 year | 982 years | 25k
crypt-bf/5 | 13504 | 188 days | 521 years | 12.5k
crypt-md5 | 171584 | 15 days | 41 years | 1k
crypt-des | 23221568 | 157.5 minutes | 108 days | 7
sha1 | 37774272 | 90 minutes | 68 days | 4
md5 (hash) | 150085504 | 22.5 minutes | 17 days | 1

Notes:

  • The machine used is an Intel Mobile Core i3.

  • crypt-des and crypt-md5 algorithm numbers are taken from John the Ripper v1.6.38 -test output.

  • md5 hash numbers are from mdcrack 1.2.

  • sha1 numbers are from lcrack-20031130-beta.

  • crypt-bf numbers are taken using a simple program that loops over 1000 8-character passwords. That way I can show the speed with different numbers of iterations. For reference: john -test shows 13506 loops/sec for crypt-bf/5. (The very small difference in results is in accordance with the fact that the crypt-bf implementation in pgcrypto is the same one used in John the Ripper.)

Note that “try all combinations” is not a realistic exercise. Usually password cracking is done with the help of dictionaries, which contain both regular words and various mutations of them. So, even somewhat word-like passwords could be cracked much faster than the above numbers suggest, while a 6-character non-word-like password may escape cracking. Or not.

F.26.3. PGP Encryption Functions

The functions here implement the encryption part of the OpenPGP (RFC 4880) standard. Supported are both symmetric-key and public-key encryption.

An encrypted PGP message consists of 2 parts, or packets:

  • Packet containing a session key — either symmetric-key or public-key encrypted.

  • Packet containing data encrypted with the session key.

When encrypting with a symmetric key (i.e., a password):

  1. The given password is hashed using a String2Key (S2K) algorithm. This is rather similar to crypt() algorithms — purposefully slow and with random salt — but it produces a full-length binary key.

  2. If a separate session key is requested, a new random key will be generated. Otherwise the S2K key will be used directly as the session key.

  3. If the S2K key is to be used directly, then only S2K settings will be put into the session key packet. Otherwise the session key will be encrypted with the S2K key and put into the session key packet.

When encrypting with a public key:

  1. A new random session key is generated.

  2. It is encrypted using the public key and put into the session key packet.

In either case the data to be encrypted is processed as follows:

  1. Optional data-manipulation: compression, conversion to UTF-8, and/or conversion of line-endings.

  2. The data is prefixed with a block of random bytes. This is equivalent to using a random IV.

  3. A SHA1 hash of the random prefix and data is appended.

  4. All this is encrypted with the session key and placed in the data packet.

F.26.3.1. pgp_sym_encrypt()

pgp_sym_encrypt(data text, psw text [, options text ]) returns bytea
pgp_sym_encrypt_bytea(data bytea, psw text [, options text ]) returns bytea

Encrypt data with a symmetric PGP key psw. The options parameter can contain option settings, as described below.

F.26.3.2. pgp_sym_decrypt()

pgp_sym_decrypt(msg bytea, psw text [, options text ]) returns text
pgp_sym_decrypt_bytea(msg bytea, psw text [, options text ]) returns bytea

Decrypt a symmetric-key-encrypted PGP message.

Decrypting bytea data with pgp_sym_decrypt is disallowed. This is to avoid outputting invalid character data. Decrypting originally textual data with pgp_sym_decrypt_bytea is fine.

The options parameter can contain option settings, as described below.

F.26.3.3. pgp_pub_encrypt()

pgp_pub_encrypt(data text, key bytea [, options text ]) returns bytea
pgp_pub_encrypt_bytea(data bytea, key bytea [, options text ]) returns bytea

Encrypt data with a public PGP key key. Giving this function a secret key will produce an error.

The options parameter can contain option settings, as described below.

F.26.3.4. pgp_pub_decrypt()

pgp_pub_decrypt(msg bytea, key bytea [, psw text [, options text ]]) returns text
pgp_pub_decrypt_bytea(msg bytea, key bytea [, psw text [, options text ]]) returns bytea

Decrypt a public-key-encrypted message. key must be the secret key corresponding to the public key that was used to encrypt. If the secret key is password-protected, you must give the password in psw. If there is no password, but you want to specify options, you need to give an empty password.

Decrypting bytea data with pgp_pub_decrypt is disallowed. This is to avoid outputting invalid character data. Decrypting originally textual data with pgp_pub_decrypt_bytea is fine.

The options parameter can contain option settings, as described below.

F.26.3.5. pgp_key_id()

pgp_key_id(bytea) returns text

pgp_key_id extracts the key ID of a PGP public or secret key. Or it gives the key ID that was used for encrypting the data, if given an encrypted message.

It can return 2 special key IDs:

  • SYMKEY

    The message is encrypted with a symmetric key.

  • ANYKEY

    The message is public-key encrypted, but the key ID has been removed. That means you will need to try all your secret keys on it to see which one decrypts it. pgcrypto itself does not produce such messages.

Note that different keys may have the same ID. This is rare but a normal event. The client application should then try to decrypt with each one, to see which fits — like handling ANYKEY.

F.26.3.6. armor(), dearmor()

armor(data bytea [ , keys text[], values text[] ]) returns text
dearmor(data text) returns bytea

These functions wrap/unwrap binary data into PGP ASCII-armor format, which is basically Base64 with CRC and additional formatting.

If the keys and values arrays are specified, an armor header is added to the armored format for each key/value pair. Both arrays must be single-dimensional, and they must be of the same length. The keys and values cannot contain any non-ASCII characters.

F.26.3.7. pgp_armor_headers

pgp_armor_headers(data text, key out text, value out text) returns setof record

pgp_armor_headers() extracts the armor headers from data. The return value is a set of rows with two columns, key and value. If the keys or values contain any non-ASCII characters, they are treated as UTF-8.

F.26.3.8. Options for PGP Functions

Options are named to be similar to GnuPG. An option's value should be given after an equal sign; separate options from each other with commas. For example:

pgp_sym_encrypt(data, psw, 'compress-algo=1, cipher-algo=aes256')

All of the options except convert-crlf apply only to encrypt functions. Decrypt functions get the parameters from the PGP data.

The most interesting options are probably compress-algo and unicode-mode. The rest should have reasonable defaults.

F.26.3.8.1. cipher-algo

Which cipher algorithm to use.

Values: bf, aes128, aes192, aes256 (OpenSSL-only: 3des, cast5)
Default: aes128
Applies to: pgp_sym_encrypt, pgp_pub_encrypt

F.26.3.8.2. compress-algo

Which compression algorithm to use. Only available if PostgreSQL was built with zlib.

Values:
0 - no compression
1 - ZIP compression
2 - ZLIB compression (= ZIP plus meta-data and block CRCs)
Default: 0
Applies to: pgp_sym_encrypt, pgp_pub_encrypt

F.26.3.8.3. compress-level

How much to compress. Higher levels compress smaller but are slower. 0 disables compression.

Values: 0, 1-9
Default: 6
Applies to: pgp_sym_encrypt, pgp_pub_encrypt

F.26.3.8.4. convert-crlf

Whether to convert \n into \r\n when encrypting and \r\n to \n when decrypting. RFC 4880 specifies that text data should be stored using \r\n line-feeds. Use this to get fully RFC-compliant behavior.

Values: 0, 1
Default: 0
Applies to: pgp_sym_encrypt, pgp_pub_encrypt, pgp_sym_decrypt, pgp_pub_decrypt

F.26.3.8.5. disable-mdc

Do not protect data with SHA-1. The only good reason to use this option is to achieve compatibility with ancient PGP products, predating the addition of SHA-1 protected packets to RFC 4880. Recent gnupg.org and pgp.com software supports it fine.

Values: 0, 1
Default: 0
Applies to: pgp_sym_encrypt, pgp_pub_encrypt

F.26.3.8.6. sess-key

Use separate session key. Public-key encryption always uses a separate session key; this option is for symmetric-key encryption, which by default uses the S2K key directly.

Values: 0, 1
Default: 0
Applies to: pgp_sym_encrypt

F.26.3.8.7. s2k-mode

Which S2K algorithm to use.

Values:
0 - Without salt. Dangerous!
1 - With salt but with fixed iteration count.
3 - Variable iteration count.
Default: 3
Applies to: pgp_sym_encrypt

F.26.3.8.8. s2k-count

The number of iterations of the S2K algorithm to use. It must be a value between 1024 and 65011712, inclusive.

Default: A random value between 65536 and 253952
Applies to: pgp_sym_encrypt, only with s2k-mode=3

F.26.3.8.9. s2k-digest-algo

Which digest algorithm to use in S2K calculation.

Values: md5, sha1
Default: sha1
Applies to: pgp_sym_encrypt

F.26.3.8.10. s2k-cipher-algo

Which cipher to use for encrypting separate session key.

Values: bf, aes, aes128, aes192, aes256
Default: use cipher-algo
Applies to: pgp_sym_encrypt

F.26.3.8.11. unicode-mode

Whether to convert textual data from database internal encoding to UTF-8 and back. If your database already is UTF-8, no conversion will be done, but the message will be tagged as UTF-8. Without this option it will not be.

Values: 0, 1
Default: 0
Applies to: pgp_sym_encrypt, pgp_pub_encrypt

F.26.3.9. Generating PGP Keys with GnuPG

To generate a new key:

gpg --gen-key

The preferred key type is “DSA and Elgamal”.

For RSA encryption you must create either DSA or RSA sign-only key as master and then add an RSA encryption subkey with gpg --edit-key.

To list keys:

gpg --list-secret-keys

To export a public key in ASCII-armor format:

gpg -a --export KEYID > public.key

To export a secret key in ASCII-armor format:

gpg -a --export-secret-keys KEYID > secret.key

You need to use dearmor() on these keys before giving them to the PGP functions. Or if you can handle binary data, you can drop -a from the command.

For more details see man gpg, The GNU Privacy Handbook and other documentation on https://www.gnupg.org/.

F.26.3.10. Limitations of PGP Code

  • No support for signing. That also means that it is not checked whether the encryption subkey belongs to the master key.

  • No support for encryption key as master key. As such practice is generally discouraged, this should not be a problem.

  • No support for several subkeys. This may seem like a problem, as this is common practice. On the other hand, you should not use your regular GPG/PGP keys with pgcrypto, but create new ones, as the usage scenario is rather different.

F.26.4. Raw Encryption Functions

These functions only run a cipher over data; they don't have any advanced features of PGP encryption. Therefore they have some major problems:

  1. They use user key directly as cipher key.

  2. They don't provide any integrity checking, to see if the encrypted data was modified.

  3. They expect that users manage all encryption parameters themselves, even IV.

  4. They don't handle text.

So, with the introduction of PGP encryption, usage of raw encryption functions is discouraged.

encrypt(data bytea, key bytea, type text) returns bytea
decrypt(data bytea, key bytea, type text) returns bytea

encrypt_iv(data bytea, key bytea, iv bytea, type text) returns bytea
decrypt_iv(data bytea, key bytea, iv bytea, type text) returns bytea

Encrypt/decrypt data using the cipher method specified by type. The syntax of the type string is:

algorithm [ - mode ] [ /pad: padding ]

where algorithm is one of:

  • bf — Blowfish

  • aes — AES (Rijndael-128, -192 or -256)

and mode is one of:

  • cbc — next block depends on previous (default)

  • ecb — each block is encrypted separately (for testing only)

and padding is one of:

  • pkcs — data may be any length (default)

  • none — data must be multiple of cipher block size

So, for example, these are equivalent:

encrypt(data, 'fooz', 'bf')
encrypt(data, 'fooz', 'bf-cbc/pad:pkcs')

In encrypt_iv and decrypt_iv, the iv parameter is the initial value for the CBC mode; it is ignored for ECB. It is clipped or padded with zeroes if not exactly block size. It defaults to all zeroes in the functions without this parameter.

F.26.5. Random-Data Functions

gen_random_bytes(count integer) returns bytea

Returns count cryptographically strong random bytes. At most 1024 bytes can be extracted at a time. This is to avoid draining the randomness generator pool.

gen_random_uuid() returns uuid

Returns a version 4 (random) UUID. (Obsolete, this function is now also included in core PostgreSQL.)

F.26.6. Notes

F.26.6.1. Configuration

pgcrypto configures itself according to the findings of the main PostgreSQL configure script. The options that affect it are --with-zlib and --with-ssl=openssl.

When compiled with zlib, PGP encryption functions are able to compress data before encrypting.

When compiled with OpenSSL, there will be more algorithms available. Also public-key encryption functions will be faster as OpenSSL has more optimized BIGNUM functions.

Table F.19. Summary of Functionality with and without OpenSSL

Functionality | Built-in | With OpenSSL
MD5 | yes | yes
SHA1 | yes | yes
SHA224/256/384/512 | yes | yes
Other digest algorithms | no | yes (Note 1)
Blowfish | yes | yes
AES | yes | yes
DES/3DES/CAST5 | no | yes
Raw encryption | yes | yes
PGP Symmetric encryption | yes | yes
PGP Public-Key encryption | yes | yes

When compiled against OpenSSL 3.0.0 and later versions, the legacy provider must be activated in the openssl.cnf configuration file in order to use older ciphers like DES or Blowfish.

Notes:

  1. Any digest algorithm OpenSSL supports is automatically picked up. This is not possible with ciphers, which need to be supported explicitly.

F.26.6.2. NULL Handling

As is standard in SQL, all functions return NULL, if any of the arguments are NULL. This may create security risks on careless usage.

F.26.6.3. Security Limitations

All pgcrypto functions run inside the database server. That means that all the data and passwords move between pgcrypto and client applications in clear text. Thus you must:

  1. Connect locally or use SSL connections.

  2. Trust both system and database administrator.

If you cannot, then better do crypto inside client application.

The implementation does not resist side-channel attacks. For example, the time required for a pgcrypto decryption function to complete varies among ciphertexts of a given size.

F.26.6.4. Useful Reading

  • https://www.gnupg.org/gph/en/manual.html

    The GNU Privacy Handbook.

  • https://www.openwall.com/crypt/

    Describes the crypt-blowfish algorithm.

  • https://www.iusmentis.com/security/passphrasefaq/

    How to choose a good password.

  • http://world.std.com/~reinhold/diceware.html

    Interesting idea for picking passwords.

  • http://www.interhack.net/people/cmcurtin/snake-oil-faq.html

    Describes good and bad cryptography.

F.26.6.5. Technical References

  • https://tools.ietf.org/html/rfc4880

    OpenPGP message format.

  • https://tools.ietf.org/html/rfc1321

    The MD5 Message-Digest Algorithm.

  • https://tools.ietf.org/html/rfc2104

    HMAC: Keyed-Hashing for Message Authentication.

  • https://www.usenix.org/legacy/events/usenix99/provos.html

    Comparison of crypt-des, crypt-md5 and bcrypt algorithms.

  • https://en.wikipedia.org/wiki/Fortuna_(PRNG)

    Description of Fortuna CSPRNG.

  • https://jlcooke.ca/random/

    Jean-Luc Cooke Fortuna-based /dev/random driver for Linux.

F.26.7. Author

Marko Kreen <markokr@gmail.com>

pgcrypto uses code from the following sources:

Algorithm | Author | Source origin
DES crypt | David Burren and others | FreeBSD libcrypt
MD5 crypt | Poul-Henning Kamp | FreeBSD libcrypt
Blowfish crypt | Solar Designer | www.openwall.com
Blowfish cipher | Simon Tatham | PuTTY
Rijndael cipher | Brian Gladman | OpenBSD sys/crypto
MD5 hash and SHA1 | WIDE Project | KAME kame/sys/crypto
SHA256/384/512 | Aaron D. Gifford | OpenBSD sys/crypto
BIGNUM math | Michael J. Fromberger | dartmouth.edu/~sting/sw/imath
Prev | Up | Next
F.25. pg_buffercache | Home | F.27. pg_freespacemap


F.27. pg_freespacemap
Prev Up Appendix F. Additional Supplied Modules Home Next

F.27. pg_freespacemap

F.27.1. Functions
F.27.2. Sample Output
F.27.3. Author

The pg_freespacemap module provides a means for examining the free space map (FSM). It provides a function called pg_freespace, or two overloaded functions, to be precise. The functions show the value recorded in the free space map for a given page, or for all pages in the relation.

By default use is restricted to superusers and members of the pg_stat_scan_tables role. Access may be granted to others using GRANT.

F.27.1. Functions

pg_freespace(rel regclass IN, blkno bigint IN) returns int2

Returns the amount of free space on the page of the relation, specified by blkno, according to the FSM.

pg_freespace(rel regclass IN, blkno OUT bigint, avail OUT int2)

Displays the amount of free space on each page of the relation, according to the FSM. A set of (blkno bigint, avail int2) tuples is returned, one tuple for each page in the relation.

The values stored in the free space map are not exact. They're rounded to precision of 1/256th of BLCKSZ (32 bytes with default BLCKSZ), and they're not kept fully up-to-date as tuples are inserted and updated.

For indexes, what is tracked is entirely-unused pages, rather than free space within pages. Therefore, the values are not meaningful, just whether a page is full or empty.

F.27.2. Sample Output

postgres=# SELECT * FROM pg_freespace('foo');
 blkno | avail
-------+-------
     0 |     0
     1 |     0
     2 |     0
     3 |    32
     4 |   704
     5 |   704
     6 |   704
     7 |  1216
     8 |   704
     9 |   704
    10 |   704
    11 |   704
    12 |   704
    13 |   704
    14 |   704
    15 |   704
    16 |   704
    17 |   704
    18 |   704
    19 |  3648
(20 rows)

postgres=# SELECT * FROM pg_freespace('foo', 7);
 pg_freespace
--------------
         1216
(1 row)

F.27.3. Author

Original version by Mark Kirkwood <markir@paradise.net.nz>. Rewritten in version 8.4 to suit new FSM implementation by Heikki Linnakangas <heikki@enterprisedb.com>

Prev | Up | Next
F.26. pgcrypto | Home | F.28. pg_prewarm


F.28. pg_prewarm
Prev Up Appendix F. Additional Supplied Modules Home Next

F.28. pg_prewarm

F.28.1. Functions
F.28.2. Configuration Parameters
F.28.3. Author

The pg_prewarm module provides a convenient way to load relation data into either the operating system buffer cache or the PostgreSQL buffer cache. Prewarming can be performed manually using the pg_prewarm function, or can be performed automatically by including pg_prewarm in shared_preload_libraries. In the latter case, the system will run a background worker which periodically records the contents of shared buffers in a file called autoprewarm.blocks and will, using 2 background workers, reload those same blocks after a restart.

F.28.1. Functions

pg_prewarm(regclass, mode text default 'buffer', fork text default 'main',
           first_block int8 default null,
           last_block int8 default null) RETURNS int8

The first argument is the relation to be prewarmed. The second argument is the prewarming method to be used, as further discussed below; the third is the relation fork to be prewarmed, usually main. The fourth argument is the first block number to prewarm (NULL is accepted as a synonym for zero). The fifth argument is the last block number to prewarm (NULL means prewarm through the last block in the relation). The return value is the number of blocks prewarmed.

There are three available prewarming methods. prefetch issues asynchronous prefetch requests to the operating system, if this is supported, or throws an error otherwise. read reads the requested range of blocks; unlike prefetch, this is synchronous and supported on all platforms and builds, but may be slower. buffer reads the requested range of blocks into the database buffer cache.

Note that with any of these methods, attempting to prewarm more blocks than can be cached — by the OS when using prefetch or read, or by PostgreSQL when using buffer — will likely result in lower-numbered blocks being evicted as higher numbered blocks are read in. Prewarmed data also enjoys no special protection from cache evictions, so it is possible that other system activity may evict the newly prewarmed blocks shortly after they are read; conversely, prewarming may also evict other data from cache. For these reasons, prewarming is typically most useful at startup, when caches are largely empty.

autoprewarm_start_worker() RETURNS void

Launch the main autoprewarm worker. This will normally happen automatically, but is useful if automatic prewarm was not configured at server startup time and you wish to start up the worker at a later time.

autoprewarm_dump_now() RETURNS int8

Update autoprewarm.blocks immediately. This may be useful if the autoprewarm worker is not running but you anticipate running it after the next restart. The return value is the number of records written to autoprewarm.blocks.

F.28.2. Configuration Parameters

pg_prewarm.autoprewarm (boolean)

Controls whether the server should run the autoprewarm worker. This is on by default. This parameter can only be set at server start.

pg_prewarm.autoprewarm_interval (int)

This is the interval between updates to autoprewarm.blocks. The default is 300 seconds. If set to 0, the file will not be dumped at regular intervals, but only when the server is shut down.

F.28.3. Author

Robert Haas <rhaas@postgresql.org>

Prev | Up | Next
F.27. pg_freespacemap | Home | F.29. pgrowlocks


F.29. pgrowlocks
Prev Up Appendix F. Additional Supplied Modules Home Next

F.29. pgrowlocks

F.29.1. Overview
F.29.2. Sample Output
F.29.3. Author

The pgrowlocks module provides a function to show row locking information for a specified table.

By default use is restricted to superusers, members of the pg_stat_scan_tables role, and users with SELECT permissions on the table.

F.29.1. Overview

pgrowlocks(text) returns setof record

The parameter is the name of a table. The result is a set of records, with one row for each locked row within the table. The output columns are shown in Table F.20.

Table F.20. pgrowlocks Output Columns

Name | Type | Description
locked_row | tid | Tuple ID (TID) of locked row
locker | xid | Transaction ID of locker, or multixact ID if multitransaction
multi | boolean | True if locker is a multitransaction
xids | xid[] | Transaction IDs of lockers (more than one if multitransaction)
modes | text[] | Lock mode of lockers (more than one if multitransaction), an array of Key Share, Share, For No Key Update, No Key Update, For Update, Update.
pids | integer[] | Process IDs of locking backends (more than one if multitransaction)

pgrowlocks takes AccessShareLock for the target table and reads each row one by one to collect the row locking information. This is not very speedy for a large table. Note that:

  1. If an ACCESS EXCLUSIVE lock is taken on the table, pgrowlocks will be blocked.

  2. pgrowlocks is not guaranteed to produce a self-consistent snapshot. It is possible that a new row lock is taken, or an old lock is freed, during its execution.

pgrowlocks does not show the contents of locked rows. If you want to take a look at the row contents at the same time, you could do something like this:

SELECT * FROM accounts AS a, pgrowlocks('accounts') AS p
  WHERE p.locked_row = a.ctid;

Be aware however that such a query will be very inefficient.

F.29.2. Sample Output

=# SELECT * FROM pgrowlocks('t1');
 locked_row | locker | multi | xids  |     modes      |  pids
------------+--------+-------+-------+----------------+--------
 (0,1)      |    609 | f     | {609} | {"For Share"}  | {3161}
 (0,2)      |    609 | f     | {609} | {"For Share"}  | {3161}
 (0,3)      |    607 | f     | {607} | {"For Update"} | {3107}
 (0,4)      |    607 | f     | {607} | {"For Update"} | {3107}
(4 rows)

F.29.3. Author

Tatsuo Ishii

Prev | Up | Next
F.28. pg_prewarm | Home | F.30. pg_stat_statements


F.30. pg_stat_statements
Prev Up Appendix F. Additional Supplied Modules Home Next

F.30. pg_stat_statements

F.30.1. The pg_stat_statements View
F.30.2. The pg_stat_statements_info View
F.30.3. Functions
F.30.4. Configuration Parameters
F.30.5. Sample Output
F.30.6. Authors

The pg_stat_statements module provides a means for tracking planning and execution statistics of all SQL statements executed by a server.

The module must be loaded by adding pg_stat_statements to shared_preload_libraries in postgresql.conf, because it requires additional shared memory. This means that a server restart is needed to add or remove the module. In addition, query identifier calculation must be enabled in order for the module to be active, which is done automatically if compute_query_id is set to auto or on, or any third-party module that calculates query identifiers is loaded.

When pg_stat_statements is active, it tracks statistics across all databases of the server. To access and manipulate these statistics, the module provides views pg_stat_statements and pg_stat_statements_info, and the utility functions pg_stat_statements_reset and pg_stat_statements. These are not available globally but can be enabled for a specific database with CREATE EXTENSION pg_stat_statements.

F.30.1. The pg_stat_statements View

The statistics gathered by the module are made available via a view named pg_stat_statements. This view contains one row for each distinct combination of database ID, user ID, query ID and whether it's a top-level statement or not (up to the maximum number of distinct statements that the module can track). The columns of the view are shown in Table F.21.

Table F.21. pg_stat_statements Columns

Column Type

Description

userid oid (references pg_authid.oid)

OID of user who executed the statement

dbid oid (references pg_database.oid)

OID of database in which the statement was executed

toplevel bool

True if the query was executed as a top-level statement (always true if pg_stat_statements.track is set to top)

queryid bigint

Hash code to identify identical normalized queries.

query text

Text of a representative statement

plans bigint

Number of times the statement was planned (if pg_stat_statements.track_planning is enabled, otherwise zero)

total_plan_time double precision

Total time spent planning the statement, in milliseconds (if pg_stat_statements.track_planning is enabled, otherwise zero)

min_plan_time double precision

Minimum time spent planning the statement, in milliseconds (if pg_stat_statements.track_planning is enabled, otherwise zero)

max_plan_time double precision

Maximum time spent planning the statement, in milliseconds (if pg_stat_statements.track_planning is enabled, otherwise zero)

mean_plan_time double precision

Mean time spent planning the statement, in milliseconds (if pg_stat_statements.track_planning is enabled, otherwise zero)

stddev_plan_time double precision

Population standard deviation of time spent planning the statement, in milliseconds (if pg_stat_statements.track_planning is enabled, otherwise zero)

calls bigint

Number of times the statement was executed

total_exec_time double precision

Total time spent executing the statement, in milliseconds

min_exec_time double precision

Minimum time spent executing the statement, in milliseconds

max_exec_time double precision

Maximum time spent executing the statement, in milliseconds

mean_exec_time double precision

Mean time spent executing the statement, in milliseconds

stddev_exec_time double precision

Population standard deviation of time spent executing the statement, in milliseconds

rows bigint

Total number of rows retrieved or affected by the statement

shared_blks_hit bigint

Total number of shared block cache hits by the statement

shared_blks_read bigint

Total number of shared blocks read by the statement

shared_blks_dirtied bigint

Total number of shared blocks dirtied by the statement

shared_blks_written bigint

Total number of shared blocks written by the statement

local_blks_hit bigint

Total number of local block cache hits by the statement

local_blks_read bigint

Total number of local blocks read by the statement

local_blks_dirtied bigint

Total number of local blocks dirtied by the statement

local_blks_written bigint

Total number of local blocks written by the statement

temp_blks_read bigint

Total number of temp blocks read by the statement

temp_blks_written bigint

Total number of temp blocks written by the statement

blk_read_time double precision

Total time the statement spent reading blocks, in milliseconds (if track_io_timing is enabled, otherwise zero)

blk_write_time double precision

Total time the statement spent writing blocks, in milliseconds (if track_io_timing is enabled, otherwise zero)

wal_records bigint

Total number of WAL records generated by the statement

wal_fpi bigint

Total number of WAL full page images generated by the statement

wal_bytes numeric

Total amount of WAL generated by the statement in bytes

For security reasons, only superusers and members of the pg_read_all_stats role are allowed to see the SQL text and queryid of queries executed by other users. Other users can see the statistics, however, if the view has been installed in their database.

Plannable queries (that is, SELECT, INSERT, UPDATE, and DELETE) are combined into a single pg_stat_statements entry whenever they have identical query structures according to an internal hash calculation. Typically, two queries will be considered the same for this purpose if they are semantically equivalent except for the values of literal constants appearing in the query. Utility commands (that is, all other commands) are compared strictly on the basis of their textual query strings, however.

Note

The following details about constant replacement and queryid only apply when compute_query_id is enabled. If you use an external module instead to compute queryid, you should refer to its documentation for details.

When a constant's value has been ignored for purposes of matching the query to other queries, the constant is replaced by a parameter symbol, such as $1, in the pg_stat_statements display. The rest of the query text is that of the first query that had the particular queryid hash value associated with the pg_stat_statements entry.

In some cases, queries with visibly different texts might get merged into a single pg_stat_statements entry. Normally this will happen only for semantically equivalent queries, but there is a small chance of hash collisions causing unrelated queries to be merged into one entry. (This cannot happen for queries belonging to different users or databases, however.)

Since the queryid hash value is computed on the post-parse-analysis representation of the queries, the opposite is also possible: queries with identical texts might appear as separate entries, if they have different meanings as a result of factors such as different search_path settings.

Consumers of pg_stat_statements may wish to use queryid (perhaps in combination with dbid and userid) as a more stable and reliable identifier for each entry than its query text. However, it is important to understand that there are only limited guarantees around the stability of the queryid hash value. Since the identifier is derived from the post-parse-analysis tree, its value is a function of, among other things, the internal object identifiers appearing in this representation. This has some counterintuitive implications. For example, pg_stat_statements will consider two apparently-identical queries to be distinct, if they reference a table that was dropped and recreated between the executions of the two queries. The hashing process is also sensitive to differences in machine architecture and other facets of the platform. Furthermore, it is not safe to assume that queryid will be stable across major versions of PostgreSQL.

As a rule of thumb, queryid values can be assumed to be stable and comparable only so long as the underlying server version and catalog metadata details stay exactly the same. Two servers participating in replication based on physical WAL replay can be expected to have identical queryid values for the same query. However, logical replication schemes do not promise to keep replicas identical in all relevant details, so queryid will not be a useful identifier for accumulating costs across a set of logical replicas. If in doubt, direct testing is recommended.

The parameter symbols used to replace constants in representative query texts start from the next number after the highest $n parameter in the original query text, or $1 if there was none. It's worth noting that in some cases there may be hidden parameter symbols that affect this numbering. For example, PL/pgSQL uses hidden parameter symbols to insert values of function local variables into queries, so that a PL/pgSQL statement like SELECT i + 1 INTO j would have representative text like SELECT i + $2.

The representative query texts are kept in an external disk file, and do not consume shared memory. Therefore, even very lengthy query texts can be stored successfully. However, if many long query texts are accumulated, the external file might grow unmanageably large. As a recovery method if that happens, pg_stat_statements may choose to discard the query texts, whereupon all existing entries in the pg_stat_statements view will show null query fields, though the statistics associated with each queryid are preserved. If this happens, consider reducing pg_stat_statements.max to prevent recurrences.

plans and calls aren't always expected to match because planning and execution statistics are updated at their respective end phase, and only for successful operations. For example, if a statement is successfully planned but fails during the execution phase, only its planning statistics will be updated. If planning is skipped because a cached plan is used, only its execution statistics will be updated.

F.30.2. The pg_stat_statements_info View

The statistics of the pg_stat_statements module itself are tracked and made available via a view named pg_stat_statements_info. This view contains only a single row. The columns of the view are shown in Table F.22.

Table F.22. pg_stat_statements_info Columns

Column Type

Description

dealloc bigint

Total number of times pg_stat_statements entries about the least-executed statements were deallocated because more distinct statements than pg_stat_statements.max were observed

stats_reset timestamp with time zone

Time at which all statistics in the pg_stat_statements view were last reset.

F.30.3. Functions

pg_stat_statements_reset(userid Oid, dbid Oid, queryid bigint) returns void

pg_stat_statements_reset discards statistics gathered so far by pg_stat_statements corresponding to the specified userid, dbid and queryid. If any of the parameters are not specified, the default value 0(invalid) is used for each of them and the statistics that match with other parameters will be reset. If no parameter is specified or all the specified parameters are 0(invalid), it will discard all statistics. If all statistics in the pg_stat_statements view are discarded, it will also reset the statistics in the pg_stat_statements_info view. By default, this function can only be executed by superusers. Access may be granted to others using GRANT.

pg_stat_statements(showtext boolean) returns setof record

The pg_stat_statements view is defined in terms of a function also named pg_stat_statements. It is possible for clients to call the pg_stat_statements function directly, and by specifying showtext := false have query text be omitted (that is, the OUT argument that corresponds to the view's query column will return nulls). This feature is intended to support external tools that might wish to avoid the overhead of repeatedly retrieving query texts of indeterminate length. Such tools can instead cache the first query text observed for each entry themselves, since that is all pg_stat_statements itself does, and then retrieve query texts only as needed. Since the server stores query texts in a file, this approach may reduce physical I/O for repeated examination of the pg_stat_statements data.

F.30.4. Configuration Parameters

pg_stat_statements.max (integer)

pg_stat_statements.max is the maximum number of statements tracked by the module (i.e., the maximum number of rows in the pg_stat_statements view). If more distinct statements than that are observed, information about the least-executed statements is discarded. The number of times such information was discarded can be seen in the pg_stat_statements_info view. The default value is 5000. This parameter can only be set at server start.

pg_stat_statements.track (enum)

pg_stat_statements.track controls which statements are counted by the module. Specify top to track top-level statements (those issued directly by clients), all to also track nested statements (such as statements invoked within functions), or none to disable statement statistics collection. The default value is top. Only superusers can change this setting.

pg_stat_statements.track_utility (boolean)

pg_stat_statements.track_utility controls whether utility commands are tracked by the module. Utility commands are all those other than SELECT, INSERT, UPDATE and DELETE. The default value is on. Only superusers can change this setting.

pg_stat_statements.track_planning (boolean)

pg_stat_statements.track_planning controls whether planning operations and duration are tracked by the module. Enabling this parameter may incur a noticeable performance penalty, especially when statements with identical query structure are executed by many concurrent connections which compete to update a small number of pg_stat_statements entries. The default value is off. Only superusers can change this setting.

pg_stat_statements.save (boolean)

pg_stat_statements.save specifies whether to save statement statistics across server shutdowns. If it is off then statistics are not saved at shutdown nor reloaded at server start. The default value is on. This parameter can only be set in the postgresql.conf file or on the server command line.

The module requires additional shared memory proportional to pg_stat_statements.max. Note that this memory is consumed whenever the module is loaded, even if pg_stat_statements.track is set to none.

These parameters must be set in postgresql.conf. Typical usage might be:

# postgresql.conf
shared_preload_libraries = 'pg_stat_statements'

compute_query_id = on
pg_stat_statements.max = 10000
pg_stat_statements.track = all

F.30.5. Sample Output

bench=# SELECT pg_stat_statements_reset();

$ pgbench -i bench
$ pgbench -c10 -t300 bench

bench=# \x
bench=# SELECT query, calls, total_exec_time, rows, 100.0 * shared_blks_hit /
               nullif(shared_blks_hit + shared_blks_read, 0) AS hit_percent
          FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 5;
-[ RECORD 1 ]---+--------------------------------------------------​------------------
query           | UPDATE pgbench_branches SET bbalance = bbalance + $1 WHERE bid = $2
calls           | 3000
total_exec_time | 25565.855387
rows            | 3000
hit_percent     | 100.0000000000000000
-[ RECORD 2 ]---+--------------------------------------------------​------------------
query           | UPDATE pgbench_tellers SET tbalance = tbalance + $1 WHERE tid = $2
calls           | 3000
total_exec_time | 20756.669379
rows            | 3000
hit_percent     | 100.0000000000000000
-[ RECORD 3 ]---+--------------------------------------------------​------------------
query           | copy pgbench_accounts from stdin
calls           | 1
total_exec_time | 291.865911
rows            | 100000
hit_percent     | 100.0000000000000000
-[ RECORD 4 ]---+--------------------------------------------------​------------------
query           | UPDATE pgbench_accounts SET abalance = abalance + $1 WHERE aid = $2
calls           | 3000
total_exec_time | 271.232977
rows            | 3000
hit_percent     | 98.8454011741682975
-[ RECORD 5 ]---+--------------------------------------------------​------------------
query           | alter table pgbench_accounts add primary key (aid)
calls           | 1
total_exec_time | 160.588563
rows            | 0
hit_percent     | 100.0000000000000000

bench=# SELECT pg_stat_statements_reset(0,0,s.queryid) FROM pg_stat_statements AS s
            WHERE s.query = 'UPDATE pgbench_branches SET bbalance = bbalance + $1 WHERE bid = $2';

bench=# SELECT query, calls, total_exec_time, rows, 100.0 * shared_blks_hit /
               nullif(shared_blks_hit + shared_blks_read, 0) AS hit_percent
          FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 5;
-[ RECORD 1 ]---+--------------------------------------------------​------------------
query           | UPDATE pgbench_tellers SET tbalance = tbalance + $1 WHERE tid = $2
calls           | 3000
total_exec_time | 20756.669379
rows            | 3000
hit_percent     | 100.0000000000000000
-[ RECORD 2 ]---+--------------------------------------------------​------------------
query           | copy pgbench_accounts from stdin
calls           | 1
total_exec_time | 291.865911
rows            | 100000
hit_percent     | 100.0000000000000000
-[ RECORD 3 ]---+--------------------------------------------------​------------------
query           | UPDATE pgbench_accounts SET abalance = abalance + $1 WHERE aid = $2
calls           | 3000
total_exec_time | 271.232977
rows            | 3000
hit_percent     | 98.8454011741682975
-[ RECORD 4 ]---+--------------------------------------------------​------------------
query           | alter table pgbench_accounts add primary key (aid)
calls           | 1
total_exec_time | 160.588563
rows            | 0
hit_percent     | 100.0000000000000000
-[ RECORD 5 ]---+--------------------------------------------------​------------------
query           | vacuum analyze pgbench_accounts
calls           | 1
total_exec_time | 136.448116
rows            | 0
hit_percent     | 99.9201915403032721

bench=# SELECT pg_stat_statements_reset(0,0,0);

bench=# SELECT query, calls, total_exec_time, rows, 100.0 * shared_blks_hit /
               nullif(shared_blks_hit + shared_blks_read, 0) AS hit_percent
          FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 5;
-[ RECORD 1 ]---+--------------------------------------------------​---------------------------
query           | SELECT pg_stat_statements_reset(0,0,0)
calls           | 1
total_exec_time | 0.189497
rows            | 1
hit_percent     |
-[ RECORD 2 ]---+--------------------------------------------------​---------------------------
query           | SELECT query, calls, total_exec_time, rows, $1 * shared_blks_hit /          +
                |                nullif(shared_blks_hit + shared_blks_read, $2) AS hit_percent+
                |           FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT $3
calls           | 0
total_exec_time | 0
rows            | 0
hit_percent     |

F.30.6. Authors

Takahiro Itagaki <itagaki.takahiro@oss.ntt.co.jp>. Query normalization added by Peter Geoghegan <peter@2ndquadrant.com>.

Prev | Up | Next
F.29. pgrowlocks | Home | F.31. pgstattuple


F.31. pgstattuple
Prev Up Appendix F. Additional Supplied Modules Home Next

F.31. pgstattuple

F.31.1. Functions
F.31.2. Authors

The pgstattuple module provides various functions to obtain tuple-level statistics.

Because these functions return detailed page-level information, access is restricted by default. By default, only the role pg_stat_scan_tables has EXECUTE privilege. Superusers of course bypass this restriction. After the extension has been installed, users may issue GRANT commands to change the privileges on the functions to allow others to execute them. However, it might be preferable to add those users to the pg_stat_scan_tables role instead.

F.31.1. Functions

pgstattuple(regclass) returns record

pgstattuple returns a relation's physical length, percentage of “dead” tuples, and other info. This may help users to determine whether vacuum is necessary or not. The argument is the target relation's name (optionally schema-qualified) or OID. For example:

test=> SELECT * FROM pgstattuple('pg_catalog.pg_proc');
-[ RECORD 1 ]------+-------
table_len          | 458752
tuple_count        | 1470
tuple_len          | 438896
tuple_percent      | 95.67
dead_tuple_count   | 11
dead_tuple_len     | 3157
dead_tuple_percent | 0.69
free_space         | 8932
free_percent       | 1.95

The output columns are described in Table F.23.

Table F.23. pgstattuple Output Columns

Column | Type | Description
table_len | bigint | Physical relation length in bytes
tuple_count | bigint | Number of live tuples
tuple_len | bigint | Total length of live tuples in bytes
tuple_percent | float8 | Percentage of live tuples
dead_tuple_count | bigint | Number of dead tuples
dead_tuple_len | bigint | Total length of dead tuples in bytes
dead_tuple_percent | float8 | Percentage of dead tuples
free_space | bigint | Total free space in bytes
free_percent | float8 | Percentage of free space

Note

The table_len will always be greater than the sum of the tuple_len, dead_tuple_len and free_space. The difference is accounted for by fixed page overhead, the per-page table of pointers to tuples, and padding to ensure that tuples are correctly aligned.

pgstattuple acquires only a read lock on the relation. So the results do not reflect an instantaneous snapshot; concurrent updates will affect them.

pgstattuple judges a tuple is “dead” if HeapTupleSatisfiesDirty returns false.

pgstattuple(text) returns record

This is the same as pgstattuple(regclass), except that the target relation is specified as TEXT. This function is kept because of backward-compatibility so far, and will be deprecated in some future release.

pgstatindex(regclass) returns record

pgstatindex returns a record showing information about a B-tree index. For example:

test=> SELECT * FROM pgstatindex('pg_cast_oid_index');
-[ RECORD 1 ]------+------
version            | 2
tree_level         | 0
index_size         | 16384
root_block_no      | 1
internal_pages     | 0
leaf_pages         | 1
empty_pages        | 0
deleted_pages      | 0
avg_leaf_density   | 54.27
leaf_fragmentation | 0

The output columns are:

Column | Type | Description
version | integer | B-tree version number
tree_level | integer | Tree level of the root page
index_size | bigint | Total index size in bytes
root_block_no | bigint | Location of root page (zero if none)
internal_pages | bigint | Number of “internal” (upper-level) pages
leaf_pages | bigint | Number of leaf pages
empty_pages | bigint | Number of empty pages
deleted_pages | bigint | Number of deleted pages
avg_leaf_density | float8 | Average density of leaf pages
leaf_fragmentation | float8 | Leaf page fragmentation

The reported index_size will normally correspond to one more page than is accounted for by internal_pages + leaf_pages + empty_pages + deleted_pages, because it also includes the index's metapage.

As with pgstattuple, the results are accumulated page-by-page, and should not be expected to represent an instantaneous snapshot of the whole index.

pgstatindex(text) returns record

This is the same as pgstatindex(regclass), except that the target index is specified as TEXT. This function is kept because of backward-compatibility so far, and will be deprecated in some future release.

pgstatginindex(regclass) returns record

pgstatginindex returns a record showing information about a GIN index. For example:

test=> SELECT * FROM pgstatginindex('test_gin_index');
-[ RECORD 1 ]--+--
version        | 1
pending_pages  | 0
pending_tuples | 0

The output columns are:

Column | Type | Description
version | integer | GIN version number
pending_pages | integer | Number of pages in the pending list
pending_tuples | bigint | Number of tuples in the pending list

pgstathashindex(regclass) returns record

pgstathashindex returns a record showing information about a HASH index. For example:

test=> select * from pgstathashindex('con_hash_index');
-[ RECORD 1 ]--+-----------------
version        | 4
bucket_pages   | 33081
overflow_pages | 0
bitmap_pages   | 1
unused_pages   | 32455
live_items     | 10204006
dead_items     | 0
free_percent   | 61.8005949100872

The output columns are:

Column | Type | Description
version | integer | HASH version number
bucket_pages | bigint | Number of bucket pages
overflow_pages | bigint | Number of overflow pages
bitmap_pages | bigint | Number of bitmap pages
unused_pages | bigint | Number of unused pages
live_items | bigint | Number of live tuples
dead_tuples | bigint | Number of dead tuples
free_percent | float | Percentage of free space

pg_relpages(regclass) returns bigint

pg_relpages returns the number of pages in the relation.

pg_relpages(text) returns bigint

This is the same as pg_relpages(regclass), except that the target relation is specified as TEXT. This function is kept because of backward-compatibility so far, and will be deprecated in some future release.

pgstattuple_approx(regclass) returns record

pgstattuple_approx is a faster alternative to pgstattuple that returns approximate results. The argument is the target relation's name or OID. For example:

test=> SELECT * FROM pgstattuple_approx('pg_catalog.pg_proc'::regclass);
-[ RECORD 1 ]--------+-------
table_len            | 573440
scanned_percent      | 2
approx_tuple_count   | 2740
approx_tuple_len     | 561210
approx_tuple_percent | 97.87
dead_tuple_count     | 0
dead_tuple_len       | 0
dead_tuple_percent   | 0
approx_free_space    | 11996
approx_free_percent  | 2.09

The output columns are described in Table F.24.

Whereas pgstattuple always performs a full-table scan and returns an exact count of live and dead tuples (and their sizes) and free space, pgstattuple_approx tries to avoid the full-table scan and returns exact dead tuple statistics along with an approximation of the number and size of live tuples and free space.

It does this by skipping pages that have only visible tuples according to the visibility map (if a page has the corresponding VM bit set, then it is assumed to contain no dead tuples). For such pages, it derives the free space value from the free space map, and assumes that the rest of the space on the page is taken up by live tuples.

For pages that cannot be skipped, it scans each tuple, recording its presence and size in the appropriate counters, and adding up the free space on the page. At the end, it estimates the total number of live tuples based on the number of pages and tuples scanned (in the same way that VACUUM estimates pg_class.reltuples).

Table F.24. pgstattuple_approx Output Columns

Column | Type | Description
table_len | bigint | Physical relation length in bytes (exact)
scanned_percent | float8 | Percentage of table scanned
approx_tuple_count | bigint | Number of live tuples (estimated)
approx_tuple_len | bigint | Total length of live tuples in bytes (estimated)
approx_tuple_percent | float8 | Percentage of live tuples
dead_tuple_count | bigint | Number of dead tuples (exact)
dead_tuple_len | bigint | Total length of dead tuples in bytes (exact)
dead_tuple_percent | float8 | Percentage of dead tuples
approx_free_space | bigint | Total free space in bytes (estimated)
approx_free_percent | float8 | Percentage of free space

In the above output, the free space figures may not match the pgstattuple output exactly, because the free space map gives us an exact figure, but is not guaranteed to be accurate to the byte.

F.31.2. Authors

Tatsuo Ishii, Satoshi Nagayasu and Abhijit Menon-Sen

Prev | Up | Next
F.30. pg_stat_statements | Home | F.32. pg_surgery


F.32. pg_surgery
Prev Up Appendix F. Additional Supplied Modules Home Next

F.32. pg_surgery

F.32.1. Functions
F.32.2. Authors

The pg_surgery module provides various functions to perform surgery on a damaged relation. These functions are unsafe by design and using them may corrupt (or further corrupt) your database. For example, these functions can easily be used to make a table inconsistent with its own indexes, to cause UNIQUE or FOREIGN KEY constraint violations, or even to make tuples visible which, when read, will cause a database server crash. They should be used with great caution and only as a last resort.

F.32.1. Functions

heap_force_kill(regclass, tid[]) returns void

heap_force_kill marks “used” line pointers as “dead” without examining the tuples. The intended use of this function is to forcibly remove tuples that are not otherwise accessible. For example:

test=> select * from t1 where ctid = '(0, 1)';
ERROR:  could not access status of transaction 4007513275
DETAIL:  Could not open file "pg_xact/0EED": No such file or directory.

test=# select heap_force_kill('t1'::regclass, ARRAY['(0, 1)']::tid[]);
 heap_force_kill
-----------------

(1 row)

test=# select * from t1 where ctid = '(0, 1)';
(0 rows)

heap_force_freeze(regclass, tid[]) returns void

heap_force_freeze marks tuples as frozen without examining the tuple data. The intended use of this function is to make accessible tuples which are inaccessible due to corrupted visibility information, or which prevent the table from being successfully vacuumed due to corrupted visibility information. For example:

test=> vacuum t1;
ERROR:  found xmin 507 from before relfrozenxid 515
CONTEXT:  while scanning block 0 of relation "public.t1"

test=# select ctid from t1 where xmin = 507;
 ctid
-------
 (0,3)
(1 row)

test=# select heap_force_freeze('t1'::regclass, ARRAY['(0, 3)']::tid[]);
 heap_force_freeze
-------------------

(1 row)

test=# select ctid from t1 where xmin = 2;
 ctid
-------
 (0,3)
(1 row)

F.32.2. Authors

Ashutosh Sharma <ashu.coek88@gmail.com>

Prev | Up | Next
F.31. pgstattuple | Home | F.33. pg_trgm


F.33. pg_trgm
Prev Up Appendix F. Additional Supplied Modules Home Next

F.33. pg_trgm

F.33.1. Trigram (or Trigraph) Concepts
F.33.2. Functions and Operators
F.33.3. GUC Parameters
F.33.4. Index Support
F.33.5. Text Search Integration
F.33.6. References
F.33.7. Authors

The pg_trgm module provides functions and operators for determining the similarity of alphanumeric text based on trigram matching, as well as index operator classes that support fast searching for similar strings.

This module is considered “trusted”, that is, it can be installed by non-superusers who have CREATE privilege on the current database.

F.33.1. Trigram (or Trigraph) Concepts

A trigram is a group of three consecutive characters taken from a string. We can measure the similarity of two strings by counting the number of trigrams they share. This simple idea turns out to be very effective for measuring the similarity of words in many natural languages.

Note

pg_trgm ignores non-word characters (non-alphanumerics) when extracting trigrams from a string. Each word is considered to have two spaces prefixed and one space suffixed when determining the set of trigrams contained in the string. For example, the set of trigrams in the string “cat” is “ c”, “ ca”, “cat”, and “at ”. The set of trigrams in the string “foo|bar” is “ f”, “ fo”, “foo”, “oo ”, “ b”, “ ba”, “bar”, and “ar ”.

F.33.2. Functions and Operators

The functions provided by the pg_trgm module are shown in Table F.25, the operators in Table F.26.

Table F.25. pg_trgm Functions

Function

Description

similarity ( text, text ) → real

Returns a number that indicates how similar the two arguments are. The range of the result is zero (indicating that the two strings are completely dissimilar) to one (indicating that the two strings are identical).

show_trgm ( text ) → text[]

Returns an array of all the trigrams in the given string. (In practice this is seldom useful except for debugging.)

word_similarity ( text, text ) → real

Returns a number that indicates the greatest similarity between the set of trigrams in the first string and any continuous extent of an ordered set of trigrams in the second string. For details, see the explanation below.

strict_word_similarity ( text, text ) → real

Same as word_similarity, but forces extent boundaries to match word boundaries. Since we don't have cross-word trigrams, this function actually returns greatest similarity between first string and any continuous extent of words of the second string.

show_limit () → real

Returns the current similarity threshold used by the % operator. This sets the minimum similarity between two words for them to be considered similar enough to be misspellings of each other, for example. (Deprecated; instead use SHOW pg_trgm.similarity_threshold.)

set_limit ( real ) → real

Sets the current similarity threshold that is used by the % operator. The threshold must be between 0 and 1 (default is 0.3). Returns the same value passed in. (Deprecated; instead use SET pg_trgm.similarity_threshold.)

Consider the following example:

# SELECT word_similarity('word', 'two words');
 word_similarity
-----------------
             0.8
(1 row)

In the first string, the set of trigrams is {" w"," wo","wor","ord","rd "}. In the second string, the ordered set of trigrams is {" t"," tw","two","wo "," w"," wo","wor","ord","rds","ds "}. The most similar extent of an ordered set of trigrams in the second string is {" w"," wo","wor","ord"}, and the similarity is 0.8.

This function returns a value that can be approximately understood as the greatest similarity between the first string and any substring of the second string. However, this function does not add padding to the boundaries of the extent. Thus, the number of additional characters present in the second string is not considered, except for the mismatched word boundaries.

At the same time, strict_word_similarity selects an extent of words in the second string. In the example above, strict_word_similarity would select the extent of a single word 'words', whose set of trigrams is {" w"," wo","wor","ord","rds","ds "}.

# SELECT strict_word_similarity('word', 'two words'), similarity('word', 'words');
 strict_word_similarity | similarity
------------------------+------------
               0.571429 |   0.571429
(1 row)

Thus, the strict_word_similarity function is useful for finding the similarity to whole words, while word_similarity is more suitable for finding the similarity for parts of words.

Table F.26. pg_trgm Operators

Operator

Description

text % text → boolean

Returns true if its arguments have a similarity that is greater than the current similarity threshold set by pg_trgm.similarity_threshold.

text <% text → boolean

Returns true if the similarity between the trigram set in the first argument and a continuous extent of an ordered trigram set in the second argument is greater than the current word similarity threshold set by pg_trgm.word_similarity_threshold parameter.

text %> text → boolean

Commutator of the <% operator.

text <<% text → boolean

Returns true if its second argument has a continuous extent of an ordered trigram set that matches word boundaries, and its similarity to the trigram set of the first argument is greater than the current strict word similarity threshold set by the pg_trgm.strict_word_similarity_threshold parameter.

text %>> text → boolean

Commutator of the <<% operator.

text <-> text → real

Returns the “distance” between the arguments, that is one minus the similarity() value.

text <<-> text → real

Returns the “distance” between the arguments, that is one minus the word_similarity() value.

text <->> text → real

Commutator of the <<-> operator.

text <<<-> text → real

Returns the “distance” between the arguments, that is one minus the strict_word_similarity() value.

text <->>> text → real

Commutator of the <<<-> operator.

F.33.3. GUC Parameters

pg_trgm.similarity_threshold (real)

Sets the current similarity threshold that is used by the % operator. The threshold must be between 0 and 1 (default is 0.3).

pg_trgm.word_similarity_threshold (real)

Sets the current word similarity threshold that is used by the <% and %> operators. The threshold must be between 0 and 1 (default is 0.6).

pg_trgm.strict_word_similarity_threshold (real)

Sets the current strict word similarity threshold that is used by the <<% and %>> operators. The threshold must be between 0 and 1 (default is 0.5).

F.33.4. Index Support

The pg_trgm module provides GiST and GIN index operator classes that allow you to create an index over a text column for the purpose of very fast similarity searches. These index types support the above-described similarity operators, and additionally support trigram-based index searches for LIKE, ILIKE, ~, ~* and = queries. Inequality operators are not supported. Note that those indexes may not be as efficient as regular B-tree indexes for equality operator.

Example:

CREATE TABLE test_trgm (t text);
CREATE INDEX trgm_idx ON test_trgm USING GIST (t gist_trgm_ops);

or

CREATE INDEX trgm_idx ON test_trgm USING GIN (t gin_trgm_ops);

gist_trgm_ops GiST opclass approximates a set of trigrams as a bitmap signature. Its optional integer parameter siglen determines the signature length in bytes. The default length is 12 bytes. Valid values of signature length are between 1 and 2024 bytes. Longer signatures lead to a more precise search (scanning a smaller fraction of the index and fewer heap pages), at the cost of a larger index.

Example of creating such an index with a signature length of 32 bytes:

CREATE INDEX trgm_idx ON test_trgm USING GIST (t gist_trgm_ops(siglen=32));

At this point, you will have an index on the t column that you can use for similarity searching. A typical query is

SELECT t, similarity(t, 'word') AS sml
  FROM test_trgm
  WHERE t % 'word'
  ORDER BY sml DESC, t;

This will return all values in the text column that are sufficiently similar to word, sorted from best match to worst. The index will be used to make this a fast operation even over very large data sets.

A variant of the above query is

SELECT t, t <-> 'word' AS dist
  FROM test_trgm
  ORDER BY dist LIMIT 10;

This can be implemented quite efficiently by GiST indexes, but not by GIN indexes. It will usually beat the first formulation when only a small number of the closest matches is wanted.

Also you can use an index on the t column for word similarity or strict word similarity. Typical queries are:

SELECT t, word_similarity('word', t) AS sml
  FROM test_trgm
  WHERE 'word' <% t
  ORDER BY sml DESC, t;

and

SELECT t, strict_word_similarity('word', t) AS sml
  FROM test_trgm
  WHERE 'word' <<% t
  ORDER BY sml DESC, t;

This will return all values in the text column for which there is a continuous extent in the corresponding ordered trigram set that is sufficiently similar to the trigram set of word, sorted from best match to worst. The index will be used to make this a fast operation even over very large data sets.

Possible variants of the above queries are:

SELECT t, 'word' <<-> t AS dist
  FROM test_trgm
  ORDER BY dist LIMIT 10;

and

SELECT t, 'word' <<<-> t AS dist
  FROM test_trgm
  ORDER BY dist LIMIT 10;

This can be implemented quite efficiently by GiST indexes, but not by GIN indexes.

Beginning in PostgreSQL 9.1, these index types also support index searches for LIKE and ILIKE, for example

SELECT * FROM test_trgm WHERE t LIKE '%foo%bar';

The index search works by extracting trigrams from the search string and then looking these up in the index. The more trigrams in the search string, the more effective the index search is. Unlike B-tree based searches, the search string need not be left-anchored.

Beginning in PostgreSQL 9.3, these index types also support index searches for regular-expression matches (~ and ~* operators), for example

SELECT * FROM test_trgm WHERE t ~ '(foo|bar)';

The index search works by extracting trigrams from the regular expression and then looking these up in the index. The more trigrams that can be extracted from the regular expression, the more effective the index search is. Unlike B-tree based searches, the search string need not be left-anchored.

For both LIKE and regular-expression searches, keep in mind that a pattern with no extractable trigrams will degenerate to a full-index scan.

The choice between GiST and GIN indexing depends on the relative performance characteristics of GiST and GIN, which are discussed elsewhere.

F.33.5. Text Search Integration

Trigram matching is a very useful tool when used in conjunction with a full text index. In particular it can help to recognize misspelled input words that will not be matched directly by the full text search mechanism.

The first step is to generate an auxiliary table containing all the unique words in the documents:

CREATE TABLE words AS SELECT word FROM
        ts_stat('SELECT to_tsvector(''simple'', bodytext) FROM documents');

where documents is a table that has a text field bodytext that we wish to search. The reason for using the simple configuration with the to_tsvector function, instead of using a language-specific configuration, is that we want a list of the original (unstemmed) words.

Next, create a trigram index on the word column:

CREATE INDEX words_idx ON words USING GIN (word gin_trgm_ops);

Now, a SELECT query similar to the previous example can be used to suggest spellings for misspelled words in user search terms. A useful extra test is to require that the selected words are also of similar length to the misspelled word.

Note

Since the words table has been generated as a separate, static table, it will need to be periodically regenerated so that it remains reasonably up-to-date with the document collection. Keeping it exactly current is usually unnecessary.

F.33.6. References

GiST Development Site http://www.sai.msu.su/~megera/postgres/gist/

Tsearch2 Development Site http://www.sai.msu.su/~megera/postgres/gist/tsearch/V2/

F.33.7. Authors

Oleg Bartunov <oleg@sai.msu.su>, Moscow, Moscow University, Russia

Teodor Sigaev <teodor@sigaev.ru>, Moscow, Delta-Soft Ltd.,Russia

Alexander Korotkov <a.korotkov@postgrespro.ru>, Moscow, Postgres Professional, Russia

Documentation: Christopher Kings-Lynne

This module is sponsored by Delta-Soft Ltd., Moscow, Russia.

Prev | Up | Next
F.32. pg_surgery | Home | F.34. pg_visibility


F.34. pg_visibility
Prev Up Appendix F. Additional Supplied Modules Home Next

F.34. pg_visibility

F.34.1. Functions
F.34.2. Author

The pg_visibility module provides a means for examining the visibility map (VM) and page-level visibility information of a table. It also provides functions to check the integrity of a visibility map and to force it to be rebuilt.

Three different bits are used to store information about page-level visibility. The all-visible bit in the visibility map indicates that every tuple in the corresponding page of the relation is visible to every current and future transaction. The all-frozen bit in the visibility map indicates that every tuple in the page is frozen; that is, no future vacuum will need to modify the page until such time as a tuple is inserted, updated, deleted, or locked on that page. The page header's PD_ALL_VISIBLE bit has the same meaning as the all-visible bit in the visibility map, but is stored within the data page itself rather than in a separate data structure. These two bits will normally agree, but the page's all-visible bit can sometimes be set while the visibility map bit is clear after a crash recovery. The reported values can also disagree because of a change that occurs after pg_visibility examines the visibility map and before it examines the data page. Any event that causes data corruption can also cause these bits to disagree.

Functions that display information about PD_ALL_VISIBLE bits are much more costly than those that only consult the visibility map, because they must read the relation's data blocks rather than only the (much smaller) visibility map. Functions that check the relation's data blocks are similarly expensive.

F.34.1. Functions

pg_visibility_map(relation regclass, blkno bigint, all_visible OUT boolean, all_frozen OUT boolean) returns record

Returns the all-visible and all-frozen bits in the visibility map for the given block of the given relation.

pg_visibility(relation regclass, blkno bigint, all_visible OUT boolean, all_frozen OUT boolean, pd_all_visible OUT boolean) returns record

Returns the all-visible and all-frozen bits in the visibility map for the given block of the given relation, plus the PD_ALL_VISIBLE bit of that block.

pg_visibility_map(relation regclass, blkno OUT bigint, all_visible OUT boolean, all_frozen OUT boolean) returns setof record

Returns the all-visible and all-frozen bits in the visibility map for each block of the given relation.

pg_visibility(relation regclass, blkno OUT bigint, all_visible OUT boolean, all_frozen OUT boolean, pd_all_visible OUT boolean) returns setof record

Returns the all-visible and all-frozen bits in the visibility map for each block of the given relation, plus the PD_ALL_VISIBLE bit of each block.

pg_visibility_map_summary(relation regclass, all_visible OUT bigint, all_frozen OUT bigint) returns record

Returns the number of all-visible pages and the number of all-frozen pages in the relation according to the visibility map.

pg_check_frozen(relation regclass, t_ctid OUT tid) returns setof tid

Returns the TIDs of non-frozen tuples stored in pages marked all-frozen in the visibility map. If this function returns a non-empty set of TIDs, the visibility map is corrupt.

pg_check_visible(relation regclass, t_ctid OUT tid) returns setof tid

Returns the TIDs of non-all-visible tuples stored in pages marked all-visible in the visibility map. If this function returns a non-empty set of TIDs, the visibility map is corrupt.

pg_truncate_visibility_map(relation regclass) returns void

Truncates the visibility map for the given relation. This function is useful if you believe that the visibility map for the relation is corrupt and wish to force rebuilding it. The first VACUUM executed on the given relation after this function is executed will scan every page in the relation and rebuild the visibility map. (Until that is done, queries will treat the visibility map as containing all zeroes.)

By default, these functions are executable only by superusers and members of the pg_stat_scan_tables role, with the exception of pg_truncate_visibility_map(relation regclass) which can only be executed by superusers.

F.34.2. Author

Robert Haas <rhaas@postgresql.org>

Prev | Up | Next
F.33. pg_trgm | Home | F.35. postgres_fdw


F.35. postgres_fdw
Prev Up Appendix F. Additional Supplied Modules Home Next

F.35. postgres_fdw

F.35.1. FDW Options of postgres_fdw
F.35.2. Functions
F.35.3. Connection Management
F.35.4. Transaction Management
F.35.5. Remote Query Optimization
F.35.6. Remote Query Execution Environment
F.35.7. Cross-Version Compatibility
F.35.8. Examples
F.35.9. Author

The postgres_fdw module provides the foreign-data wrapper postgres_fdw, which can be used to access data stored in external PostgreSQL servers.

The functionality provided by this module overlaps substantially with the functionality of the older dblink module. But postgres_fdw provides more transparent and standards-compliant syntax for accessing remote tables, and can give better performance in many cases.

To prepare for remote access using postgres_fdw:

  1. Install the postgres_fdw extension using CREATE EXTENSION.

  2. Create a foreign server object, using CREATE SERVER, to represent each remote database you want to connect to. Specify connection information, except user and password, as options of the server object.

  3. Create a user mapping, using CREATE USER MAPPING, for each database user you want to allow to access each foreign server. Specify the remote user name and password to use as user and password options of the user mapping.

  4. Create a foreign table, using CREATE FOREIGN TABLE or IMPORT FOREIGN SCHEMA, for each remote table you want to access. The columns of the foreign table must match the referenced remote table. You can, however, use table and/or column names different from the remote table's, if you specify the correct remote names as options of the foreign table object.

Now you need only SELECT from a foreign table to access the data stored in its underlying remote table. You can also modify the remote table using INSERT, UPDATE, DELETE, COPY, or TRUNCATE. (Of course, the remote user you have specified in your user mapping must have privileges to do these things.)

Note that the ONLY option specified in SELECT, UPDATE, DELETE or TRUNCATE has no effect when accessing or modifying the remote table.

Note that postgres_fdw currently lacks support for INSERT statements with an ON CONFLICT DO UPDATE clause. However, the ON CONFLICT DO NOTHING clause is supported, provided a unique index inference specification is omitted. Note also that postgres_fdw supports row movement invoked by UPDATE statements executed on partitioned tables, but it currently does not handle the case where a remote partition chosen to insert a moved row into is also an UPDATE target partition that will be updated elsewhere in the same command.

It is generally recommended that the columns of a foreign table be declared with exactly the same data types, and collations if applicable, as the referenced columns of the remote table. Although postgres_fdw is currently rather forgiving about performing data type conversions at need, surprising semantic anomalies may arise when types or collations do not match, due to the remote server interpreting query conditions differently from the local server.

Note that a foreign table can be declared with fewer columns, or with a different column order, than its underlying remote table has. Matching of columns to the remote table is by name, not position.

F.35.1. FDW Options of postgres_fdw

F.35.1.1. Connection Options

A foreign server using the postgres_fdw foreign data wrapper can have the same options that libpq accepts in connection strings, as described in Section 34.1.2, except that these options are not allowed or have special handling:

  • user, password and sslpassword (specify these in a user mapping, instead, or use a service file)

  • client_encoding (this is automatically set from the local server encoding)

  • fallback_application_name (always set to postgres_fdw)

  • sslkey and sslcert - these may appear in either or both a connection and a user mapping. If both are present, the user mapping setting overrides the connection setting.

Only superusers may create or modify user mappings with the sslcert or sslkey settings.

Only superusers may connect to foreign servers without password authentication, so always specify the password option for user mappings belonging to non-superusers.

A superuser may override this check on a per-user-mapping basis by setting the user mapping option password_required 'false', e.g.,

ALTER USER MAPPING FOR some_non_superuser SERVER loopback_nopw
OPTIONS (ADD password_required 'false');

To prevent unprivileged users from exploiting the authentication rights of the unix user the postgres server is running as to escalate to superuser rights, only the superuser may set this option on a user mapping.

Care is required to ensure that this does not allow the mapped user the ability to connect as superuser to the mapped database per CVE-2007-3278 and CVE-2007-6601. Don't set password_required=false on the public role. Keep in mind that the mapped user can potentially use any client certificates, .pgpass, .pg_service.conf etc in the unix home directory of the system user the postgres server runs as. They can also use any trust relationship granted by authentication modes like peer or ident authentication.

F.35.1.2. Object Name Options

These options can be used to control the names used in SQL statements sent to the remote PostgreSQL server. These options are needed when a foreign table is created with names different from the underlying remote table's names.

schema_name

This option, which can be specified for a foreign table, gives the schema name to use for the foreign table on the remote server. If this option is omitted, the name of the foreign table's schema is used.

table_name

This option, which can be specified for a foreign table, gives the table name to use for the foreign table on the remote server. If this option is omitted, the foreign table's name is used.

column_name

This option, which can be specified for a column of a foreign table, gives the column name to use for the column on the remote server. If this option is omitted, the column's name is used.

F.35.1.3. Cost Estimation Options

postgres_fdw retrieves remote data by executing queries against remote servers, so ideally the estimated cost of scanning a foreign table should be whatever it costs to be done on the remote server, plus some overhead for communication. The most reliable way to get such an estimate is to ask the remote server and then add something for overhead — but for simple queries, it may not be worth the cost of an additional remote query to get a cost estimate. So postgres_fdw provides the following options to control how cost estimation is done:

use_remote_estimate

This option, which can be specified for a foreign table or a foreign server, controls whether postgres_fdw issues remote EXPLAIN commands to obtain cost estimates. A setting for a foreign table overrides any setting for its server, but only for that table. The default is false.

fdw_startup_cost

This option, which can be specified for a foreign server, is a floating point value that is added to the estimated startup cost of any foreign-table scan on that server. This represents the additional overhead of establishing a connection, parsing and planning the query on the remote side, etc. The default value is 100.

fdw_tuple_cost

This option, which can be specified for a foreign server, is a floating point value that is used as extra cost per-tuple for foreign-table scans on that server. This represents the additional overhead of data transfer between servers. You might increase or decrease this number to reflect higher or lower network delay to the remote server. The default value is 0.01.

When use_remote_estimate is true, postgres_fdw obtains row count and cost estimates from the remote server and then adds fdw_startup_cost and fdw_tuple_cost to the cost estimates. When use_remote_estimate is false, postgres_fdw performs local row count and cost estimation and then adds fdw_startup_cost and fdw_tuple_cost to the cost estimates. This local estimation is unlikely to be very accurate unless local copies of the remote table's statistics are available. Running ANALYZE on the foreign table is the way to update the local statistics; this will perform a scan of the remote table and then calculate and store statistics just as though the table were local. Keeping local statistics can be a useful way to reduce per-query planning overhead for a remote table — but if the remote table is frequently updated, the local statistics will soon be obsolete.

F.35.1.4. Remote Execution Options

By default, only WHERE clauses using built-in operators and functions will be considered for execution on the remote server. Clauses involving non-built-in functions are checked locally after rows are fetched. If such functions are available on the remote server and can be relied on to produce the same results as they do locally, performance can be improved by sending such WHERE clauses for remote execution. This behavior can be controlled using the following option:

extensions

This option is a comma-separated list of names of PostgreSQL extensions that are installed, in compatible versions, on both the local and remote servers. Functions and operators that are immutable and belong to a listed extension will be considered shippable to the remote server. This option can only be specified for foreign servers, not per-table.

When using the extensions option, it is the user's responsibility that the listed extensions exist and behave identically on both the local and remote servers. Otherwise, remote queries may fail or behave unexpectedly.

fetch_size

This option specifies the number of rows postgres_fdw should get in each fetch operation. It can be specified for a foreign table or a foreign server. The option specified on a table overrides an option specified for the server. The default is 100.

batch_size

This option specifies the number of rows postgres_fdw should insert in each insert operation. It can be specified for a foreign table or a foreign server. The option specified on a table overrides an option specified for the server. The default is 1.

Note the actual number of rows postgres_fdw inserts at once depends on the number of columns and the provided batch_size value. The batch is executed as a single query, and the libpq protocol (which postgres_fdw uses to connect to a remote server) limits the number of parameters in a single query to 65535. When the number of columns * batch_size exceeds the limit, the batch_size will be adjusted to avoid an error.

F.35.1.5. Asynchronous Execution Options

postgres_fdw supports asynchronous execution, which runs multiple parts of an Append node concurrently rather than serially to improve performance. This execution can be controlled using the following option:

async_capable

This option controls whether postgres_fdw allows foreign tables to be scanned concurrently for asynchronous execution. It can be specified for a foreign table or a foreign server. A table-level option overrides a server-level option. The default is false.

In order to ensure that the data being returned from a foreign server is consistent, postgres_fdw will only open one connection for a given foreign server and will run all queries against that server sequentially even if there are multiple foreign tables involved, unless those tables are subject to different user mappings. In such a case, it may be more performant to disable this option to eliminate the overhead associated with running queries asynchronously.

Asynchronous execution is applied even when an Append node contains subplan(s) executed synchronously as well as subplan(s) executed asynchronously. In such a case, if the asynchronous subplans are ones processed using postgres_fdw, tuples from the asynchronous subplans are not returned until after at least one synchronous subplan returns all tuples, as that subplan is executed while the asynchronous subplans are waiting for the results of asynchronous queries sent to foreign servers. This behavior might change in a future release.

F.35.1.6. Updatability Options

By default all foreign tables using postgres_fdw are assumed to be updatable. This may be overridden using the following option:

updatable

This option controls whether postgres_fdw allows foreign tables to be modified using INSERT, UPDATE and DELETE commands. It can be specified for a foreign table or a foreign server. A table-level option overrides a server-level option. The default is true.

Of course, if the remote table is not in fact updatable, an error would occur anyway. Use of this option primarily allows the error to be thrown locally without querying the remote server. Note however that the information_schema views will report a postgres_fdw foreign table to be updatable (or not) according to the setting of this option, without any check of the remote server.

F.35.1.7. Truncatability Options

By default all foreign tables using postgres_fdw are assumed to be truncatable. This may be overridden using the following option:

truncatable

This option controls whether postgres_fdw allows foreign tables to be truncated using the TRUNCATE command. It can be specified for a foreign table or a foreign server. A table-level option overrides a server-level option. The default is true.

Of course, if the remote table is not in fact truncatable, an error would occur anyway. Use of this option primarily allows the error to be thrown locally without querying the remote server.

F.35.1.8. Importing Options

postgres_fdw is able to import foreign table definitions using IMPORT FOREIGN SCHEMA. This command creates foreign table definitions on the local server that match tables or views present on the remote server. If the remote tables to be imported have columns of user-defined data types, the local server must have compatible types of the same names.

Importing behavior can be customized with the following options (given in the IMPORT FOREIGN SCHEMA command):

import_collate

This option controls whether column COLLATE options are included in the definitions of foreign tables imported from a foreign server. The default is true. You might need to turn this off if the remote server has a different set of collation names than the local server does, which is likely to be the case if it's running on a different operating system. If you do so, however, there is a very severe risk that the imported table columns' collations will not match the underlying data, resulting in anomalous query behavior.

Even when this parameter is set to true, importing columns whose collation is the remote server's default can be risky. They will be imported with COLLATE "default", which will select the local server's default collation, which could be different.

import_default

This option controls whether column DEFAULT expressions are included in the definitions of foreign tables imported from a foreign server. The default is false. If you enable this option, be wary of defaults that might get computed differently on the local server than they would be on the remote server; nextval() is a common source of problems. The IMPORT will fail altogether if an imported default expression uses a function or operator that does not exist locally.

import_generated

This option controls whether column GENERATED expressions are included in the definitions of foreign tables imported from a foreign server. The default is true. The IMPORT will fail altogether if an imported generated expression uses a function or operator that does not exist locally.

import_not_null

This option controls whether column NOT NULL constraints are included in the definitions of foreign tables imported from a foreign server. The default is true.

Note that constraints other than NOT NULL will never be imported from the remote tables. Although PostgreSQL does support check constraints on foreign tables, there is no provision for importing them automatically, because of the risk that a constraint expression could evaluate differently on the local and remote servers. Any such inconsistency in the behavior of a check constraint could lead to hard-to-detect errors in query optimization. So if you wish to import check constraints, you must do so manually, and you should verify the semantics of each one carefully. For more detail about the treatment of check constraints on foreign tables, see CREATE FOREIGN TABLE.

Tables or foreign tables which are partitions of some other table are imported only when they are explicitly specified in LIMIT TO clause. Otherwise they are automatically excluded from IMPORT FOREIGN SCHEMA. Since all data can be accessed through the partitioned table which is the root of the partitioning hierarchy, importing only partitioned tables should allow access to all the data without creating extra objects.

F.35.1.9. Connection Management Options

By default, all connections that postgres_fdw establishes to foreign servers are kept open in the local session for re-use.

keep_connections

This option controls whether postgres_fdw keeps the connections to the foreign server open so that subsequent queries can re-use them. It can only be specified for a foreign server. The default is on. If set to off, all connections to this foreign server will be discarded at the end of each transaction.

F.35.2. Functions

postgres_fdw_get_connections(OUT server_name text, OUT valid boolean) returns setof record

This function returns the foreign server names of all the open connections that postgres_fdw established from the local session to the foreign servers. It also returns whether each connection is valid or not. false is returned if the foreign server connection is used in the current local transaction but its foreign server or user mapping is changed or dropped (Note that server name of an invalid connection will be NULL if the server is dropped), and then such invalid connection will be closed at the end of that transaction. true is returned otherwise. If there are no open connections, no record is returned. Example usage of the function:

postgres=# SELECT * FROM postgres_fdw_get_connections() ORDER BY 1;
 server_name | valid
-------------+-------
 loopback1   | t
 loopback2   | f

postgres_fdw_disconnect(server_name text) returns boolean

This function discards the open connections that are established by postgres_fdw from the local session to the foreign server with the given name. Note that there can be multiple connections to the given server using different user mappings. If the connections are used in the current local transaction, they are not disconnected and warning messages are reported. This function returns true if it disconnects at least one connection, otherwise false. If no foreign server with the given name is found, an error is reported. Example usage of the function:

postgres=# SELECT postgres_fdw_disconnect('loopback1');
 postgres_fdw_disconnect
-------------------------
 t

postgres_fdw_disconnect_all() returns boolean

This function discards all the open connections that are established by postgres_fdw from the local session to foreign servers. If the connections are used in the current local transaction, they are not disconnected and warning messages are reported. This function returns true if it disconnects at least one connection, otherwise false. Example usage of the function:

postgres=# SELECT postgres_fdw_disconnect_all();
 postgres_fdw_disconnect_all
-----------------------------
 t

F.35.3. Connection Management

postgres_fdw establishes a connection to a foreign server during the first query that uses a foreign table associated with the foreign server. By default this connection is kept and re-used for subsequent queries in the same session. This behavior can be controlled using keep_connections option for a foreign server. If multiple user identities (user mappings) are used to access the foreign server, a connection is established for each user mapping.

When changing the definition of or removing a foreign server or a user mapping, the associated connections are closed. But note that if any connections are in use in the current local transaction, they are kept until the end of the transaction. Closed connections will be re-established when they are necessary by future queries using a foreign table.

Once a connection to a foreign server has been established, it's by default kept until the local or corresponding remote session exits. To disconnect a connection explicitly, keep_connections option for a foreign server may be disabled, or postgres_fdw_disconnect and postgres_fdw_disconnect_all functions may be used. For example, these are useful to close connections that are no longer necessary, thereby releasing connections on the foreign server.

F.35.4. Transaction Management

During a query that references any remote tables on a foreign server, postgres_fdw opens a transaction on the remote server if one is not already open corresponding to the current local transaction. The remote transaction is committed or aborted when the local transaction commits or aborts. Savepoints are similarly managed by creating corresponding remote savepoints.

The remote transaction uses SERIALIZABLE isolation level when the local transaction has SERIALIZABLE isolation level; otherwise it uses REPEATABLE READ isolation level. This choice ensures that if a query performs multiple table scans on the remote server, it will get snapshot-consistent results for all the scans. A consequence is that successive queries within a single transaction will see the same data from the remote server, even if concurrent updates are occurring on the remote server due to other activities. That behavior would be expected anyway if the local transaction uses SERIALIZABLE or REPEATABLE READ isolation level, but it might be surprising for a READ COMMITTED local transaction. A future PostgreSQL release might modify these rules.

Note that it is currently not supported by postgres_fdw to prepare the remote transaction for two-phase commit.

F.35.5. Remote Query Optimization

postgres_fdw attempts to optimize remote queries to reduce the amount of data transferred from foreign servers. This is done by sending query WHERE clauses to the remote server for execution, and by not retrieving table columns that are not needed for the current query. To reduce the risk of misexecution of queries, WHERE clauses are not sent to the remote server unless they use only data types, operators, and functions that are built-in or belong to an extension that's listed in the foreign server's extensions option. Operators and functions in such clauses must be IMMUTABLE as well. For an UPDATE or DELETE query, postgres_fdw attempts to optimize the query execution by sending the whole query to the remote server if there are no query WHERE clauses that cannot be sent to the remote server, no local joins for the query, no row-level local BEFORE or AFTER triggers or stored generated columns on the target table, and no CHECK OPTION constraints from parent views. In UPDATE, expressions to assign to target columns must use only built-in data types, IMMUTABLE operators, or IMMUTABLE functions, to reduce the risk of misexecution of the query.

When postgres_fdw encounters a join between foreign tables on the same foreign server, it sends the entire join to the foreign server, unless for some reason it believes that it will be more efficient to fetch rows from each table individually, or unless the table references involved are subject to different user mappings. While sending the JOIN clauses, it takes the same precautions as mentioned above for the WHERE clauses.

The query that is actually sent to the remote server for execution can be examined using EXPLAIN VERBOSE.

F.35.6. Remote Query Execution Environment

In the remote sessions opened by postgres_fdw, the search_path parameter is set to just pg_catalog, so that only built-in objects are visible without schema qualification. This is not an issue for queries generated by postgres_fdw itself, because it always supplies such qualification. However, this can pose a hazard for functions that are executed on the remote server via triggers or rules on remote tables. For example, if a remote table is actually a view, any functions used in that view will be executed with the restricted search path. It is recommended to schema-qualify all names in such functions, or else attach SET search_path options (see CREATE FUNCTION) to such functions to establish their expected search path environment.

postgres_fdw likewise establishes remote session settings for various parameters:

  • TimeZone is set to UTC

  • DateStyle is set to ISO

  • IntervalStyle is set to postgres

  • extra_float_digits is set to 3 for remote servers 9.0 and newer and is set to 2 for older versions

These are less likely to be problematic than search_path, but can be handled with function SET options if the need arises.

It is not recommended that you override this behavior by changing the session-level settings of these parameters; that is likely to cause postgres_fdw to malfunction.

F.35.7. Cross-Version Compatibility

postgres_fdw can be used with remote servers dating back to PostgreSQL 8.3. Read-only capability is available back to 8.1. A limitation however is that postgres_fdw generally assumes that immutable built-in functions and operators are safe to send to the remote server for execution, if they appear in a WHERE clause for a foreign table. Thus, a built-in function that was added since the remote server's release might be sent to it for execution, resulting in “function does not exist” or a similar error. This type of failure can be worked around by rewriting the query, for example by embedding the foreign table reference in a sub-SELECT with OFFSET 0 as an optimization fence, and placing the problematic function or operator outside the sub-SELECT.

F.35.8. Examples

Here is an example of creating a foreign table with postgres_fdw. First install the extension:

CREATE EXTENSION postgres_fdw;

Then create a foreign server using CREATE SERVER. In this example we wish to connect to a PostgreSQL server on host 192.83.123.89 listening on port 5432. The database to which the connection is made is named foreign_db on the remote server:

CREATE SERVER foreign_server
        FOREIGN DATA WRAPPER postgres_fdw
        OPTIONS (host '192.83.123.89', port '5432', dbname 'foreign_db');

A user mapping, defined with CREATE USER MAPPING, is needed as well to identify the role that will be used on the remote server:

CREATE USER MAPPING FOR local_user
        SERVER foreign_server
        OPTIONS (user 'foreign_user', password 'password');

Now it is possible to create a foreign table with CREATE FOREIGN TABLE. In this example we wish to access the table named some_schema.some_table on the remote server. The local name for it will be foreign_table:

CREATE FOREIGN TABLE foreign_table (
        id integer NOT NULL,
        data text
)
        SERVER foreign_server
        OPTIONS (schema_name 'some_schema', table_name 'some_table');

It's essential that the data types and other properties of the columns declared in CREATE FOREIGN TABLE match the actual remote table. Column names must match as well, unless you attach column_name options to the individual columns to show how they are named in the remote table. In many cases, use of IMPORT FOREIGN SCHEMA is preferable to constructing foreign table definitions manually.

F.35.9. Author

Shigeru Hanada <shigeru.hanada@gmail.com>

Prev | Up | Next
F.34. pg_visibility | Home | F.36. seg


F.38. spi
Prev Up Appendix F. Additional Supplied Modules Home Next

F.38. spi

F.38.1. refint — Functions for Implementing Referential Integrity
F.38.2. autoinc — Functions for Autoincrementing Fields
F.38.3. insert_username — Functions for Tracking Who Changed a Table
F.38.4. moddatetime — Functions for Tracking Last Modification Time

The spi module provides several workable examples of using the Server Programming Interface (SPI) and triggers. While these functions are of some value in their own right, they are even more useful as examples to modify for your own purposes. The functions are general enough to be used with any table, but you have to specify table and field names (as described below) while creating a trigger.

Each of the groups of functions described below is provided as a separately-installable extension.

F.38.1. refint — Functions for Implementing Referential Integrity

check_primary_key() and check_foreign_key() are used to check foreign key constraints. (This functionality is long since superseded by the built-in foreign key mechanism, of course, but the module is still useful as an example.)

check_primary_key() checks the referencing table. To use, create a BEFORE INSERT OR UPDATE trigger using this function on a table referencing another table. Specify as the trigger arguments: the referencing table's column name(s) which form the foreign key, the referenced table name, and the column names in the referenced table which form the primary/unique key. To handle multiple foreign keys, create a trigger for each reference.

check_foreign_key() checks the referenced table. To use, create a BEFORE DELETE OR UPDATE trigger using this function on a table referenced by other table(s). Specify as the trigger arguments: the number of referencing tables for which the function has to perform checking, the action if a referencing key is found (cascade — to delete the referencing row, restrict — to abort transaction if referencing keys exist, setnull — to set referencing key fields to null), the triggered table's column names which form the primary/unique key, then the referencing table name and column names (repeated for as many referencing tables as were specified by first argument). Note that the primary/unique key columns should be marked NOT NULL and should have a unique index.

There are examples in refint.example.

F.38.2. autoinc — Functions for Autoincrementing Fields

autoinc() is a trigger that stores the next value of a sequence into an integer field. This has some overlap with the built-in “serial column” feature, but it is not the same: autoinc() will override attempts to substitute a different field value during inserts, and optionally it can be used to increment the field during updates, too.

To use, create a BEFORE INSERT (or optionally BEFORE INSERT OR UPDATE) trigger using this function. Specify two trigger arguments: the name of the integer column to be modified, and the name of the sequence object that will supply values. (Actually, you can specify any number of pairs of such names, if you'd like to update more than one autoincrementing column.)

There is an example in autoinc.example.

F.38.3. insert_username — Functions for Tracking Who Changed a Table

insert_username() is a trigger that stores the current user's name into a text field. This can be useful for tracking who last modified a particular row within a table.

To use, create a BEFORE INSERT and/or UPDATE trigger using this function. Specify a single trigger argument: the name of the text column to be modified.

There is an example in insert_username.example.

F.38.4. moddatetime — Functions for Tracking Last Modification Time

moddatetime() is a trigger that stores the current time into a timestamp field. This can be useful for tracking the last modification time of a particular row within a table.

To use, create a BEFORE UPDATE trigger using this function. Specify a single trigger argument: the name of the column to be modified. The column must be of type timestamp or timestamp with time zone.

There is an example in moddatetime.example.

Prev | Up | Next
F.37. sepgsql | Home | F.39. sslinfo


Appendix G. Additional Supplied Programs
Prev Up Part VIII. Appendixes Home Next

Appendix G. Additional Supplied Programs

Table of Contents

G.1. Client Applications
oid2name — resolve OIDs and file nodes in a PostgreSQL data directory
vacuumlo — remove orphaned large objects from a PostgreSQL database
G.2. Server Applications

This appendix and the previous one contain information regarding the modules that can be found in the contrib directory of the PostgreSQL distribution. See Appendix F for more information about the contrib section in general and server extensions and plug-ins found in contrib specifically.

This appendix covers utility programs found in contrib. Once installed, either from source or a packaging system, they are found in the bin directory of the PostgreSQL installation and can be used like any other program.

Prev | Up | Next
F.47. xml2 | Home | G.1. Client Applications


G.1. Client Applications
Prev Up Appendix G. Additional Supplied Programs Home Next

G.1. Client Applications

oid2name — resolve OIDs and file nodes in a PostgreSQL data directory
vacuumlo — remove orphaned large objects from a PostgreSQL database

This section covers PostgreSQL client applications in contrib. They can be run from anywhere, independent of where the database server resides. See also PostgreSQL Client Applications for information about client applications that are part of the core PostgreSQL distribution.

Prev | Up | Next
Appendix G. Additional Supplied Programs | Home | oid2name


oid2name
Prev Up G.1. Client Applications Home Next

oid2name

oid2name — resolve OIDs and file nodes in a PostgreSQL data directory

Synopsis

oid2name [option...]

Description

oid2name is a utility program that helps administrators to examine the file structure used by PostgreSQL. To make use of it, you need to be familiar with the database file structure, which is described in Chapter 70.

Note

The name “oid2name” is historical, and is actually rather misleading, since most of the time when you use it, you will really be concerned with tables' filenode numbers (which are the file names visible in the database directories). Be sure you understand the difference between table OIDs and table filenodes!

oid2name connects to a target database and extracts OID, filenode, and/or table name information. You can also have it show database OIDs or tablespace OIDs.

Options

oid2name accepts the following command-line arguments:

-f filenode
--filenode=filenode

show info for table with filenode filenode.

-i
--indexes

include indexes and sequences in the listing.

-o oid
--oid=oid

show info for table with OID oid.

-q
--quiet

omit headers (useful for scripting).

-s
--tablespaces

show tablespace OIDs.

-S
--system-objects

include system objects (those in information_schema, pg_toast and pg_catalog schemas).

-t tablename_pattern
--table=tablename_pattern

show info for table(s) matching tablename_pattern.

-V
--version

Print the oid2name version and exit.

-x
--extended

display more information about each object shown: tablespace name, schema name, and OID.

-?
--help

Show help about oid2name command line arguments, and exit.

oid2name also accepts the following command-line arguments for connection parameters:

-d database
--dbname=database

database to connect to.

-h host
--host=host

database server's host.

-H host

database server's host. Use of this parameter is deprecated as of PostgreSQL 12.

-p port
--port=port

database server's port.

-U username
--username=username

user name to connect as.

To display specific tables, select which tables to show by using -o, -f and/or -t. -o takes an OID, -f takes a filenode, and -t takes a table name (actually, it's a LIKE pattern, so you can use things like foo%). You can use as many of these options as you like, and the listing will include all objects matched by any of the options. But note that these options can only show objects in the database given by -d.

If you don't give any of -o, -f or -t, but do give -d, it will list all tables in the database named by -d. In this mode, the -S and -i options control what gets listed.

If you don't give -d either, it will show a listing of database OIDs. Alternatively you can give -s to get a tablespace listing.

Environment

PGHOST
PGPORT
PGUSER

Default connection parameters.

This utility, like most other PostgreSQL utilities, also uses the environment variables supported by libpq (see Section 34.15).

The environment variable PG_COLOR specifies whether to use color in diagnostic messages. Possible values are always, auto and never.

Notes

oid2name requires a running database server with non-corrupt system catalogs. It is therefore of only limited use for recovering from catastrophic database corruption situations.

Examples

$ # what's in this database server, anyway?
$ oid2name
All databases:
    Oid  Database Name  Tablespace
----------------------------------
  17228       alvherre  pg_default
  17255     regression  pg_default
  17227      template0  pg_default
      1      template1  pg_default

$ oid2name -s
All tablespaces:
     Oid  Tablespace Name
-------------------------
    1663       pg_default
    1664        pg_global
  155151         fastdisk
  155152          bigdisk

$ # OK, let's look into database alvherre
$ cd $PGDATA/base/17228

$ # get top 10 db objects in the default tablespace, ordered by size
$ ls -lS * | head -10
-rw-------  1 alvherre alvherre 136536064 sep 14 09:51 155173
-rw-------  1 alvherre alvherre  17965056 sep 14 09:51 1155291
-rw-------  1 alvherre alvherre   1204224 sep 14 09:51 16717
-rw-------  1 alvherre alvherre    581632 sep  6 17:51 1255
-rw-------  1 alvherre alvherre    237568 sep 14 09:50 16674
-rw-------  1 alvherre alvherre    212992 sep 14 09:51 1249
-rw-------  1 alvherre alvherre    204800 sep 14 09:51 16684
-rw-------  1 alvherre alvherre    196608 sep 14 09:50 16700
-rw-------  1 alvherre alvherre    163840 sep 14 09:50 16699
-rw-------  1 alvherre alvherre    122880 sep  6 17:51 16751

$ # I wonder what file 155173 is ...
$ oid2name -d alvherre -f 155173
From database "alvherre":
  Filenode  Table Name
----------------------
    155173    accounts

$ # you can ask for more than one object
$ oid2name -d alvherre -f 155173 -f 1155291
From database "alvherre":
  Filenode     Table Name
-------------------------
    155173       accounts
   1155291  accounts_pkey

$ # you can mix the options, and get more details with -x
$ oid2name -d alvherre -t accounts -f 1155291 -x
From database "alvherre":
  Filenode     Table Name      Oid  Schema  Tablespace
------------------------------------------------------
    155173       accounts   155173  public  pg_default
   1155291  accounts_pkey  1155291  public  pg_default

$ # show disk space for every db object
$ du [0-9]* |
> while read SIZE FILENODE
> do
>   echo "$SIZE       `oid2name -q -d alvherre -i -f $FILENODE`"
> done
16            1155287  branches_pkey
16            1155289  tellers_pkey
17561            1155291  accounts_pkey
...

$ # same, but sort by size
$ du [0-9]* | sort -rn | while read SIZE FN
> do
>   echo "$SIZE   `oid2name -q -d alvherre -f $FN`"
> done
133466             155173    accounts
17561            1155291  accounts_pkey
1177              16717  pg_proc_proname_args_nsp_index
...

$ # If you want to see what's in tablespaces, use the pg_tblspc directory
$ cd $PGDATA/pg_tblspc
$ oid2name -s
All tablespaces:
     Oid  Tablespace Name
-------------------------
    1663       pg_default
    1664        pg_global
  155151         fastdisk
  155152          bigdisk

$ # what databases have objects in tablespace "fastdisk"?
$ ls -d 155151/*
155151/17228/  155151/PG_VERSION

$ # Oh, what was database 17228 again?
$ oid2name
All databases:
    Oid  Database Name  Tablespace
----------------------------------
  17228       alvherre  pg_default
  17255     regression  pg_default
  17227      template0  pg_default
      1      template1  pg_default

$ # Let's see what objects does this database have in the tablespace.
$ cd 155151/17228
$ ls -l
total 0
-rw-------  1 postgres postgres 0 sep 13 23:20 155156

$ # OK, this is a pretty small table ... but which one is it?
$ oid2name -d alvherre -f 155156
From database "alvherre":
  Filenode  Table Name
----------------------
    155156         foo

Author

B. Palmer <bpalmer@crimelabs.net>

Prev | Up | Next
G.1. Client Applications | Home | vacuumlo


G.2. Server Applications
Prev Up Appendix G. Additional Supplied Programs Home Next

G.2. Server Applications

Some applications run on the PostgreSQL server itself. Currently, no such applications are included in the contrib directory. See also PostgreSQL Server Applications for information about server applications that are part of the core PostgreSQL distribution.

Prev | Up | Next
vacuumlo | Home | Appendix H. External Projects


Appendix H. External Projects
Prev Up Part VIII. Appendixes Home Next

Appendix H. External Projects

Table of Contents

H.1. Client Interfaces
H.2. Administration Tools
H.3. Procedural Languages
H.4. Extensions

PostgreSQL is a complex software project, and managing the project is difficult. We have found that many enhancements to PostgreSQL can be more efficiently developed separately from the core project.

Prev | Up | Next
G.2. Server Applications | Home | H.1. Client Interfaces


H.1. Client Interfaces
Prev Up Appendix H. External Projects Home Next

H.1. Client Interfaces

There are only two client interfaces included in the base PostgreSQL distribution:

  • libpq is included because it is the primary C language interface, and because many other client interfaces are built on top of it.

  • ECPG is included because it depends on the server-side SQL grammar, and is therefore sensitive to changes in PostgreSQL itself.

All other language interfaces are external projects and are distributed separately. Table H.1 includes a list of some of these projects. Note that some of these packages might not be released under the same license as PostgreSQL. For more information on each language interface, including licensing terms, refer to its website and documentation.

Table H.1. Externally Maintained Client Interfaces

Name | Language | Comments | Website
DBD::Pg | Perl | Perl DBI driver | https://metacpan.org/release/DBD-Pg
JDBC | Java | Type 4 JDBC driver | https://jdbc.postgresql.org/
libpqxx | C++ | C++ interface | https://pqxx.org/
node-postgres | JavaScript | Node.js driver | https://node-postgres.com/
Npgsql | .NET | .NET data provider | https://www.npgsql.org/
pgtcl | Tcl |  | https://github.com/flightaware/Pgtcl
pgtclng | Tcl |  | https://sourceforge.net/projects/pgtclng/
pq | Go | Pure Go driver for Go's database/sql | https://github.com/lib/pq
psqlODBC | ODBC | ODBC driver | https://odbc.postgresql.org/
psycopg | Python | DB API 2.0-compliant | https://www.psycopg.org/

Prev | Up | Next
Appendix H. External Projects | Home | H.2. Administration Tools


H.2. Administration Tools
Prev Up Appendix H. External Projects Home Next

H.2. Administration Tools

There are several administration tools available for PostgreSQL. The most popular is pgAdmin, and there are several commercially available ones as well.

Prev | Up | Next
H.1. Client Interfaces | Home | H.3. Procedural Languages


H.3. Procedural Languages
Prev Up Appendix H. External Projects Home Next

H.3. Procedural Languages

PostgreSQL includes several procedural languages with the base distribution: PL/pgSQL, PL/Tcl, PL/Perl, and PL/Python.

In addition, there are a number of procedural languages that are developed and maintained outside the core PostgreSQL distribution. Table H.2 lists some of these packages. Note that some of these projects might not be released under the same license as PostgreSQL. For more information on each procedural language, including licensing information, refer to its website and documentation.

Table H.2. Externally Maintained Procedural Languages

Name | Language | Website
PL/Java | Java | https://tada.github.io/pljava/
PL/Lua | Lua | https://github.com/pllua/pllua-ng
PL/R | R | https://github.com/postgres-plr/plr
PL/sh | Unix shell | https://github.com/petere/plsh
PL/v8 | JavaScript | https://github.com/plv8/plv8

Prev | Up | Next
H.2. Administration Tools | Home | H.4. Extensions


H.4. Extensions
Prev Up Appendix H. External Projects Home Next

H.4. Extensions

PostgreSQL is designed to be easily extensible. For this reason, extensions loaded into the database can function just like features that are built in. The contrib/ directory shipped with the source code contains several extensions, which are described in Appendix F. Other extensions are developed independently, like PostGIS. Even PostgreSQL replication solutions can be developed externally. For example, Slony-I is a popular primary/standby replication solution that is developed independently from the core project.

Prev | Up | Next
H.3. Procedural Languages | Home | Appendix I. The Source Code Repository


I.1. Getting the Source via Git
Prev Up Appendix I. The Source Code Repository Home Next

I.1. Getting the Source via Git

With Git you will make a copy of the entire code repository on your local machine, so you will have access to all history and branches offline. This is the fastest and most flexible way to develop or test patches.

Git

  1. You will need an installed version of Git, which you can get from https://git-scm.com. Many systems already have a recent version of Git installed by default, or available in their package distribution system.

  2. To begin using the Git repository, make a clone of the official mirror:

    git clone https://git.postgresql.org/git/postgresql.git

    This will copy the full repository to your local machine, so it may take a while to complete, especially if you have a slow Internet connection. The files will be placed in a new subdirectory postgresql of your current directory.

    The Git mirror can also be reached via the Git protocol. Just change the URL prefix to git, as in:

    git clone git://git.postgresql.org/git/postgresql.git

  3. Whenever you want to get the latest updates in the system, cd into the repository, and run:

    git fetch

Git can do a lot more things than just fetch the source. For more information, consult the Git man pages, or see the website at https://git-scm.com.

Prev | Up | Next
Appendix I. The Source Code Repository | Home | Appendix J. Documentation


Appendix J. Documentation
Prev Up Part VIII. Appendixes Home Next

Appendix J. Documentation

Table of Contents

J.1. DocBook
J.2. Tool Sets
J.2.1. Installation on Fedora, RHEL, and Derivatives
J.2.2. Installation on FreeBSD
J.2.3. Debian Packages
J.2.4. macOS
J.2.5. Detection by configure
J.3. Building the Documentation
J.3.1. HTML
J.3.2. Manpages
J.3.3. PDF
J.3.4. Plain Text Files
J.3.5. Syntax Check
J.4. Documentation Authoring
J.4.1. Emacs
J.5. Style Guide
J.5.1. Reference Pages

PostgreSQL has four primary documentation formats:

  • Plain text, for pre-installation information

  • HTML, for on-line browsing and reference

  • PDF, for printing

  • man pages, for quick reference.

Additionally, a number of plain-text README files can be found throughout the PostgreSQL source tree, documenting various implementation issues.

HTML documentation and man pages are part of a standard distribution and are installed by default. PDF format documentation is available separately for download.

Prev | Up | Next
I.1. Getting the Source via Git | Home | J.1. DocBook


J.1. DocBook
Prev Up Appendix J. Documentation Home Next

J.1. DocBook

The documentation sources are written in DocBook, which is a markup language defined in XML. In what follows, the terms DocBook and XML are both used, but technically they are not interchangeable.

DocBook allows an author to specify the structure and content of a technical document without worrying about presentation details. A document style defines how that content is rendered into one of several final forms. DocBook is maintained by the OASIS group. The official DocBook site has good introductory and reference documentation and a complete O'Reilly book for your online reading pleasure. The NewbieDoc Docbook Guide is very helpful for beginners. The FreeBSD Documentation Project also uses DocBook and has some good information, including a number of style guidelines that might be worth considering.

Prev | Up | Next
Appendix J. Documentation | Home | J.2. Tool Sets


J.2. Tool Sets
Prev Up Appendix J. Documentation Home Next

J.2. Tool Sets

J.2.1. Installation on Fedora, RHEL, and Derivatives
J.2.2. Installation on FreeBSD
J.2.3. Debian Packages
J.2.4. macOS
J.2.5. Detection by configure

The following tools are used to process the documentation. Some might be optional, as noted.

DocBook DTD

This is the definition of DocBook itself. We currently use version 4.5; you cannot use later or earlier versions. You need the XML variant of the DocBook DTD, not the SGML variant.

DocBook XSL Stylesheets

These contain the processing instructions for converting the DocBook sources to other formats, such as HTML.

The minimum required version is currently 1.77.0, but it is recommended to use the latest available version for best results.

Libxml2 for xmllint

This library and the xmllint tool it contains are used for processing XML. Many developers will already have Libxml2 installed, because it is also used when building the PostgreSQL code. Note, however, that xmllint might need to be installed from a separate subpackage.

Libxslt for xsltproc

xsltproc is an XSLT processor, that is, a program to convert XML to other formats using XSLT stylesheets.

FOP

This is a program for converting, among other things, XML to PDF.

We have documented experience with several installation methods for the various tools that are needed to process the documentation. These will be described below. There might be some other packaged distributions for these tools. Please report package status to the documentation mailing list, and we will include that information here.

You can get away with not installing DocBook XML and the DocBook XSLT stylesheets locally, because the required files will be downloaded from the Internet and cached locally. This may in fact be the preferred solution if your operating system packages provide only an old version of these files, or if no packages are available at all. If you want to prevent any attempt to access the Internet while building the documentation, you need to pass the --nonet option to xmllint and xsltproc; see below for an example.

J.2.1. Installation on Fedora, RHEL, and Derivatives

To install the required packages, use:

yum install docbook-dtds docbook-style-xsl fop libxslt

J.2.2. Installation on FreeBSD

To install the required packages with pkg, use:

pkg install docbook-xml docbook-xsl fop libxslt

When building the documentation from the doc directory you'll need to use gmake, because the makefile provided is not suitable for FreeBSD's make.

J.2.3. Debian Packages

There is a full set of packages of the documentation tools available for Debian GNU/Linux. To install, simply use:

apt-get install docbook-xml docbook-xsl fop libxml2-utils xsltproc

J.2.4. macOS

On macOS, you can build the HTML and man documentation without installing anything extra. If you want to build PDFs or want to install a local copy of DocBook, you can get those from your preferred package manager.

If you use MacPorts, the following will get you set up:

sudo port install docbook-xml-4.5 docbook-xsl fop

If you use Homebrew, use this:

brew install docbook docbook-xsl fop

J.2.5. Detection by configure

Before you can build the documentation you need to run the configure script, as you would when building the PostgreSQL programs themselves. Check the output near the end of the run; it should look something like this:

checking for xmllint... xmllint
checking for xsltproc... xsltproc
checking for fop... fop
checking for dbtoepub... dbtoepub

If xmllint or xsltproc is not found, you will not be able to build any of the documentation. fop is only needed to build the documentation in PDF format. dbtoepub is only needed to build the documentation in EPUB format.

If necessary, you can tell configure where to find these programs, for example

./configure ... XMLLINT=/opt/local/bin/xmllint ...

Also, if you want to ensure that xmllint and xsltproc will not perform any network access, you can do something like

./configure ... XMLLINT="xmllint --nonet" XSLTPROC="xsltproc --nonet" ...

Prev | Up | Next
J.1. DocBook | Home | J.3. Building the Documentation


J.3. Building the Documentation
Prev Up Appendix J. Documentation Home Next

J.3. Building the Documentation

J.3.1. HTML
J.3.2. Manpages
J.3.3. PDF
J.3.4. Plain Text Files
J.3.5. Syntax Check

Once you have everything set up, change to the directory doc/src/sgml and run one of the commands described in the following subsections to build the documentation. (Remember to use GNU make.)

J.3.1. HTML

To build the HTML version of the documentation:

doc/src/sgml$ make html

This is also the default target. The output appears in the subdirectory html.

To produce HTML documentation with the stylesheet used on postgresql.org instead of the default simple style use:

doc/src/sgml$ make STYLE=website html

If the STYLE=website option is used, the generated HTML files include references to stylesheets hosted on postgresql.org and require network access to view.

J.3.2. Manpages

We use the DocBook XSL stylesheets to convert DocBook refentry pages to *roff output suitable for man pages. To create the man pages, use the command:

doc/src/sgml$ make man

J.3.3. PDF

To produce a PDF rendition of the documentation using FOP, you can use one of the following commands, depending on the preferred paper format:

  • For A4 format:

    doc/src/sgml$ make postgres-A4.pdf

  • For U.S. letter format:

    doc/src/sgml$ make postgres-US.pdf

Because the PostgreSQL documentation is fairly big, FOP will require a significant amount of memory. Because of that, on some systems, the build will fail with a memory-related error message. This can usually be fixed by configuring Java heap settings in the configuration file ~/.foprc, for example:

# FOP binary distribution
FOP_OPTS='-Xmx1500m'
# Debian
JAVA_ARGS='-Xmx1500m'
# Red Hat
ADDITIONAL_FLAGS='-Xmx1500m'

There is a minimum amount of memory that is required, and to some extent more memory appears to make things a bit faster. On systems with very little memory (less than 1 GB), the build will either be very slow due to swapping or will not work at all.

Other XSL-FO processors can also be used manually, but the automated build process only supports FOP.

J.3.4. Plain Text Files

The installation instructions are also distributed as plain text, in case they are needed in a situation where better reading tools are not available. The INSTALL file corresponds to Chapter 17, with some minor changes to account for the different context. To recreate the file, change to the directory doc/src/sgml and enter make INSTALL. Building text output requires Pandoc version 1.13 or newer as an additional build tool.

In the past, the release notes and regression testing instructions were also distributed as plain text, but this practice has been discontinued.

J.3.5. Syntax Check

Building the documentation can take very long. But there is a method to just check the correct syntax of the documentation files, which only takes a few seconds:

doc/src/sgml$ make check

Prev | Up | Next
J.2. Tool Sets | Home | J.4. Documentation Authoring


J.4. Documentation Authoring
Prev Up Appendix J. Documentation Home Next

J.4. Documentation Authoring

J.4.1. Emacs

The documentation sources are most conveniently modified with an editor that has a mode for editing XML, and even more so if it has some awareness of XML schema languages so that it can know about DocBook syntax specifically.

Note that for historical reasons the documentation source files are named with an extension .sgml even though they are now XML files. So you might need to adjust your editor configuration to set the correct mode.

J.4.1. Emacs

nXML Mode, which ships with Emacs, is the most common mode for editing XML documents with Emacs. It will allow you to use Emacs to insert tags and check markup consistency, and it supports DocBook out of the box. Check the nXML manual for detailed documentation.

src/tools/editors/emacs.samples contains recommended settings for this mode.

Prev | Up | Next
J.3. Building the Documentation | Home | J.5. Style Guide


J.5. Style Guide
Prev Up Appendix J. Documentation Home Next

J.5. Style Guide

J.5.1. Reference Pages

J.5.1. Reference Pages

Reference pages should follow a standard layout. This allows users to find the desired information more quickly, and it also encourages writers to document all relevant aspects of a command. Consistency is not only desired among PostgreSQL reference pages, but also with reference pages provided by the operating system and other packages. Hence the following guidelines have been developed. They are for the most part consistent with similar guidelines established by various operating systems.

Reference pages that describe executable commands should contain the following sections, in this order. Sections that do not apply can be omitted. Additional top-level sections should only be used in special circumstances; often that information belongs in the “Usage” section.

Name

This section is generated automatically. It contains the command name and a half-sentence summary of its functionality.

Synopsis

This section contains the syntax diagram of the command. The synopsis should normally not list each command-line option; that is done below. Instead, list the major components of the command line, such as where input and output files go.

Description

Several paragraphs explaining what the command does.

Options

A list describing each command-line option. If there are a lot of options, subsections can be used.

Exit Status

If the program uses 0 for success and non-zero for failure, then you do not need to document it. If there is a meaning behind the different non-zero exit codes, list them here.

Usage

Describe any sublanguage or run-time interface of the program. If the program is not interactive, this section can usually be omitted. Otherwise, this section is a catch-all for describing run-time features. Use subsections if appropriate.

Environment

List all environment variables that the program might use. Try to be complete; even seemingly trivial variables like SHELL might be of interest to the user.

Files

List any files that the program might access implicitly. That is, do not list input and output files that were specified on the command line, but list configuration files, etc.

Diagnostics

Explain any unusual output that the program might create. Refrain from listing every possible error message. This is a lot of work and has little use in practice. But if, say, the error messages have a standard format that the user can parse, this would be the place to explain it.

Notes

Anything that doesn't fit elsewhere, but in particular bugs, implementation flaws, security considerations, compatibility issues.

Examples

Examples

History

If there were some major milestones in the history of the program, they might be listed here. Usually, this section can be omitted.

Author

Author (only used in the contrib section)

See Also

Cross-references, listed in the following order: other PostgreSQL command reference pages, PostgreSQL SQL command reference pages, citation of PostgreSQL manuals, other reference pages (e.g., operating system, other packages), other documentation. Items in the same group are listed alphabetically.

Reference pages describing SQL commands should contain the following sections: Name, Synopsis, Description, Parameters, Outputs, Notes, Examples, Compatibility, History, See Also. The Parameters section is like the Options section, but there is more freedom about which clauses of the command can be listed. The Outputs section is only needed if the command returns something other than a default command-completion tag. The Compatibility section should explain to what extent this command conforms to the SQL standard(s), or to which other database system it is compatible. The See Also section of SQL commands should list SQL commands before cross-references to programs.

Prev | Up | Next
J.4. Documentation Authoring | Home | Appendix K. PostgreSQL Limits


Appendix K. PostgreSQL Limits
Prev Up Part VIII. Appendixes Home Next

Appendix K. PostgreSQL Limits

Table K.1 describes various hard limits of PostgreSQL. However, practical limits, such as performance limitations or available disk space may apply before absolute hard limits are reached.

Table K.1. PostgreSQL Limitations

Item | Upper Limit | Comment
database size | unlimited |
number of databases | 4,294,950,911 |
relations per database | 1,431,650,303 |
relation size | 32 TB | with the default BLCKSZ of 8192 bytes
rows per table | limited by the number of tuples that can fit onto 4,294,967,295 pages |
columns per table | 1600 | further limited by tuple size fitting on a single page; see note below
columns in a result set | 1664 |
field size | 1 GB |
identifier length | 63 bytes | can be increased by recompiling PostgreSQL
indexes per table | unlimited | constrained by maximum relations per database
columns per index | 32 | can be increased by recompiling PostgreSQL
partition keys | 32 | can be increased by recompiling PostgreSQL

The maximum number of columns for a table is further reduced as the tuple being stored must fit in a single 8192-byte heap page. For example, excluding the tuple header, a tuple made up of 1600 int columns would consume 6400 bytes and could be stored in a heap page, but a tuple of 1600 bigint columns would consume 12800 bytes and would therefore not fit inside a heap page. Variable-length fields of types such as text, varchar, and char can have their values stored out of line in the table's TOAST table when the values are large enough to require it. Only an 18-byte pointer must remain inside the tuple in the table's heap. For shorter length variable-length fields, either a 4-byte or 1-byte field header is used and the value is stored inside the heap tuple.

Columns that have been dropped from the table also contribute to the maximum column limit. Moreover, although the dropped column values for newly created tuples are internally marked as null in the tuple's null bitmap, the null bitmap also occupies space.

Prev | Up | Next
J.5. Style Guide | Home | Appendix L. Acronyms


Appendix L. Acronyms
Prev Up Part VIII. Appendixes Home Next

Appendix L. Acronyms

This is a list of acronyms commonly used in the PostgreSQL documentation and in discussions about PostgreSQL.

ANSI

American National Standards Institute

API

Application Programming Interface

ASCII

American Standard Code for Information Interchange

BKI

Backend Interface

CA

Certificate Authority

CIDR

Classless Inter-Domain Routing

CPAN

Comprehensive Perl Archive Network

CRL

Certificate Revocation List

CSV

Comma Separated Values

CTE

Common Table Expression

CVE

Common Vulnerabilities and Exposures

DBA

Database Administrator

DBI

Database Interface (Perl)

DBMS

Database Management System

DDL

Data Definition Language, SQL commands such as CREATE TABLE, ALTER USER

DML

Data Manipulation Language, SQL commands such as INSERT, UPDATE, DELETE

DST

Daylight Saving Time

ECPG

Embedded C for PostgreSQL

ESQL

Embedded SQL

FAQ

Frequently Asked Questions

FSM

Free Space Map

GEQO

Genetic Query Optimizer

GIN

Generalized Inverted Index

GiST

Generalized Search Tree

Git

Git

GMT

Greenwich Mean Time

GSSAPI

Generic Security Services Application Programming Interface

GUC

Grand Unified Configuration, the PostgreSQL subsystem that handles server configuration

HBA

Host-Based Authentication

HOT

Heap-Only Tuples

IEC

International Electrotechnical Commission

IEEE

Institute of Electrical and Electronics Engineers

IPC

Inter-Process Communication

ISO

International Organization for Standardization

ISSN

International Standard Serial Number

JDBC

Java Database Connectivity

JIT

Just-in-Time compilation

JSON

JavaScript Object Notation

LDAP

Lightweight Directory Access Protocol

LSN

Log Sequence Number, see pg_lsn and WAL Internals.

MITM

Man-in-the-middle attack

MSVC

Microsoft Visual C

MVCC

Multi-Version Concurrency Control

NLS

National Language Support

ODBC

Open Database Connectivity

OID

Object Identifier

OLAP

Online Analytical Processing

OLTP

Online Transaction Processing

ORDBMS

Object-Relational Database Management System

PAM

Pluggable Authentication Modules

PGSQL

PostgreSQL

PGXS

PostgreSQL Extension System

PID

Process Identifier

PITR

Point-In-Time Recovery (Continuous Archiving)

PL

Procedural Languages (server-side)

POSIX

Portable Operating System Interface

RDBMS

Relational Database Management System

RFC

Request For Comments

SGML

Standard Generalized Markup Language

SNI

Server Name Indication, RFC 6066

SPI

Server Programming Interface

SP-GiST

Space-Partitioned Generalized Search Tree

SQL

Structured Query Language

SRF

Set-Returning Function

SSH

Secure Shell

SSL

Secure Sockets Layer

SSPI

Security Support Provider Interface

SYSV

Unix System V

TCP/IP

Transmission Control Protocol (TCP) / Internet Protocol (IP)

TID

Tuple Identifier

TLS

Transport Layer Security

TOAST

The Oversized-Attribute Storage Technique

TPC

Transaction Processing Performance Council

URL

Uniform Resource Locator

UTC

Coordinated Universal Time

UTF

Unicode Transformation Format

UTF8

Eight-Bit Unicode Transformation Format

UUID

Universally Unique Identifier

WAL

Write-Ahead Log

XID

Transaction Identifier

XML

Extensible Markup Language

Prev | Up | Next
Appendix K. PostgreSQL Limits | Home | Appendix M. Glossary


Appendix M. Glossary
Prev Up Part VIII. Appendixes Home Next

Appendix M. Glossary

This is a list of terms and their meaning in the context of PostgreSQL and relational database systems in general.

ACID

Atomicity, Consistency, Isolation, and Durability. This set of properties of database transactions is intended to guarantee validity in concurrent operation and even in event of errors, power failures, etc.

Aggregate function (routine)

A function that combines (aggregates) multiple input values, for example by counting, averaging or adding, yielding a single output value.

For more information, see Section 9.21.

See Also Window function (routine).

Analytic function

See Window function (routine).

Analyze (operation)

The process of collecting statistics from data in tables and other relations to help the query planner to make decisions about how to execute queries.

(Don't confuse this term with the ANALYZE option to the EXPLAIN command.)

For more information, see ANALYZE.

Atomic

In reference to a datum: the fact that its value cannot be broken down into smaller components.

In reference to a database transaction: see atomicity.

Atomicity

The property of a transaction that either all its operations complete as a single unit or none do. In addition, if a system failure occurs during the execution of a transaction, no partial results are visible after recovery. This is one of the ACID properties.

Attribute

An element with a certain name and data type found within a tuple.

Autovacuum (process)

A set of background processes that routinely perform vacuum and analyze operations.

For more information, see Section 25.1.6.

Backend (process)

Process of an instance which acts on behalf of a client session and handles its requests.

(Don't confuse this term with the similar terms Background Worker or Background Writer).

Background worker (process)

Process within an instance, which runs system- or user-supplied code. Serves as infrastructure for several features in PostgreSQL, such as logical replication and parallel queries. In addition, Extensions can add custom background worker processes.

For more information, see Chapter 48.

Background writer (process)

A process that writes dirty data pages from shared memory to the file system. It wakes up periodically, but works only for a short period in order to distribute its expensive I/O activity over time to avoid generating larger I/O peaks which could block other processes.

For more information, see Section 20.4.5.

Base Backup

A binary copy of all database cluster files. It is generated by the tool pg_basebackup. In combination with WAL files it can be used as the starting point for recovery, log shipping, or streaming replication.

Bloat

Space in data pages which does not contain current row versions, such as unused (free) space or outdated row versions.

Cast

A conversion of a datum from its current data type to another data type.

For more information, see CREATE CAST.

Catalog

The SQL standard uses this term to indicate what is called a database in PostgreSQL's terminology.

(Don't confuse this term with system catalog).

For more information, see Section 23.1.

Check constraint

A type of constraint defined on a relation which restricts the values allowed in one or more attributes. The check constraint can make reference to any attribute of the same row in the relation, but cannot reference other rows of the same relation or other relations.

For more information, see Section 5.4.

Checkpoint

A point in the WAL sequence at which it is guaranteed that the heap and index data files have been updated with all information from shared memory modified before that checkpoint; a checkpoint record is written and flushed to WAL to mark that point.

A checkpoint is also the act of carrying out all the actions that are necessary to reach a checkpoint as defined above. This process is initiated when predefined conditions are met, such as a specified amount of time has passed, or a certain volume of records has been written; or it can be invoked by the user with the command CHECKPOINT.

For more information, see Section 30.5.

Checkpointer (process)

A specialized process responsible for executing checkpoints.

Class (archaic)

See Relation.

Client (process)

Any process, possibly remote, that establishes a session by connecting to an instance to interact with a database.

Column

An attribute found in a table or view.

Commit

The act of finalizing a transaction within the database, which makes it visible to other transactions and assures its durability.

For more information, see COMMIT.

Concurrency

The concept that multiple independent operations happen within the database at the same time. In PostgreSQL, concurrency is controlled by the multiversion concurrency control mechanism.

Connection

An established line of communication between a client process and a backend process, usually over a network, supporting a session. This term is sometimes used as a synonym for session.

For more information, see Section 20.3.

Consistency

The property that the data in the database is always in compliance with integrity constraints. Transactions may be allowed to violate some of the constraints transiently before it commits, but if such violations are not resolved by the time it commits, such a transaction is automatically rolled back. This is one of the ACID properties.

Constraint

A restriction on the values of data allowed within a table, or in attributes of a domain.

For more information, see Section 5.4.

Data area

See Data directory.

Database

A named collection of local SQL objects.

For more information, see Section 23.1.

Database cluster

A collection of databases and global SQL objects, and their common static and dynamic metadata. Sometimes referred to as a cluster.

In PostgreSQL, the term cluster is also sometimes used to refer to an instance. (Don't confuse this term with the SQL command CLUSTER.)

Database server

See Instance.

Data directory

The base directory on the file system of a server that contains all data files and subdirectories associated with a database cluster (with the exception of tablespaces, and optionally WAL). The environment variable PGDATA is commonly used to refer to the data directory.

A cluster's storage space comprises the data directory plus any additional tablespaces.

For more information, see Section 70.1.

Data page

The basic structure used to store relation data. All pages are of the same size. Data pages are typically stored on disk, each in a specific file, and can be read to shared buffers where they can be modified, becoming dirty. They become clean when written to disk. New pages, which initially exist in memory only, are also dirty until written.

Datum

The internal representation of one value of an SQL data type.

Delete

An SQL command which removes rows from a given table or relation.

For more information, see DELETE.

Durability

The assurance that once a transaction has been committed, the changes remain even after a system failure or crash. This is one of the ACID properties.

Epoch

See Transaction ID.

Extension

A software add-on package that can be installed on an instance to get extra features.

For more information, see Section 38.17.

File segment

A physical file which stores data for a given relation. File segments are limited in size by a configuration value (typically 1 gigabyte), so if a relation exceeds that size, it is split into multiple segments.

For more information, see Section 70.1.

(Don't confuse this term with the similar term WAL segment).

Foreign data wrapper

A means of representing data that is not contained in the local database so that it appears as if were in local table(s). With a foreign data wrapper it is possible to define a foreign server and foreign tables.

For more information, see CREATE FOREIGN DATA WRAPPER.

Foreign key

A type of constraint defined on one or more columns in a table which requires the value(s) in those columns to identify zero or one row in another (or, infrequently, the same) table.

Foreign server

A named collection of foreign tables which all use the same foreign data wrapper and have other configuration values in common.

For more information, see CREATE SERVER.

Foreign table (relation)

A relation which appears to have rows and columns similar to a regular table, but will forward requests for data through its foreign data wrapper, which will return result sets structured according to the definition of the foreign table.

For more information, see CREATE FOREIGN TABLE.

Fork

Each of the separate segmented file sets in which a relation is stored. The main fork is where the actual data resides. There also exist two secondary forks for metadata: the free space map and the visibility map. Unlogged relations also have an init fork.

Free space map (fork)

A storage structure that keeps metadata about each data page of a table's main fork. The free space map entry for each page stores the amount of free space that's available for future tuples, and is structured to be efficiently searched for available space for a new tuple of a given size.

For more information, see Section 70.3.

Function (routine)

A type of routine that receives zero or more arguments, returns zero or more output values, and is constrained to run within one transaction. Functions are invoked as part of a query, for example via SELECT. Certain functions can return sets; those are called set-returning functions.

Functions can also be used for triggers to invoke.

For more information, see CREATE FUNCTION.

Grant

An SQL command that is used to allow a user or role to access specific objects within the database.

For more information, see GRANT.

Heap

Contains the values of row attributes (i.e., the data) for a relation. The heap is realized within one or more file segments in the relation's main fork.

Host

A computer that communicates with other computers over a network. This is sometimes used as a synonym for server. It is also used to refer to a computer where client processes run.

Index (relation)

A relation that contains data derived from a table or materialized view. Its internal structure supports fast retrieval of and access to the original data.

For more information, see CREATE INDEX.

Insert

An SQL command used to add new data into a table.

For more information, see INSERT.

Instance

A group of backend and auxiliary processes that communicate using a common shared memory area. One postmaster process manages the instance; one instance manages exactly one database cluster with all its databases. Many instances can run on the same server as long as their TCP ports do not conflict.

The instance handles all key features of a DBMS: read and write access to files and shared memory, assurance of the ACID properties, connections to client processes, privilege verification, crash recovery, replication, etc.

Isolation

The property that the effects of a transaction are not visible to concurrent transactions before it commits. This is one of the ACID properties.

For more information, see Section 13.2.

Join

An operation and SQL keyword used in queries for combining data from multiple relations.

Key

A means of identifying a row within a table or other relation by values contained within one or more attributes in that relation.

Lock

A mechanism that allows a process to limit or prevent simultaneous access to a resource.

Log file

Log files contain human-readable text lines about events. Examples include login failures, long-running queries, etc.

For more information, see Section 25.3.

Logged

A table is considered logged if changes to it are sent to the WAL. By default, all regular tables are logged. A table can be specified as unlogged either at creation time or via the ALTER TABLE command.

Logger (process)

If activated, the process writes information about database events into the current log file. When reaching certain time- or volume-dependent criteria, a new log file is created. Also called syslogger.

For more information, see Section 20.8.

Log record

Archaic term for a WAL record.

Master (server)

See Primary (server).

Materialized

The property that some information has been pre-computed and stored for later use, rather than computing it on-the-fly.

This term is used in materialized view, to mean that the data derived from the view's query is stored on disk separately from the sources of that data.

This term is also used to refer to some multi-step queries to mean that the data resulting from executing a given step is stored in memory (with the possibility of spilling to disk), so that it can be read multiple times by another step.

Materialized view (relation)

A relation that is defined by a SELECT statement (just like a view), but stores data in the same way that a table does. It cannot be modified via INSERT, UPDATE, or DELETE operations.

For more information, see CREATE MATERIALIZED VIEW.

Multi-version concurrency control (MVCC)

A mechanism designed to allow several transactions to be reading and writing the same rows without one process causing other processes to stall. In PostgreSQL, MVCC is implemented by creating copies (versions) of tuples as they are modified; after transactions that can see the old versions terminate, those old versions need to be removed.

Null

A concept of non-existence that is a central tenet of relational database theory. It represents the absence of a definite value.

Optimizer

See Query planner.

Parallel query

The ability to handle parts of executing a query to take advantage of parallel processes on servers with multiple CPUs.

Partition

One of several disjoint (not overlapping) subsets of a larger set.

In reference to a partitioned table: One of the tables that each contain part of the data of the partitioned table, which is said to be the parent. The partition is itself a table, so it can also be queried directly; at the same time, a partition can sometimes be a partitioned table, allowing hierarchies to be created.

In reference to a window function in a query, a partition is a user-defined criterion that identifies which neighboring rows of the query's result set can be considered by the function.

Partitioned table (relation)

A relation that is in semantic terms the same as a table, but whose storage is distributed across several partitions.

Postmaster (process)

The very first process of an instance. It starts and manages the other auxiliary processes and creates backend processes on demand.

For more information, see Section 19.3.

Primary key

A special case of a unique constraint defined on a table or other relation that also guarantees that all of the attributes within the primary key do not have null values. As the name implies, there can be only one primary key per table, though it is possible to have multiple unique constraints that also have no null-capable attributes.

Primary (server)

When two or more databases are linked via replication, the server that is considered the authoritative source of information is called the primary, also known as a master.

Procedure (routine)

A type of routine. Their distinctive qualities are that they do not return values, and that they are allowed to make transactional statements such as COMMIT and ROLLBACK. They are invoked via the CALL command.

For more information, see CREATE PROCEDURE.

Query

A request sent by a client to a backend, usually to return results or to modify data on the database.

Query planner

The part of PostgreSQL that is devoted to determining (planning) the most efficient way to execute queries. Also known as query optimizer, optimizer, or simply planner.

Record

See Tuple.

Recycling

See WAL file.

Referential integrity

A means of restricting data in one relation by a foreign key so that it must have matching data in another relation.

Relation

The generic term for all objects in a database that have a name and a list of attributes defined in a specific order. Tables, sequences, views, foreign tables, materialized views, composite types, and indexes are all relations.

More generically, a relation is a set of tuples; for example, the result of a query is also a relation.

In PostgreSQL, Class is an archaic synonym for relation.

Replica (server)

A database that is paired with a primary database and is maintaining a copy of some or all of the primary database's data. The foremost reasons for doing this are to allow for greater access to that data, and to maintain availability of the data in the event that the primary becomes unavailable.

Replication

The act of reproducing data on one server onto another server called a replica. This can take the form of physical replication, where all file changes from one server are copied verbatim, or logical replication where a defined subset of data changes are conveyed using a higher-level representation.

Result set

A relation transmitted from a backend process to a client upon the completion of an SQL command, usually a SELECT but it can be an INSERT, UPDATE, or DELETE command if the RETURNING clause is specified.

The fact that a result set is a relation means that a query can be used in the definition of another query, becoming a subquery.

Revoke

A command to prevent access to a named set of database objects for a named list of roles.

For more information, see REVOKE.

Role

A collection of access privileges to the instance. Roles are themselves a privilege that can be granted to other roles. This is often done for convenience or to ensure completeness when multiple users need the same privileges.

For more information, see CREATE ROLE.

Rollback

A command to undo all of the operations performed since the beginning of a transaction.

For more information, see ROLLBACK.

Routine

A defined set of instructions stored in the database system that can be invoked for execution. A routine can be written in a variety of programming languages. Routines can be functions (including set-returning functions and trigger functions), aggregate functions, and procedures.

Many routines are already defined within PostgreSQL itself, but user-defined ones can also be added.

Row

See Tuple.

Savepoint

A special mark in the sequence of steps in a transaction. Data modifications after this point in time may be reverted to the time of the savepoint.

For more information, see SAVEPOINT.

Schema

A schema is a namespace for SQL objects, which all reside in the same database. Each SQL object must reside in exactly one schema.

All system-defined SQL objects reside in schema pg_catalog.

More generically, the term schema is used to mean all data descriptions ( table definitions, constraints, comments, etc) for a given database or subset thereof.

For more information, see Section 5.9.

Segment

See File segment.

Select

The SQL command used to request data from a database. Normally, SELECT commands are not expected to modify the database in any way, but it is possible that functions invoked within the query could have side effects that do modify data.

For more information, see SELECT.

Sequence (relation)

A type of relation that is used to generate values. Typically the generated values are sequential non-repeating numbers. They are commonly used to generate surrogate primary key values.

Server

A computer on which PostgreSQL instances run. The term server denotes real hardware, a container, or a virtual machine.

This term is sometimes used to refer to an instance or to a host.

Session

A state that allows a client and a backend to interact, communicating over a connection.

Shared memory

RAM which is used by the processes common to an instance. It mirrors parts of database files, provides a transient area for WAL records, and stores additional common information. Note that shared memory belongs to the complete instance, not to a single database.

The largest part of shared memory is known as shared buffers and is used to mirror part of data files, organized into pages. When a page is modified, it is called a dirty page until it is written back to the file system.

For more information, see Section 20.4.1.

SQL object

Any object that can be created with a CREATE command. Most objects are specific to one database, and are commonly known as local objects.

Most local objects reside in a specific schema in their containing database, such as relations (all types), routines (all types), data types, etc. The names of such objects of the same type in the same schema are enforced to be unique.

There also exist local objects that do not reside in schemas; some examples are extensions, data type casts, and foreign data wrappers. The names of such objects of the same type are enforced to be unique within the database.

Other object types, such as roles, tablespaces, replication origins, subscriptions for logical replication, and databases themselves are not local SQL objects since they exist entirely outside of any specific database; they are called global objects. The names of such objects are enforced to be unique within the whole database cluster.

For more information, see Section 23.1.

SQL standard

A series of documents that define the SQL language.

Standby (server)

See Replica (server).

Stats collector (process)

This process collects statistical information about the instance's activities.

For more information, see Section 28.2.

System catalog

A collection of tables which describe the structure of all SQL objects of the instance. The system catalog resides in the schema pg_catalog. These tables contain data in internal representation and are not typically considered useful for user examination; a number of user-friendlier views, also in schema pg_catalog, offer more convenient access to some of that information, while additional tables and views exist in schema information_schema (see Chapter 37) that expose some of the same and additional information as mandated by the SQL standard.

For more information, see Section 5.9.

Table

A collection of tuples having a common data structure (the same number of attributes, in the same order, having the same name and type per position). A table is the most common form of relation in PostgreSQL.

For more information, see CREATE TABLE.

Tablespace

A named location on the server file system. All SQL objects which require storage beyond their definition in the system catalog must belong to a single tablespace. Initially, a database cluster contains a single usable tablespace which is used as the default for all SQL objects, called pg_default.

For more information, see Section 23.6.

Temporary table

Tables that exist either for the lifetime of a session or a transaction, as specified at the time of creation. The data in them is not visible to other sessions, and is not logged. Temporary tables are often used to store intermediate data for a multi-step operation.

For more information, see CREATE TABLE.

TOAST

A mechanism by which large attributes of table rows are split and stored in a secondary table, called the TOAST table. Each relation with large attributes has its own TOAST table.

For more information, see Section 70.2.

Transaction

A combination of commands that must act as a single atomic command: they all succeed or all fail as a single unit, and their effects are not visible to other sessions until the transaction is complete, and possibly even later, depending on the isolation level.

For more information, see Section 13.2.

Transaction ID

The numerical, unique, sequentially-assigned identifier that each transaction receives when it first causes a database modification. Frequently abbreviated as xid. When stored on disk, xids are only 32-bits wide, so only approximately four billion write transaction IDs can be generated; to permit the system to run for longer than that, epochs are used, also 32 bits wide. When the counter reaches the maximum xid value, it starts over at 3 (values under that are reserved) and the epoch value is incremented by one. In some contexts, the epoch and xid values are considered together as a single 64-bit value.

For more information, see Section 8.19.

Transactions per second (TPS)

Average number of transactions that are executed per second, totaled across all sessions active for a measured run. This is used as a measure of the performance characteristics of an instance.

Trigger

A function which can be defined to execute whenever a certain operation (INSERT, UPDATE, DELETE, TRUNCATE) is applied to a relation. A trigger executes within the same transaction as the statement which invoked it, and if the function fails, then the invoking statement also fails.

For more information, see CREATE TRIGGER.

Tuple

A collection of attributes in a fixed order. That order may be defined by the table (or other relation) where the tuple is contained, in which case the tuple is often called a row. It may also be defined by the structure of a result set, in which case it is sometimes called a record.

Unique constraint

A type of constraint defined on a relation which restricts the values allowed in one or a combination of columns so that each value or combination of values can only appear once in the relation — that is, no other row in the relation contains values that are equal to those.

Because null values are not considered equal to each other, multiple rows with null values are allowed to exist without violating the unique constraint.

Unlogged

The property of certain relations that the changes to them are not reflected in the WAL. This disables replication and crash recovery for these relations.

The primary use of unlogged tables is for storing transient work data that must be shared across processes.

Temporary tables are always unlogged.

Update

An SQL command used to modify rows that may already exist in a specified table. It cannot create or remove rows.

For more information, see UPDATE.

User

A role that has the LOGIN privilege.

User mapping

The translation of login credentials in the local database to credentials in a remote data system defined by a foreign data wrapper.

For more information, see CREATE USER MAPPING.

Vacuum

The process of removing outdated tuple versions from tables or materialized views, and other closely related processing required by PostgreSQL's implementation of MVCC. This can be initiated through the use of the VACUUM command, but can also be handled automatically via autovacuum processes.

For more information, see Section 25.1 .

View

A relation that is defined by a SELECT statement, but has no storage of its own. Any time a query references a view, the definition of the view is substituted into the query as if the user had typed it as a subquery instead of the name of the view.

For more information, see CREATE VIEW.

Visibility map (fork)

A storage structure that keeps metadata about each data page of a table's main fork. The visibility map entry for each page stores two bits: the first one (all-visible) indicates that all tuples in the page are visible to all transactions. The second one (all-frozen) indicates that all tuples in the page are marked frozen.

WAL

See Write-ahead log.

WAL archiver (process)

A process that saves copies of WAL files for the purpose of creating backups or keeping replicas current.

For more information, see Section 26.3.

WAL file

Also known as WAL segment or WAL segment file. Each of the sequentially-numbered files that provide storage space for WAL. The files are all of the same predefined size and are written in sequential order, interspersing changes as they occur in multiple simultaneous sessions. If the system crashes, the files are read in order, and each of the changes is replayed to restore the system to the state it was in before the crash.

Each WAL file can be released after a checkpoint writes all the changes in it to the corresponding data files. Releasing the file can be done either by deleting it, or by changing its name so that it will be used in the future, which is called recycling.

For more information, see Section 30.6.

WAL record

A low-level description of an individual data change. It contains sufficient information for the data change to be re-executed (replayed) in case a system failure causes the change to be lost. WAL records use a non-printable binary format.

For more information, see Section 30.6.

WAL segment

See WAL file.

WAL writer (process)

A process that writes WAL records from shared memory to WAL files.

For more information, see Section 20.5.

Window function (routine)

A type of function used in a query that applies to a partition of the query's result set; the function's result is based on values found in rows of the same partition or frame.

All aggregate functions can be used as window functions, but window functions can also be used to, for example, give ranks to each of the rows in the partition. Also known as analytic functions.

For more information, see Section 3.5.

Write-ahead log

The journal that keeps track of the changes in the database cluster as user- and system-invoked operations take place. It comprises many individual WAL records written sequentially to WAL files.

Prev | Up | Next
Appendix L. Acronyms | Home | Appendix N. Color Support


Appendix N. Color Support
Prev Up Part VIII. Appendixes Home Next

Appendix N. Color Support

Table of Contents

N.1. When Color is Used
N.2. Configuring the Colors

Most programs in the PostgreSQL package can produce colorized console output. This appendix describes how that is configured.

Prev | Up | Next
Appendix M. Glossary | Home | N.1. When Color is Used


N.1. When Color is Used
Prev Up Appendix N. Color Support Home Next

N.1. When Color is Used

To use colorized output, set the environment variable PG_COLOR as follows:

  1. If the value is always, then color is used.

  2. If the value is auto and the standard error stream is associated with a terminal device, then color is used.

  3. Otherwise, color is not used.

Prev | Up | Next
Appendix N. Color Support | Home | N.2. Configuring the Colors


N.2. Configuring the Colors
Prev Up Appendix N. Color Support Home Next

N.2. Configuring the Colors

The actual colors to be used are configured using the environment variable PG_COLORS (note plural). The value is a colon-separated list of key=value pairs. The keys specify what the color is to be used for. The values are SGR (Select Graphic Rendition) specifications, which are interpreted by the terminal.

The following keys are currently in use:

error

used to highlight the text “error” in error messages

warning

used to highlight the text “warning” in warning messages

locus

used to highlight location information (e.g., program name and file name) in messages

The default value is error=01;31:warning=01;35:locus=01 (01;31 = bold red, 01;35 = bold magenta, 01 = bold default color).

Tip

This color specification format is also used by other software packages such as GCC, GNU coreutils, and GNU grep.

Prev | Up | Next
N.1. When Color is Used | Home | Appendix O. Obsolete or Renamed Features


Appendix O. Obsolete or Renamed Features
Prev Up Part VIII. Appendixes Home Next

Appendix O. Obsolete or Renamed Features

Table of Contents

O.1. recovery.conf file merged into postgresql.conf
O.2. Default Roles Renamed to Predefined Roles
O.3. pg_xlogdump renamed to pg_waldump
O.4. pg_resetxlog renamed to pg_resetwal
O.5. pg_receivexlog renamed to pg_receivewal

Functionality is sometimes removed from PostgreSQL, feature, setting and file names sometimes change, or documentation moves to different places. This section directs users coming from old versions of the documentation or from external links to the appropriate new location for the information they need.

Prev | Up | Next
N.2. Configuring the Colors | Home | O.1. recovery.conf file merged into postgresql.conf


O.1. recovery.conf file merged into postgresql.conf
Prev Up Appendix O. Obsolete or Renamed Features Home Next

O.1. recovery.conf file merged into postgresql.conf

PostgreSQL 11 and below used a configuration file named recovery.conf to manage replicas and standbys. Support for this file was removed in PostgreSQL 12. See the release notes for PostgreSQL 12 for details on this change.

On PostgreSQL 12 and above, archive recovery, streaming replication, and PITR are configured using normal server configuration parameters. These are set in postgresql.conf or via ALTER SYSTEM like any other parameter.

The server will not start if a recovery.conf exists.

The trigger_file setting has been renamed to promote_trigger_file.

The standby_mode setting has been removed. A standby.signal file in the data directory is used instead. See Standby Server Operation for details.

Prev | Up | Next
Appendix O. Obsolete or Renamed Features | Home | O.2. Default Roles Renamed to Predefined Roles


O.2. Default Roles Renamed to Predefined Roles
Prev Up Appendix O. Obsolete or Renamed Features Home Next

O.2. Default Roles Renamed to Predefined Roles

PostgreSQL 13 and below used the term “Default Roles”. However, as these roles are not able to actually be changed and are installed as part of the system at initialization time, the more appropriate term to use is “Predefined Roles”. See Section 22.5 for current documentation regarding Predefined Roles, and the release notes for PostgreSQL 14 for details on this change.

Prev | Up | Next
O.1. recovery.conf file merged into postgresql.conf | Home | O.3. pg_xlogdump renamed to pg_waldump


O.3. pg_xlogdump renamed to pg_waldump
Prev Up Appendix O. Obsolete or Renamed Features Home Next

O.3. pg_xlogdump renamed to pg_waldump

PostgreSQL 9.6 and below provided a command named pg_xlogdump to read write-ahead-log (WAL) files. This command was renamed to pg_waldump, see pg_waldump for documentation of pg_waldump and see the release notes for PostgreSQL 10 for details on this change.

Prev | Up | Next
O.2. Default Roles Renamed to Predefined Roles | Home | O.4. pg_resetxlog renamed to pg_resetwal


O.4. pg_resetxlog renamed to pg_resetwal
Prev Up Appendix O. Obsolete or Renamed Features Home Next

O.4. pg_resetxlog renamed to pg_resetwal

PostgreSQL 9.6 and below provided a command named pg_resetxlog to reset the write-ahead-log (WAL) files. This command was renamed to pg_resetwal, see pg_resetwal for documentation of pg_resetwal and see the release notes for PostgreSQL 10 for details on this change.

Prev | Up | Next
O.3. pg_xlogdump renamed to pg_waldump | Home | O.5. pg_receivexlog renamed to pg_receivewal


O.5. pg_receivexlog renamed to pg_receivewal
Prev Up Appendix O. Obsolete or Renamed Features Home Next

O.5. pg_receivexlog renamed to pg_receivewal

PostgreSQL 9.6 and below provided a command named pg_receivexlog to fetch write-ahead-log (WAL) files. This command was renamed to pg_receivewal, see pg_receivewal for documentation of pg_receivewal and see the release notes for PostgreSQL 10 for details on this change.

Prev | Up | Next
O.4. pg_resetxlog renamed to pg_resetwal | Home | Bibliography


Bibliography
Prev Up PostgreSQL 14.4 Documentation Home Next

Bibliography

Selected references and readings for SQL and PostgreSQL.

Some white papers and technical reports from the original POSTGRES development team are available at the University of California, Berkeley, Computer Science Department web site.

SQL Reference Books

[bowman01] The Practical SQL Handbook. Using SQL Variants. Fourth Edition. Judith Bowman, Sandra Emerson, and Marcy Darnovsky. ISBN 0-201-70309-2. Addison-Wesley Professional. 2001.

[date97] A Guide to the SQL Standard. A user's guide to the standard database language SQL. Fourth Edition. C. J. Date and Hugh Darwen. ISBN 0-201-96426-0. Addison-Wesley. 1997.

[date04] An Introduction to Database Systems. Eighth Edition. C. J. Date. ISBN 0-321-19784-4. Addison-Wesley. 2003.

[elma04] Fundamentals of Database Systems. Fourth Edition. Ramez Elmasri and Shamkant Navathe. ISBN 0-321-12226-7. Addison-Wesley. 2003.

[melt93] Understanding the New SQL. A complete guide. Jim Melton and Alan R. Simon. ISBN 1-55860-245-3. Morgan Kaufmann. 1993.

[ull88] Principles of Database and Knowledge-Base Systems. Classical Database Systems. Jeffrey D. Ullman. Volume 1. Computer Science Press. 1988.

[sqltr-19075-6] SQL Technical Report. Part 6: SQL support for JavaScript Object Notation (JSON). First Edition. 2017.

PostgreSQL-specific Documentation

[sim98] Enhancement of the ANSI SQL Implementation of PostgreSQL. Stefan Simkovics. Department of Information Systems, Vienna University of Technology. Vienna, Austria. November 29, 1998.

[yu95] The Postgres95. User Manual. A. Yu and J. Chen. University of California. Berkeley, California. Sept. 5, 1995.

[fong] The design and implementation of the POSTGRES query optimizer. Zelaine Fong. University of California, Berkeley, Computer Science Department.

Proceedings and Articles

[ports12] “Serializable Snapshot Isolation in PostgreSQL”. D. Ports and K. Grittner. VLDB Conference, August 2012.

[berenson95] “A Critique of ANSI SQL Isolation Levels”. H. Berenson, P. Bernstein, J. Gray, J. Melton, E. O'Neil, and P. O'Neil. ACM-SIGMOD Conference on Management of Data, June 1995.

[olson93] Partial indexing in POSTGRES: research project. Nels Olson. UCB Engin T7.49.1993 O676. University of California. Berkeley, California. 1993.

[ong90] “A Unified Framework for Version Modeling Using Production Rules in a Database System”. L. Ong and J. Goh. ERL Technical Memorandum M90/33. University of California. Berkeley, California. April, 1990.

[rowe87] “The POSTGRES data model”. L. Rowe and M. Stonebraker. VLDB Conference, Sept. 1987.

[seshadri95] “Generalized Partial Indexes”. P. Seshadri and A. Swami. Eleventh International Conference on Data Engineering, 6–10 March 1995. Cat. No.95CH35724. IEEE Computer Society Press. Los Alamitos, California. 1995. 420–7.

[ston86] “The design of POSTGRES”. M. Stonebraker and L. Rowe. ACM-SIGMOD Conference on Management of Data, May 1986.

[ston87a] “The design of the POSTGRES rules system”. M. Stonebraker, E. Hanson, and C. H. Hong. IEEE Conference on Data Engineering, Feb. 1987.

[ston87b] “The design of the POSTGRES storage system”. M. Stonebraker. VLDB Conference, Sept. 1987.

[ston89] “A commentary on the POSTGRES rules system”. M. Stonebraker, M. Hearst, and S. Potamianos. SIGMOD Record 18(3). Sept. 1989.

[ston89b] “The case for partial indexes”. M. Stonebraker. SIGMOD Record 18(4). Dec. 1989. 4–11.

[ston90a] “The implementation of POSTGRES”. M. Stonebraker, L. A. Rowe, and M. Hirohama. Transactions on Knowledge and Data Engineering 2(1). IEEE. March 1990.

[ston90b] “On Rules, Procedures, Caching and Views in Database Systems”. M. Stonebraker, A. Jhingran, J. Goh, and S. Potamianos. ACM-SIGMOD Conference on Management of Data, June 1990.

Prev | Up | Next
O.5. pg_receivexlog renamed to pg_receivewal | Home | Index


Index
Prev Up PostgreSQL 14.4 Documentation Home

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

$, Positional Parameters
$libdir, Dynamic Loading
$libdir/plugins, Shared Library Preloading, Description
*, Select-List Items
.pgpass, The Password File
.pg_service.conf, The Connection Service File
::, Type Casts
_PG_fini, Dynamic Loading
_PG_init, Dynamic Loading
_PG_output_plugin_init, Initialization Function

A

abbrev, Network Address Functions and Operators
ABORT, ABORT
abs, Mathematical Functions and Operators
ACL, Privileges
aclcontains, System Information Functions and Operators
acldefault, System Information Functions and Operators
aclexplode, System Information Functions and Operators
aclitem, Privileges
aclitemeq, System Information Functions and Operators
acos, Mathematical Functions and Operators
acosd, Mathematical Functions and Operators
acosh, Mathematical Functions and Operators
administration tools
externally maintained, Administration Tools
adminpack, adminpack
advisory lock, Advisory Locks
age, Date/Time Functions and Operators
aggregate function, Aggregate Functions, Aggregate Expressions, Aggregate Functions, User-Defined Aggregates
built-in, Aggregate Functions
invocation, Aggregate Expressions
moving aggregate, Moving-Aggregate Mode
ordered set, Ordered-Set Aggregates
partial aggregation, Partial Aggregation
polymorphic, Polymorphic and Variadic Aggregates
support functions for, Support Functions for Aggregates
user-defined, User-Defined Aggregates
variadic, Polymorphic and Variadic Aggregates
AIX, AIX
installation on, AIX
IPC configuration, Shared Memory and Semaphores
akeys, hstore Operators and Functions
alias, Table and Column Aliases, Column Labels
for table name in query, Joins Between Tables
in the FROM clause, Table and Column Aliases
in the select list, Column Labels
ALL, GROUPING SETS, CUBE, and ROLLUP, DISTINCT, Subquery Expressions, Row and Array Comparisons
GROUP BY ALL, GROUPING SETS, CUBE, and ROLLUP
SELECT ALL, DISTINCT
allow_system_table_mods configuration parameter, Developer Options
ALTER AGGREGATE, ALTER AGGREGATE
ALTER COLLATION, ALTER COLLATION
ALTER CONVERSION, ALTER CONVERSION
ALTER DATABASE, ALTER DATABASE
ALTER DEFAULT PRIVILEGES, ALTER DEFAULT PRIVILEGES
ALTER DOMAIN, ALTER DOMAIN
ALTER EVENT TRIGGER, ALTER EVENT TRIGGER
ALTER EXTENSION, ALTER EXTENSION
ALTER FOREIGN DATA WRAPPER, ALTER FOREIGN DATA WRAPPER
ALTER FOREIGN TABLE, ALTER FOREIGN TABLE
ALTER FUNCTION, ALTER FUNCTION
ALTER GROUP, ALTER GROUP
ALTER INDEX, ALTER INDEX
ALTER LANGUAGE, ALTER LANGUAGE
ALTER LARGE OBJECT, ALTER LARGE OBJECT
ALTER MATERIALIZED VIEW, ALTER MATERIALIZED VIEW
ALTER OPERATOR, ALTER OPERATOR
ALTER OPERATOR CLASS, ALTER OPERATOR CLASS
ALTER OPERATOR FAMILY, ALTER OPERATOR FAMILY
ALTER POLICY, ALTER POLICY
ALTER PROCEDURE, ALTER PROCEDURE
ALTER PUBLICATION, ALTER PUBLICATION
ALTER ROLE, Role Attributes, ALTER ROLE
ALTER ROUTINE, ALTER ROUTINE
ALTER RULE, ALTER RULE
ALTER SCHEMA, ALTER SCHEMA
ALTER SEQUENCE, ALTER SEQUENCE
ALTER SERVER, ALTER SERVER
ALTER STATISTICS, ALTER STATISTICS
ALTER SUBSCRIPTION, ALTER SUBSCRIPTION
ALTER SYSTEM, ALTER SYSTEM
ALTER TABLE, ALTER TABLE
ALTER TABLESPACE, ALTER TABLESPACE
ALTER TEXT SEARCH CONFIGURATION, ALTER TEXT SEARCH CONFIGURATION
ALTER TEXT SEARCH DICTIONARY, ALTER TEXT SEARCH DICTIONARY
ALTER TEXT SEARCH PARSER, ALTER TEXT SEARCH PARSER
ALTER TEXT SEARCH TEMPLATE, ALTER TEXT SEARCH TEMPLATE
ALTER TRIGGER, ALTER TRIGGER
ALTER TYPE, ALTER TYPE
ALTER USER, ALTER USER
ALTER USER MAPPING, ALTER USER MAPPING
ALTER VIEW, ALTER VIEW
amcheck, amcheck
ANALYZE, Updating Planner Statistics, ANALYZE
AND (operator), Logical Operators
anonymous code blocks, DO
any, Pseudo-Types
ANY, Aggregate Functions, Subquery Expressions, Row and Array Comparisons
anyarray, Pseudo-Types
anycompatible, Pseudo-Types
anycompatiblearray, Pseudo-Types
anycompatiblemultirange, Pseudo-Types
anycompatiblenonarray, Pseudo-Types
anycompatiblerange, Pseudo-Types
anyelement, Pseudo-Types
anyenum, Pseudo-Types
anymultirange, Pseudo-Types
anynonarray, Pseudo-Types
anyrange, Pseudo-Types
applicable role, applicable_roles
application_name configuration parameter, What to Log
arbitrary precision numbers, Arbitrary Precision Numbers
archive_cleanup_command configuration parameter, Archive Recovery
archive_command configuration parameter, Archiving
archive_mode configuration parameter, Archiving
archive_timeout configuration parameter, Archiving
area, Geometric Functions and Operators
armor, armor(), dearmor()
array, Arrays
accessing, Accessing Arrays
constant, Array Value Input
constructor, Array Constructors
declaration, Declaration of Array Types
I/O, Array Input and Output Syntax
modifying, Modifying Arrays
of user-defined type, User-Defined Types
searching, Searching in Arrays
ARRAY, Array Constructors, UNION, CASE, and Related Constructs
determination of result type, UNION, CASE, and Related Constructs
array_agg, Aggregate Functions, Functions
array_append, Array Functions and Operators
array_cat, Array Functions and Operators
array_dims, Array Functions and Operators
array_fill, Array Functions and Operators
array_length, Array Functions and Operators
array_lower, Array Functions and Operators
array_ndims, Array Functions and Operators
array_nulls configuration parameter, Previous PostgreSQL Versions
array_position, Array Functions and Operators
array_positions, Array Functions and Operators
array_prepend, Array Functions and Operators
array_remove, Array Functions and Operators
array_replace, Array Functions and Operators
array_to_json, Processing and Creating JSON Data
array_to_string, Array Functions and Operators
array_to_tsvector, Text Search Functions and Operators
array_upper, Array Functions and Operators
ascii, String Functions and Operators
asin, Mathematical Functions and Operators
asind, Mathematical Functions and Operators
asinh, Mathematical Functions and Operators
ASSERT
in PL/pgSQL, Checking Assertions
assertions
in PL/pgSQL, Checking Assertions
asynchronous commit, Asynchronous Commit
AT TIME ZONE, AT TIME ZONE
atan, Mathematical Functions and Operators
atan2, Mathematical Functions and Operators
atan2d, Mathematical Functions and Operators
atand, Mathematical Functions and Operators
atanh, Mathematical Functions and Operators
authentication_timeout configuration parameter, Authentication
auth_delay, auth_delay
auth_delay.milliseconds configuration parameter, Configuration Parameters
auto-increment (see serial)
autocommit
bulk-loading data, Disable Autocommit
psql, Variables
autosummarize storage parameter, Index Storage Parameters
autovacuum
configuration parameters, Automatic Vacuuming
general information, The Autovacuum Daemon
autovacuum configuration parameter, Automatic Vacuuming
autovacuum_analyze_scale_factor
configuration parameter, Automatic Vacuuming
storage parameter, Storage Parameters
autovacuum_analyze_threshold
configuration parameter, Automatic Vacuuming
storage parameter, Storage Parameters
autovacuum_enabled storage parameter, Storage Parameters
autovacuum_freeze_max_age
configuration parameter, Automatic Vacuuming
storage parameter, Storage Parameters
autovacuum_freeze_min_age storage parameter, Storage Parameters
autovacuum_freeze_table_age storage parameter, Storage Parameters
autovacuum_max_workers configuration parameter, Automatic Vacuuming
autovacuum_multixact_freeze_max_age
configuration parameter, Automatic Vacuuming
storage parameter, Storage Parameters
autovacuum_multixact_freeze_min_age storage parameter, Storage Parameters
autovacuum_multixact_freeze_table_age storage parameter, Storage Parameters
autovacuum_naptime configuration parameter, Automatic Vacuuming
autovacuum_vacuum_cost_delay
configuration parameter, Automatic Vacuuming
storage parameter, Storage Parameters
autovacuum_vacuum_cost_limit
configuration parameter, Automatic Vacuuming
storage parameter, Storage Parameters
autovacuum_vacuum_insert_scale_factor
configuration parameter, Automatic Vacuuming
storage parameter, Storage Parameters
autovacuum_vacuum_insert_threshold
configuration parameter, Automatic Vacuuming
storage parameter, Storage Parameters
autovacuum_vacuum_scale_factor
configuration parameter, Automatic Vacuuming
storage parameter, Storage Parameters
autovacuum_vacuum_threshold
configuration parameter, Automatic Vacuuming
storage parameter, Storage Parameters
autovacuum_work_mem configuration parameter, Memory
auto_explain, auto_explain
auto_explain.log_analyze configuration parameter, Configuration Parameters
auto_explain.log_buffers configuration parameter, Configuration Parameters
auto_explain.log_format configuration parameter, Configuration Parameters
auto_explain.log_level configuration parameter, Configuration Parameters
auto_explain.log_min_duration configuration parameter, Configuration Parameters
auto_explain.log_nested_statements configuration parameter, Configuration Parameters
auto_explain.log_settings configuration parameter, Configuration Parameters
auto_explain.log_timing configuration parameter, Configuration Parameters
auto_explain.log_triggers configuration parameter, Configuration Parameters
auto_explain.log_verbose configuration parameter, Configuration Parameters
auto_explain.log_wal configuration parameter, Configuration Parameters
auto_explain.sample_rate configuration parameter, Configuration Parameters
avals, hstore Operators and Functions
average, Aggregate Functions
avg, Aggregate Functions

B

B-Tree (see index)
backend_flush_after configuration parameter, Asynchronous Behavior
Background workers, Background Worker Processes
backslash escapes, String Constants with C-Style Escapes
backslash_quote configuration parameter, Previous PostgreSQL Versions
backtrace_functions configuration parameter, Developer Options
backup, Backup Control Functions, Backup and Restore
Backup Manifest, Backup Manifest Format
base type, The PostgreSQL Type System
base64 format, Binary String Functions and Operators
BASE_BACKUP, Streaming Replication Protocol
batch mode, Pipeline Mode
in libpq, Pipeline Mode
BEGIN, BEGIN
BETWEEN, Comparison Functions and Operators
BETWEEN SYMMETRIC, Comparison Functions and Operators
BGWORKER_BACKEND_​DATABASE_CONNECTION, Background Worker Processes
BGWORKER_SHMEM_ACCESS, Background Worker Processes
bgwriter_delay configuration parameter, Background Writer
bgwriter_flush_after configuration parameter, Background Writer
bgwriter_lru_maxpages configuration parameter, Background Writer
bgwriter_lru_multiplier configuration parameter, Background Writer
bigint, Numeric Constants, Integer Types
bigserial, Serial Types
binary data, Binary Data Types, Binary String Functions and Operators
functions, Binary String Functions and Operators
binary string
concatenation, Binary String Functions and Operators
converting to character string, Binary String Functions and Operators
length, Binary String Functions and Operators
bison, Requirements
bit string, Bit-String Constants, Bit String Types
constant, Bit-String Constants
data type, Bit String Types
length, Bit String Functions and Operators
bit strings, Bit String Functions and Operators
functions, Bit String Functions and Operators
bitmap scan, Combining Multiple Indexes, Planner Method Configuration
bit_and, Aggregate Functions
bit_count, Binary String Functions and Operators, Bit String Functions and Operators
bit_length, String Functions and Operators, Binary String Functions and Operators, Bit String Functions and Operators
bit_or, Aggregate Functions
bit_xor, Aggregate Functions
BLOB (see large object)
block_size configuration parameter, Preset Options
bloom, bloom
bonjour configuration parameter, Connection Settings
bonjour_name configuration parameter, Connection Settings
Boolean, Boolean Type
data type, Boolean Type
operators (see operators, logical)
bool_and, Aggregate Functions
bool_or, Aggregate Functions
booting
starting the server during, Starting the Database Server
bound_box, Geometric Functions and Operators
box, Geometric Functions and Operators
box (data type), Boxes
BRIN (see index)
brin_desummarize_range, Index Maintenance Functions
brin_metapage_info, BRIN Functions
brin_page_items, BRIN Functions
brin_page_type, BRIN Functions
brin_revmap_data, BRIN Functions
brin_summarize_new_values, Index Maintenance Functions
brin_summarize_range, Index Maintenance Functions
broadcast, Network Address Functions and Operators
BSD Authentication, BSD Authentication
btree_gin, btree_gin
btree_gist, btree_gist
btrim, String Functions and Operators, Binary String Functions and Operators
bt_index_check, Functions
bt_index_parent_check, Functions
bt_metap, B-Tree Functions
bt_page_items, B-Tree Functions
bt_page_stats, B-Tree Functions
buffering storage parameter, Index Storage Parameters
bytea, Binary Data Types
bytea_output configuration parameter, Statement Behavior

C

C, libpq — C Library, ECPG — Embedded SQL in C
C++, Using C++ for Extensibility
CALL, CALL
canceling, Canceling Queries in Progress
SQL command, Canceling Queries in Progress
cardinality, Array Functions and Operators
CASCADE, Dependency Tracking
with DROP, Dependency Tracking
foreign key action, Foreign Keys
Cascading Replication, High Availability, Load Balancing, and Replication
CASE, Conditional Expressions, UNION, CASE, and Related Constructs
determination of result type, UNION, CASE, and Related Constructs
case sensitivity
of SQL commands, Identifiers and Key Words
cast, CREATE CAST
I/O conversion, CREATE CAST
cbrt, Mathematical Functions and Operators
ceil, Mathematical Functions and Operators
ceiling, Mathematical Functions and Operators
center, Geometric Functions and Operators
Certificate, Certificate Authentication
chained transactions, Transaction Management, Parameters, Parameters
in PL/pgSQL, Transaction Management
char, Character Types
character, Character Types
character set, Locale and Formatting, Preset Options, Character Set Support
character string, String Constants, Character Types
concatenation, String Functions and Operators
constant, String Constants
converting to binary string, Binary String Functions and Operators
data types, Character Types
length, String Functions and Operators
character varying, Character Types
character_length, String Functions and Operators
char_length, String Functions and Operators
check constraint, Check Constraints
CHECK OPTION, CREATE VIEW
checkpoint, WAL Configuration
CHECKPOINT, CHECKPOINT
checkpoint_completion_target configuration parameter, Checkpoints
checkpoint_flush_after configuration parameter, Checkpoints
checkpoint_timeout configuration parameter, Checkpoints
checkpoint_warning configuration parameter, Checkpoints
checksums, Data Checksums
check_function_bodies configuration parameter, Statement Behavior
chr, String Functions and Operators
cid, Object Identifier Types
cidr, cidr
circle, Circles, Geometric Functions and Operators
citext, citext
client authentication, Client Authentication
timeout during, Authentication
client_connection_check_interval configuration parameter, Connection Settings
client_encoding configuration parameter, Locale and Formatting
client_min_messages configuration parameter, Statement Behavior
clock_timestamp, Date/Time Functions and Operators
CLOSE, CLOSE
cluster
of databases (see database cluster)
CLUSTER, CLUSTER
clusterdb, clusterdb
clustering, High Availability, Load Balancing, and Replication
cluster_name configuration parameter, Process Title
cmax, System Columns
cmin, System Columns
COALESCE, COALESCE
COLLATE, Collation Expressions
collation, Collation Support
in PL/pgSQL, Collation of PL/pgSQL Variables
in SQL functions, SQL Functions with Collations
COLLATION FOR, System Information Functions and Operators
color, Color Support
column, Concepts, Table Basics
adding, Adding a Column
removing, Removing a Column
renaming, Renaming a Column
system column, System Columns
column data type
changing, Changing a Column's Data Type
column reference, Column References
col_description, System Information Functions and Operators
comment, Comments
about database objects, System Information Functions and Operators
in SQL, Comments
COMMENT, COMMENT
COMMIT, COMMIT
COMMIT PREPARED, COMMIT PREPARED
commit_delay configuration parameter, Settings
commit_siblings configuration parameter, Settings
common table expression (see WITH)
comparison, Comparison Functions and Operators, Subquery Expressions
composite type, Row and Array Comparisons
operators, Comparison Functions and Operators
row constructor, Row and Array Comparisons
subquery result row, Subquery Expressions
compiling, Building libpq Programs
libpq applications, Building libpq Programs
composite type, Composite Types, The PostgreSQL Type System
comparison, Row and Array Comparisons
constant, Constructing Composite Values
constructor, Row Constructors
computed field, Using Composite Types in Queries
compute_query_id configuration parameter, Statistics Monitoring
concat, String Functions and Operators
concat_ws, String Functions and Operators
concurrency, Concurrency Control
conditional expression, Conditional Expressions
configuration
of recovery
of a standby server, Archive Recovery
of the server, Server Configuration
of the server
functions, Configuration Settings Functions
configure, Installation Procedure
configure environment variables, configure Environment Variables
configure options, configure Options
config_file configuration parameter, File Locations
conjunction, Logical Operators
connectby, Functions Provided, connectby
connection service file, The Connection Service File
conninfo, Connection Strings
constant, Constants
constraint, Constraints
adding, Adding a Constraint
check, Check Constraints
exclusion, Exclusion Constraints
foreign key, Foreign Keys
name, Check Constraints
NOT NULL, Not-Null Constraints
primary key, Primary Keys
removing, Removing a Constraint
unique, Unique Constraints
constraint exclusion, Partitioning and Constraint Exclusion, Other Planner Options
constraint_exclusion configuration parameter, Other Planner Options
container type, The PostgreSQL Type System
CONTINUE
in PL/pgSQL, CONTINUE
continuous archiving, Backup and Restore
in standby, Continuous Archiving in Standby
control file, Extension Files
convert, Binary String Functions and Operators
convert_from, Binary String Functions and Operators
convert_to, Binary String Functions and Operators
COPY, Populating a Table With Rows, Functions Associated with the COPY Command, COPY
with libpq, Functions Associated with the COPY Command
corr, Aggregate Functions
correlation, Aggregate Functions
in the query planner, Extended Statistics
cos, Mathematical Functions and Operators
cosd, Mathematical Functions and Operators
cosh, Mathematical Functions and Operators
cot, Mathematical Functions and Operators
cotd, Mathematical Functions and Operators
count, Aggregate Functions
covariance
population, Aggregate Functions
sample, Aggregate Functions
covar_pop, Aggregate Functions
covar_samp, Aggregate Functions
covering index, Index-Only Scans and Covering Indexes
cpu_index_tuple_cost configuration parameter, Planner Cost Constants
cpu_operator_cost configuration parameter, Planner Cost Constants
cpu_tuple_cost configuration parameter, Planner Cost Constants
CREATE ACCESS METHOD, CREATE ACCESS METHOD
CREATE AGGREGATE, CREATE AGGREGATE
CREATE CAST, CREATE CAST
CREATE COLLATION, CREATE COLLATION
CREATE CONVERSION, CREATE CONVERSION
CREATE DATABASE, Creating a Database, CREATE DATABASE
CREATE DOMAIN, CREATE DOMAIN
CREATE EVENT TRIGGER, CREATE EVENT TRIGGER
CREATE EXTENSION, CREATE EXTENSION
CREATE FOREIGN DATA WRAPPER, CREATE FOREIGN DATA WRAPPER
CREATE FOREIGN TABLE, CREATE FOREIGN TABLE
CREATE FUNCTION, CREATE FUNCTION
CREATE GROUP, CREATE GROUP
CREATE INDEX, CREATE INDEX
CREATE LANGUAGE, CREATE LANGUAGE
CREATE MATERIALIZED VIEW, CREATE MATERIALIZED VIEW
CREATE OPERATOR, CREATE OPERATOR
CREATE OPERATOR CLASS, CREATE OPERATOR CLASS
CREATE OPERATOR FAMILY, CREATE OPERATOR FAMILY
CREATE POLICY, CREATE POLICY
CREATE PROCEDURE, CREATE PROCEDURE
CREATE PUBLICATION, CREATE PUBLICATION
CREATE ROLE, Database Roles, CREATE ROLE
CREATE RULE, CREATE RULE
CREATE SCHEMA, CREATE SCHEMA
CREATE SEQUENCE, CREATE SEQUENCE
CREATE SERVER, CREATE SERVER
CREATE STATISTICS, CREATE STATISTICS
CREATE SUBSCRIPTION, CREATE SUBSCRIPTION
CREATE TABLE, Creating a New Table, CREATE TABLE
CREATE TABLE AS, CREATE TABLE AS
CREATE TABLESPACE, Tablespaces, CREATE TABLESPACE
CREATE TEXT SEARCH CONFIGURATION, CREATE TEXT SEARCH CONFIGURATION
CREATE TEXT SEARCH DICTIONARY, CREATE TEXT SEARCH DICTIONARY
CREATE TEXT SEARCH PARSER, CREATE TEXT SEARCH PARSER
CREATE TEXT SEARCH TEMPLATE, CREATE TEXT SEARCH TEMPLATE
CREATE TRANSFORM, CREATE TRANSFORM
CREATE TRIGGER, CREATE TRIGGER
CREATE TYPE, CREATE TYPE
CREATE USER, CREATE USER
CREATE USER MAPPING, CREATE USER MAPPING
CREATE VIEW, CREATE VIEW
createdb, Creating a Database, Creating a Database, createdb
createuser, Database Roles, createuser
CREATE_REPLICATION_SLOT, Streaming Replication Protocol
cross compilation, Build Process Details
cross join, Joined Tables
crosstab, crosstab(text), crosstabN(text), crosstab(text, text)
crypt, crypt()
cstring, Pseudo-Types
CSV (Comma-Separated Values) format
in psql, Meta-Commands
ctid, System Columns
CTID, View Rules in Non-SELECT Statements
CUBE, GROUPING SETS, CUBE, and ROLLUP
cube (extension), cube
cume_dist, Window Functions
hypothetical, Aggregate Functions
current_catalog, System Information Functions and Operators
current_database, System Information Functions and Operators
current_date, Date/Time Functions and Operators
current_logfiles
and the log_destination configuration parameter, Where to Log
and the pg_current_logfile function, System Information Functions and Operators
current_query, System Information Functions and Operators
current_role, System Information Functions and Operators
current_schema, System Information Functions and Operators
current_schemas, System Information Functions and Operators
current_setting, Configuration Settings Functions
current_time, Date/Time Functions and Operators
current_timestamp, Date/Time Functions and Operators
current_user, System Information Functions and Operators
currval, Sequence Manipulation Functions
cursor, Cursors, CLOSE, DECLARE, EXPLAIN, FETCH, MOVE
CLOSE, CLOSE
DECLARE, DECLARE
FETCH, FETCH
in PL/pgSQL, Cursors
MOVE, MOVE
showing the query plan, EXPLAIN
cursor_tuple_fraction configuration parameter, Other Planner Options
custom scan provider, Writing a Custom Scan Provider
handler for, Writing a Custom Scan Provider
Cygwin, Cygwin
installation on, Cygwin

D

data area (see database cluster)
data partitioning, High Availability, Load Balancing, and Replication
data type, Data Types, Numeric Types, Enumerated Types, Domain Types, Type Conversion, The PostgreSQL Type System, The PostgreSQL Type System, The PostgreSQL Type System, Polymorphic Types, Base Types in C-Language Functions, User-Defined Types
base, The PostgreSQL Type System
category, Overview
composite, The PostgreSQL Type System
constant, Constants of Other Types
container, The PostgreSQL Type System
conversion, Type Conversion
domain, Domain Types
enumerated (enum), Enumerated Types
internal organization, Base Types in C-Language Functions
numeric, Numeric Types
polymorphic, Polymorphic Types
type cast, Type Casts
user-defined, User-Defined Types
database, Creating a Database, Managing Databases
creating, Creating a Database
privilege to create, Role Attributes
database activity, Monitoring Database Activity
monitoring, Monitoring Database Activity
database cluster, Concepts, Creating a Database Cluster
data_checksums configuration parameter, Preset Options
data_directory configuration parameter, File Locations
data_directory_mode configuration parameter, Preset Options
data_sync_retry configuration parameter, Error Handling
date, Date/Time Types, Dates
constants, Special Values
current, Current Date/Time
output format, Date/Time Output
(see also formatting)
DateStyle configuration parameter, Locale and Formatting
date_bin, date_bin
date_part, Date/Time Functions and Operators, EXTRACT, date_part
date_trunc, Date/Time Functions and Operators, date_trunc
dblink, dblink, dblink
dblink_build_sql_delete, dblink_build_sql_delete
dblink_build_sql_insert, dblink_build_sql_insert
dblink_build_sql_update, dblink_build_sql_update
dblink_cancel_query, dblink_cancel_query
dblink_close, dblink_close
dblink_connect, dblink_connect
dblink_connect_u, dblink_connect_u
dblink_disconnect, dblink_disconnect
dblink_error_message, dblink_error_message
dblink_exec, dblink_exec
dblink_fetch, dblink_fetch
dblink_get_connections, dblink_get_connections
dblink_get_notify, dblink_get_notify
dblink_get_pkey, dblink_get_pkey
dblink_get_result, dblink_get_result
dblink_is_busy, dblink_is_busy
dblink_open, dblink_open
dblink_send_query, dblink_send_query
db_user_namespace configuration parameter, Authentication
deadlock, Deadlocks
timeout during, Lock Management
deadlock_timeout configuration parameter, Lock Management
DEALLOCATE, DEALLOCATE
dearmor, armor(), dearmor()
debug_assertions configuration parameter, Preset Options
debug_deadlocks configuration parameter, Developer Options
debug_discard_caches configuration parameter, Developer Options
debug_pretty_print configuration parameter, What to Log
debug_print_parse configuration parameter, What to Log
debug_print_plan configuration parameter, What to Log
debug_print_rewritten configuration parameter, What to Log
decimal (see numeric)
DECLARE, DECLARE
decode, Binary String Functions and Operators
decode_bytea
in PL/Perl, Utility Functions in PL/Perl
decrypt, Raw Encryption Functions
decrypt_iv, Raw Encryption Functions
deduplicate_items storage parameter, Index Storage Parameters
default value, Default Values
changing, Changing a Column's Default Value
default-roles, Default Roles Renamed to Predefined Roles
default_statistics_target configuration parameter, Other Planner Options
default_tablespace configuration parameter, Statement Behavior
default_table_access_method configuration parameter, Statement Behavior
default_text_search_config configuration parameter, Locale and Formatting
default_toast_compression configuration parameter, Statement Behavior
default_transaction_deferrable configuration parameter, Statement Behavior
default_transaction_isolation configuration parameter, Statement Behavior
default_transaction_read_only configuration parameter, Statement Behavior
deferrable transaction, Statement Behavior
setting, SET TRANSACTION
setting default, Statement Behavior
defined, hstore Operators and Functions
degrees, Mathematical Functions and Operators
delay, Delaying Execution
DELETE, Deletions, Deleting Data, Returning Data from Modified Rows, DELETE
RETURNING, Returning Data from Modified Rows
delete, hstore Operators and Functions
deleting, Deleting Data
dense_rank, Window Functions
hypothetical, Aggregate Functions
diagonal, Geometric Functions and Operators
diameter, Geometric Functions and Operators
dict_int, dict_int
dict_xsyn, dict_xsyn
difference, Soundex
digest, digest()
dirty read, Transaction Isolation
DISCARD, DISCARD
disjunction, Logical Operators
disk drive, WAL Internals
disk space, Recovering Disk Space
disk usage, Determining Disk Usage
DISTINCT, Querying a Table, GROUPING SETS, CUBE, and ROLLUP, DISTINCT
GROUP BY DISTINCT, GROUPING SETS, CUBE, and ROLLUP
SELECT DISTINCT, DISTINCT
div, Mathematical Functions and Operators
dmetaphone, Double Metaphone
dmetaphone_alt, Double Metaphone
DO, DO
document, What Is a Document?
text search, What Is a Document?
dollar quoting, Dollar-Quoted String Constants
domain, Domain Types
double precision, Floating-Point Types
DROP ACCESS METHOD, DROP ACCESS METHOD
DROP AGGREGATE, DROP AGGREGATE
DROP CAST, DROP CAST
DROP COLLATION, DROP COLLATION
DROP CONVERSION, DROP CONVERSION
DROP DATABASE, Destroying a Database, DROP DATABASE
DROP DOMAIN, DROP DOMAIN
DROP EVENT TRIGGER, DROP EVENT TRIGGER
DROP EXTENSION, DROP EXTENSION
DROP FOREIGN DATA WRAPPER, DROP FOREIGN DATA WRAPPER
DROP FOREIGN TABLE, DROP FOREIGN TABLE
DROP FUNCTION, DROP FUNCTION
DROP GROUP, DROP GROUP
DROP INDEX, DROP INDEX
DROP LANGUAGE, DROP LANGUAGE
DROP MATERIALIZED VIEW, DROP MATERIALIZED VIEW
DROP OPERATOR, DROP OPERATOR
DROP OPERATOR CLASS, DROP OPERATOR CLASS
DROP OPERATOR FAMILY, DROP OPERATOR FAMILY
DROP OWNED, DROP OWNED
DROP POLICY, DROP POLICY
DROP PROCEDURE, DROP PROCEDURE
DROP PUBLICATION, DROP PUBLICATION
DROP ROLE, Database Roles, DROP ROLE
DROP ROUTINE, DROP ROUTINE
DROP RULE, DROP RULE
DROP SCHEMA, DROP SCHEMA
DROP SEQUENCE, DROP SEQUENCE
DROP SERVER, DROP SERVER
DROP STATISTICS, DROP STATISTICS
DROP SUBSCRIPTION, DROP SUBSCRIPTION
DROP TABLE, Creating a New Table, DROP TABLE
DROP TABLESPACE, DROP TABLESPACE
DROP TEXT SEARCH CONFIGURATION, DROP TEXT SEARCH CONFIGURATION
DROP TEXT SEARCH DICTIONARY, DROP TEXT SEARCH DICTIONARY
DROP TEXT SEARCH PARSER, DROP TEXT SEARCH PARSER
DROP TEXT SEARCH TEMPLATE, DROP TEXT SEARCH TEMPLATE
DROP TRANSFORM, DROP TRANSFORM
DROP TRIGGER, DROP TRIGGER
DROP TYPE, DROP TYPE
DROP USER, DROP USER
DROP USER MAPPING, DROP USER MAPPING
DROP VIEW, DROP VIEW
dropdb, Destroying a Database, dropdb
dropuser, Database Roles, dropuser
DROP_REPLICATION_SLOT, Streaming Replication Protocol
DTD, Creating XML Values
DTrace, Developer Options, Dynamic Tracing
duplicate, Querying a Table
duplicates, DISTINCT
dynamic loading, Other Defaults, Dynamic Loading
dynamic_library_path, Dynamic Loading
dynamic_library_path configuration parameter, Other Defaults
dynamic_shared_memory_type configuration parameter, Memory

E

each, hstore Operators and Functions
earth, Cube-Based Earth Distances
earthdistance, earthdistance
earth_box, Cube-Based Earth Distances
earth_distance, Cube-Based Earth Distances
ECPG, ECPG — Embedded SQL in C
ecpg, ecpg
effective_cache_size configuration parameter, Planner Cost Constants
effective_io_concurrency configuration parameter, Asynchronous Behavior
elog, Reporting Errors Within the Server
in PL/Perl, Utility Functions in PL/Perl
in PL/Python, Utility Functions
in PL/Tcl, Database Access from PL/Tcl
embedded SQL, ECPG — Embedded SQL in C
in C, ECPG — Embedded SQL in C
enabled role, enabled_roles
enable_async_append configuration parameter, Planner Method Configuration
enable_bitmapscan configuration parameter, Planner Method Configuration
enable_gathermerge configuration parameter, Planner Method Configuration
enable_hashagg configuration parameter, Planner Method Configuration
enable_hashjoin configuration parameter, Planner Method Configuration
enable_incremental_sort configuration parameter, Planner Method Configuration
enable_indexonlyscan configuration parameter, Planner Method Configuration
enable_indexscan configuration parameter, Planner Method Configuration
enable_material configuration parameter, Planner Method Configuration
enable_memoize configuration parameter, Planner Method Configuration
enable_mergejoin configuration parameter, Planner Method Configuration
enable_nestloop configuration parameter, Planner Method Configuration
enable_parallel_append configuration parameter, Planner Method Configuration
enable_parallel_hash configuration parameter, Planner Method Configuration
enable_partitionwise_aggregate configuration parameter, Planner Method Configuration
enable_partitionwise_join configuration parameter, Planner Method Configuration
enable_partition_pruning configuration parameter, Planner Method Configuration
enable_seqscan configuration parameter, Planner Method Configuration
enable_sort configuration parameter, Planner Method Configuration
enable_tidscan configuration parameter, Planner Method Configuration
encode, Binary String Functions and Operators
encode_array_constructor
in PL/Perl, Utility Functions in PL/Perl
encode_array_literal
in PL/Perl, Utility Functions in PL/Perl
encode_bytea
in PL/Perl, Utility Functions in PL/Perl
encode_typed_literal
in PL/Perl, Utility Functions in PL/Perl
encrypt, Raw Encryption Functions
encryption, Encryption Options, pgcrypto
for specific columns, pgcrypto
encrypt_iv, Raw Encryption Functions
END, END
enumerated types, Enumerated Types
enum_first, Enum Support Functions
enum_last, Enum Support Functions
enum_range, Enum Support Functions
environment variable, Environment Variables
ephemeral named relation
registering with SPI, SPI_register_relation, SPI_register_trigger_data
unregistering from SPI, SPI_unregister_relation
ereport, Reporting Errors Within the Server
error codes, PostgreSQL Error Codes
libpq, Main Functions
list of, PostgreSQL Error Codes
error message, Connection Status Functions
escape format, Binary String Functions and Operators
escape string syntax, String Constants with C-Style Escapes
escape_string_warning configuration parameter, Previous PostgreSQL Versions
escaping strings, Escaping Strings for Inclusion in SQL Commands
in libpq, Escaping Strings for Inclusion in SQL Commands
event log, Registering Event Log on Windows
event log, Registering Event Log on Windows
event trigger, Event Triggers, Writing Event Trigger Functions in C
in C, Writing Event Trigger Functions in C
in PL/Tcl, Event Trigger Functions in PL/Tcl
event_source configuration parameter, Where to Log
event_trigger, Pseudo-Types
every, Aggregate Functions
EXCEPT, Combining Queries (UNION, INTERSECT, EXCEPT)
exceptions
in PL/pgSQL, Trapping Errors
in PL/Tcl, Error Handling in PL/Tcl
exclusion constraint, Exclusion Constraints
EXECUTE, EXECUTE
exist, hstore Operators and Functions
EXISTS, Subquery Expressions
EXIT
in PL/pgSQL, EXIT
exit_on_error configuration parameter, Error Handling
exp, Mathematical Functions and Operators
EXPLAIN, Using EXPLAIN, EXPLAIN
expression, Value Expressions
order of evaluation, Expression Evaluation Rules
syntax, Value Expressions
extending SQL, Extending SQL
extension, Packaging Related Objects into an Extension
externally maintained, Extensions
external_pid_file configuration parameter, File Locations
extract, Date/Time Functions and Operators, EXTRACT, date_part
extra_float_digits configuration parameter, Locale and Formatting

F

factorial, Mathematical Functions and Operators
failover, High Availability, Load Balancing, and Replication
false, Boolean Type
family, Network Address Functions and Operators
fast path, The Fast-Path Interface
fastupdate storage parameter, Index Storage Parameters
fdw_handler, Pseudo-Types
FETCH, FETCH
field
computed, Using Composite Types in Queries
field selection, Field Selection
file system mount points, Use of Secondary File Systems
file_fdw, file_fdw
fillfactor storage parameter, Index Storage Parameters, Storage Parameters
FILTER, Aggregate Expressions
first_value, Window Functions
flex, Requirements
float4 (see real)
float8 (see double precision)
floating point, Floating-Point Types
floating-point
display, Locale and Formatting
floor, Mathematical Functions and Operators
force_parallel_mode configuration parameter, Developer Options
foreign data, Foreign Data
foreign data wrapper, Writing a Foreign Data Wrapper
handler for, Writing a Foreign Data Wrapper
foreign key, Foreign Keys, Foreign Keys
self-referential, Foreign Keys
foreign table, Foreign Data
format, String Functions and Operators, format
use in PL/pgSQL, Executing Dynamic Commands
formatting, Data Type Formatting Functions
format_type, System Information Functions and Operators
Free Space Map, Free Space Map
FreeBSD
IPC configuration, Shared Memory and Semaphores
shared library, Compiling and Linking Dynamically-Loaded Functions
start script, Starting the Database Server
from_collapse_limit configuration parameter, Other Planner Options
FSM (see Free Space Map)
fsm_page_contents, General Functions
fsync configuration parameter, Settings
full text search, Text Search Types, Text Search Types, Full Text Search
data types, Text Search Types
functions and operators, Text Search Types
full_page_writes configuration parameter, Settings
function, Table Functions, Functions and Operators, Statistics Information Functions, Functions, Polymorphic Types, User-Defined Functions, Query Language (SQL) Functions, Internal Functions, C-Language Functions
default values for arguments, SQL Functions with Default Values for Arguments
in the FROM clause, Table Functions
internal, Internal Functions
invocation, Function Calls
mixed notation, Using Mixed Notation
named argument, Arguments for SQL Functions
named notation, Using Named Notation
output parameter, SQL Functions with Output Parameters
polymorphic, Polymorphic Types
positional notation, Using Positional Notation
RETURNS TABLE, SQL Functions Returning TABLE
statistics, Statistics Information Functions
type resolution in an invocation, Functions
user-defined, User-Defined Functions, Query Language (SQL) Functions, C-Language Functions
in C, C-Language Functions
in SQL, Query Language (SQL) Functions
variadic, SQL Functions with Variable Numbers of Arguments
with SETOF, SQL Functions Returning Sets
functional dependency, The GROUP BY and HAVING Clauses
fuzzystrmatch, fuzzystrmatch

G

gcd, Mathematical Functions and Operators
gc_to_sec, Cube-Based Earth Distances
generated column, Generated Columns, Parameters, Parameters
in triggers, Overview of Trigger Behavior
generate_series, Set Returning Functions
generate_subscripts, Set Returning Functions
genetic query optimization, Genetic Query Optimizer
gen_random_bytes, Random-Data Functions
gen_random_uuid, UUID Functions, Random-Data Functions
gen_salt, gen_salt()
GEQO (see genetic query optimization)
geqo configuration parameter, Genetic Query Optimizer
geqo_effort configuration parameter, Genetic Query Optimizer
geqo_generations configuration parameter, Genetic Query Optimizer
geqo_pool_size configuration parameter, Genetic Query Optimizer
geqo_seed configuration parameter, Genetic Query Optimizer
geqo_selection_bias configuration parameter, Genetic Query Optimizer
geqo_threshold configuration parameter, Genetic Query Optimizer
get_bit, Binary String Functions and Operators, Bit String Functions and Operators
get_byte, Binary String Functions and Operators
get_current_ts_config, Text Search Functions and Operators
get_raw_page, General Functions
GIN (see index)
gin_clean_pending_list, Index Maintenance Functions
gin_fuzzy_search_limit configuration parameter, Other Defaults
gin_leafpage_items, GIN Functions
gin_metapage_info, GIN Functions
gin_page_opaque_info, GIN Functions
gin_pending_list_limit
configuration parameter, Statement Behavior
storage parameter, Index Storage Parameters
GiST (see index)
gist_page_items, GiST Functions
gist_page_items_bytea, GiST Functions
gist_page_opaque_info, GiST Functions
global data, Global Data in PL/Tcl
in PL/Python, Sharing Data
in PL/Tcl, Global Data in PL/Tcl
GRANT, Privileges, GRANT
GREATEST, GREATEST and LEAST, UNION, CASE, and Related Constructs
determination of result type, UNION, CASE, and Related Constructs
Gregorian calendar, History of Units
GROUP BY, Aggregate Functions, The GROUP BY and HAVING Clauses
grouping, The GROUP BY and HAVING Clauses
GROUPING, Aggregate Functions
GROUPING SETS, GROUPING SETS, CUBE, and ROLLUP
gssapi, Secure TCP/IP Connections with GSSAPI Encryption
GSSAPI, GSSAPI Authentication
with libpq, Parameter Key Words
GUID, UUID Type

H

hash (see index)
hash_bitmap_info, Hash Functions
hash_mem_multiplier configuration parameter, Memory
hash_metapage_info, Hash Functions
hash_page_items, Hash Functions
hash_page_stats, Hash Functions
hash_page_type, Hash Functions
has_any_column_privilege, System Information Functions and Operators
has_column_privilege, System Information Functions and Operators
has_database_privilege, System Information Functions and Operators
has_foreign_data_wrapper_privilege, System Information Functions and Operators
has_function_privilege, System Information Functions and Operators
has_language_privilege, System Information Functions and Operators
has_schema_privilege, System Information Functions and Operators
has_sequence_privilege, System Information Functions and Operators
has_server_privilege, System Information Functions and Operators
has_tablespace_privilege, System Information Functions and Operators
has_table_privilege, System Information Functions and Operators
has_type_privilege, System Information Functions and Operators
HAVING, Aggregate Functions, The GROUP BY and HAVING Clauses
hba_file configuration parameter, File Locations
heap_page_items, Heap Functions
heap_page_item_attrs, Heap Functions
heap_tuple_infomask_flags, Heap Functions
height, Geometric Functions and Operators
hex format, Binary String Functions and Operators
hierarchical database, Concepts
high availability, High Availability, Load Balancing, and Replication
history, A Brief History of PostgreSQL
of PostgreSQL, A Brief History of PostgreSQL
hmac, hmac()
host, Network Address Functions and Operators
host name, Parameter Key Words
hostmask, Network Address Functions and Operators
Hot Standby, High Availability, Load Balancing, and Replication
hot_standby configuration parameter, Standby Servers
hot_standby_feedback configuration parameter, Standby Servers
HP-UX
IPC configuration, Shared Memory and Semaphores
shared library, Compiling and Linking Dynamically-Loaded Functions
hstore, hstore, hstore Operators and Functions
hstore_to_array, hstore Operators and Functions
hstore_to_json, hstore Operators and Functions
hstore_to_jsonb, hstore Operators and Functions
hstore_to_jsonb_loose, hstore Operators and Functions
hstore_to_json_loose, hstore Operators and Functions
hstore_to_matrix, hstore Operators and Functions
huge_pages configuration parameter, Memory
huge_page_size configuration parameter, Memory
hypothetical-set aggregate
built-in, Aggregate Functions

I

icount, intarray Functions and Operators
ICU, PostgreSQL Features, Managing Collations, Parameters
ident, Ident Authentication
identifier, Identifiers and Key Words
length, Identifiers and Key Words
syntax of, Identifiers and Key Words
IDENTIFY_SYSTEM, Streaming Replication Protocol
ident_file configuration parameter, File Locations
idle_in_transaction_session_timeout configuration parameter, Statement Behavior
idle_session_timeout configuration parameter, Statement Behavior
idx, intarray Functions and Operators
IFNULL, COALESCE
ignore_checksum_failure configuration parameter, Developer Options
ignore_invalid_pages configuration parameter, Developer Options
ignore_system_indexes configuration parameter, Developer Options
IMMUTABLE, Function Volatility Categories
IMPORT FOREIGN SCHEMA, IMPORT FOREIGN SCHEMA
IN, Subquery Expressions, Row and Array Comparisons
INCLUDE
in index definitions, Index-Only Scans and Covering Indexes
include
in configuration file, Managing Configuration File Contents
include_dir
in configuration file, Managing Configuration File Contents
include_if_exists
in configuration file, Managing Configuration File Contents
index, Indexes, Multicolumn Indexes, Indexes and ORDER BY, Combining Multiple Indexes, Unique Indexes, Indexes on Expressions, Partial Indexes, Index-Only Scans and Covering Indexes, Index-Only Scans and Covering Indexes, Examining Index Usage, Preferred Index Types for Text Search, Preferred Index Types for Text Search, Locking and Indexes, Interfacing Extensions to Indexes, Building Indexes Concurrently, Rebuilding Indexes Concurrently, Operators and Functions
and ORDER BY, Indexes and ORDER BY
B-Tree, B-Tree, B-Tree Indexes
BRIN, BRIN, BRIN Indexes
building concurrently, Building Indexes Concurrently
combining multiple indexes, Combining Multiple Indexes
covering, Index-Only Scans and Covering Indexes
examining usage, Examining Index Usage
on expressions, Indexes on Expressions
for user-defined data type, Interfacing Extensions to Indexes
GIN, GIN, Preferred Index Types for Text Search, GIN Indexes
text search, Preferred Index Types for Text Search
GiST, GiST, Preferred Index Types for Text Search, GiST Indexes
text search, Preferred Index Types for Text Search
hash, Hash
Hash, Hash Indexes
index-only scans, Index-Only Scans and Covering Indexes
locks, Locking and Indexes
multicolumn, Multicolumn Indexes
partial, Partial Indexes
rebuilding concurrently, Rebuilding Indexes Concurrently
SP-GiST, SP-GiST, SP-GiST Indexes
unique, Unique Indexes
Index Access Method, Index Access Method Interface Definition
index scan, Planner Method Configuration
index-only scan, Index-Only Scans and Covering Indexes
indexam
Index Access Method, Index Access Method Interface Definition
index_am_handler, Pseudo-Types
inet (data type), inet
inet_client_addr, System Information Functions and Operators
inet_client_port, System Information Functions and Operators
inet_merge, Network Address Functions and Operators
inet_same_family, Network Address Functions and Operators
inet_server_addr, System Information Functions and Operators
inet_server_port, System Information Functions and Operators
infinity
floating point, Floating-Point Types
numeric (data type), Arbitrary Precision Numbers
information schema, The Information Schema
inheritance, Inheritance, Inheritance
initcap, String Functions and Operators
initdb, Creating a Database Cluster, initdb
Initialization Fork, The Initialization Fork
input function, User-Defined Types
INSERT, Populating a Table With Rows, Inserting Data, Returning Data from Modified Rows, INSERT
RETURNING, Returning Data from Modified Rows
inserting, Inserting Data
installation, Installation from Source Code
binaries, Installation from Binaries
on Windows, Installation from Source Code on Windows
instr function, Appendix
int2 (see smallint)
int4 (see integer)
int8 (see bigint)
intagg, intagg
intarray, intarray
integer, Numeric Constants, Integer Types
integer_datetimes configuration parameter, Preset Options
interfaces
externally maintained, Client Interfaces
internal, Pseudo-Types
INTERSECT, Combining Queries (UNION, INTERSECT, EXCEPT)
interval, Date/Time Types, Interval Input
output format, Interval Output
(see also formatting)
IntervalStyle configuration parameter, Locale and Formatting
intset, intarray Functions and Operators
int_array_aggregate, Functions
int_array_enum, Functions
inverse distribution, Aggregate Functions
in_hot_standby configuration parameter, Preset Options
in_range support functions, B-Tree Support Functions
IS DISTINCT FROM, Comparison Functions and Operators, Row and Array Comparisons
IS DOCUMENT, IS DOCUMENT
IS FALSE, Comparison Functions and Operators
IS NOT DISTINCT FROM, Comparison Functions and Operators, Row and Array Comparisons
IS NOT DOCUMENT, IS NOT DOCUMENT
IS NOT FALSE, Comparison Functions and Operators
IS NOT NULL, Comparison Functions and Operators
IS NOT TRUE, Comparison Functions and Operators
IS NOT UNKNOWN, Comparison Functions and Operators
IS NULL, Comparison Functions and Operators, Platform and Client Compatibility
IS TRUE, Comparison Functions and Operators
IS UNKNOWN, Comparison Functions and Operators
isclosed, Geometric Functions and Operators
isempty, Range/Multirange Functions and Operators
isfinite, Date/Time Functions and Operators
isn, isn
ISNULL, Comparison Functions and Operators
isn_weak, Functions and Operators
isopen, Geometric Functions and Operators
is_array_ref
in PL/Perl, Utility Functions in PL/Perl
is_valid, Functions and Operators

J

JIT, Just-in-Time Compilation (JIT)
jit configuration parameter, Other Planner Options
jit_above_cost configuration parameter, Planner Cost Constants
jit_debugging_support configuration parameter, Developer Options
jit_dump_bitcode configuration parameter, Developer Options
jit_expressions configuration parameter, Developer Options
jit_inline_above_cost configuration parameter, Planner Cost Constants
jit_optimize_above_cost configuration parameter, Planner Cost Constants
jit_profiling_support configuration parameter, Developer Options
jit_provider configuration parameter, Shared Library Preloading
jit_tuple_deforming configuration parameter, Developer Options
join, Joins Between Tables, Joined Tables, Controlling the Planner with Explicit JOIN Clauses
controlling the order, Controlling the Planner with Explicit JOIN Clauses
cross, Joined Tables
left, Joined Tables
natural, Joined Tables
outer, Joins Between Tables, Joined Tables
right, Joined Tables
self, Joins Between Tables
join_collapse_limit configuration parameter, Other Planner Options
JSON, JSON Types, JSON Functions and Operators
functions and operators, JSON Functions and Operators
JSONB, JSON Types
jsonb
containment, jsonb Containment and Existence
existence, jsonb Containment and Existence
indexes on, jsonb Indexing
jsonb_agg, Aggregate Functions
jsonb_array_elements, Processing and Creating JSON Data
jsonb_array_elements_text, Processing and Creating JSON Data
jsonb_array_length, Processing and Creating JSON Data
jsonb_build_array, Processing and Creating JSON Data
jsonb_build_object, Processing and Creating JSON Data
jsonb_each, Processing and Creating JSON Data
jsonb_each_text, Processing and Creating JSON Data
jsonb_extract_path, Processing and Creating JSON Data
jsonb_extract_path_text, Processing and Creating JSON Data
jsonb_insert, Processing and Creating JSON Data
jsonb_object, Processing and Creating JSON Data
jsonb_object_agg, Aggregate Functions
jsonb_object_keys, Processing and Creating JSON Data
jsonb_path_exists, Processing and Creating JSON Data
jsonb_path_exists_tz, Processing and Creating JSON Data
jsonb_path_match, Processing and Creating JSON Data
jsonb_path_match_tz, Processing and Creating JSON Data
jsonb_path_query, Processing and Creating JSON Data
jsonb_path_query_array, Processing and Creating JSON Data
jsonb_path_query_array_tz, Processing and Creating JSON Data
jsonb_path_query_first, Processing and Creating JSON Data
jsonb_path_query_first_tz, Processing and Creating JSON Data
jsonb_path_query_tz, Processing and Creating JSON Data
jsonb_populate_record, Processing and Creating JSON Data
jsonb_populate_recordset, Processing and Creating JSON Data
jsonb_pretty, Processing and Creating JSON Data
jsonb_set, Processing and Creating JSON Data
jsonb_set_lax, Processing and Creating JSON Data
jsonb_strip_nulls, Processing and Creating JSON Data
jsonb_to_record, Processing and Creating JSON Data
jsonb_to_recordset, Processing and Creating JSON Data
jsonb_to_tsvector, Text Search Functions and Operators
jsonb_typeof, Processing and Creating JSON Data
jsonpath, jsonpath Type
json_agg, Aggregate Functions
json_array_elements, Processing and Creating JSON Data
json_array_elements_text, Processing and Creating JSON Data
json_array_length, Processing and Creating JSON Data
json_build_array, Processing and Creating JSON Data
json_build_object, Processing and Creating JSON Data
json_each, Processing and Creating JSON Data
json_each_text, Processing and Creating JSON Data
json_extract_path, Processing and Creating JSON Data
json_extract_path_text, Processing and Creating JSON Data
json_object, Processing and Creating JSON Data
json_object_agg, Aggregate Functions
json_object_keys, Processing and Creating JSON Data
json_populate_record, Processing and Creating JSON Data
json_populate_recordset, Processing and Creating JSON Data
json_strip_nulls, Processing and Creating JSON Data
json_to_record, Processing and Creating JSON Data
json_to_recordset, Processing and Creating JSON Data
json_to_tsvector, Text Search Functions and Operators
json_typeof, Processing and Creating JSON Data
Julian date, Julian Dates
Just-In-Time compilation (see JIT)
justify_days, Date/Time Functions and Operators
justify_hours, Date/Time Functions and Operators
justify_interval, Date/Time Functions and Operators

K

key word, Identifiers and Key Words, SQL Key Words
list of, SQL Key Words
syntax of, Identifiers and Key Words
krb_caseins_users configuration parameter, Authentication
krb_server_keyfile configuration parameter, Authentication

L

label (see alias)
lag, Window Functions
language_handler, Pseudo-Types
large object, Large Objects
lastval, Sequence Manipulation Functions
last_value, Window Functions
LATERAL, LATERAL Subqueries
in the FROM clause, LATERAL Subqueries
latitude, Cube-Based Earth Distances
lca, Operators and Functions
lcm, Mathematical Functions and Operators
lc_collate configuration parameter, Preset Options
lc_ctype configuration parameter, Preset Options
lc_messages configuration parameter, Locale and Formatting
lc_monetary configuration parameter, Locale and Formatting
lc_numeric configuration parameter, Locale and Formatting
lc_time configuration parameter, Locale and Formatting
LDAP, PostgreSQL Features, LDAP Authentication
LDAP connection parameter lookup, LDAP Lookup of Connection Parameters
ldconfig, Shared Libraries
lead, Window Functions
LEAST, GREATEST and LEAST, UNION, CASE, and Related Constructs
determination of result type, UNION, CASE, and Related Constructs
left, String Functions and Operators
left join, Joined Tables
length, String Functions and Operators, Binary String Functions and Operators, Bit String Functions and Operators, Geometric Functions and Operators, Text Search Functions and Operators
of a binary string (see binary strings, length)
of a character string (see character string, length)
length(tsvector), Manipulating Documents
levenshtein, Levenshtein
levenshtein_less_equal, Levenshtein
lex, Requirements
libedit, Requirements
libperl, Requirements
libpq, libpq — C Library, Pipeline Mode, Retrieving Query Results Row-by-Row
pipeline mode, Pipeline Mode
single-row mode, Retrieving Query Results Row-by-Row
libpq-fe.h, libpq — C Library, Connection Status Functions
libpq-int.h, Connection Status Functions
libpython, Requirements
library finalization function, Dynamic Loading
library initialization function, Dynamic Loading
LIKE, LIKE
and locales, Behavior
LIKE_REGEX, Differences from XQuery (LIKE_REGEX), SQL/JSON Regular Expressions
in SQL/JSON, SQL/JSON Regular Expressions
LIMIT, LIMIT and OFFSET
line, Lines, Geometric Functions and Operators
line segment, Line Segments
linear regression, Aggregate Functions
Linux
IPC configuration, Shared Memory and Semaphores
shared library, Compiling and Linking Dynamically-Loaded Functions
start script, Starting the Database Server
LISTEN, LISTEN
listen_addresses configuration parameter, Connection Settings
llvm-config, PostgreSQL Features
ll_to_earth, Cube-Based Earth Distances
ln, Mathematical Functions and Operators
lo, lo
LOAD, LOAD
load balancing, High Availability, Load Balancing, and Replication
locale, Creating a Database Cluster, Locale Support
localtime, Date/Time Functions and Operators
localtimestamp, Date/Time Functions and Operators
local_preload_libraries configuration parameter, Shared Library Preloading
lock, Explicit Locking, Advisory Locks, Viewing Locks
advisory, Advisory Locks
monitoring, Viewing Locks
LOCK, Table-Level Locks, LOCK
lock_timeout configuration parameter, Statement Behavior
log, Mathematical Functions and Operators
log shipping, High Availability, Load Balancing, and Replication
log10, Mathematical Functions and Operators
Logging
current_logfiles file and the pg_current_logfile function, System Information Functions and Operators
pg_current_logfile function, System Information Functions and Operators
logging_collector configuration parameter, Where to Log
Logical Decoding, Logical Decoding, Logical Decoding
logical_decoding_work_mem configuration parameter, Memory
login privilege, Role Attributes
log_autovacuum_min_duration
configuration parameter, What to Log
storage parameter, Storage Parameters
log_btree_build_stats configuration parameter, Developer Options
log_checkpoints configuration parameter, What to Log
log_connections configuration parameter, What to Log
log_destination configuration parameter, Where to Log
log_directory configuration parameter, Where to Log
log_disconnections configuration parameter, What to Log
log_duration configuration parameter, What to Log
log_error_verbosity configuration parameter, What to Log
log_executor_stats configuration parameter, Statistics Monitoring
log_filename configuration parameter, Where to Log
log_file_mode configuration parameter, Where to Log
log_hostname configuration parameter, What to Log
log_line_prefix configuration parameter, What to Log
log_lock_waits configuration parameter, What to Log
log_min_duration_sample configuration parameter, When to Log
log_min_duration_statement configuration parameter, When to Log
log_min_error_statement configuration parameter, When to Log
log_min_messages configuration parameter, When to Log
log_parameter_max_length configuration parameter, What to Log
log_parameter_max_length_on_error configuration parameter, What to Log
log_parser_stats configuration parameter, Statistics Monitoring
log_planner_stats configuration parameter, Statistics Monitoring
log_recovery_conflict_waits configuration parameter, What to Log
log_replication_commands configuration parameter, What to Log
log_rotation_age configuration parameter, Where to Log
log_rotation_size configuration parameter, Where to Log
log_statement configuration parameter, What to Log
log_statement_sample_rate configuration parameter, When to Log
log_statement_stats configuration parameter, Statistics Monitoring
log_temp_files configuration parameter, What to Log
log_timezone configuration parameter, What to Log
log_transaction_sample_rate configuration parameter, When to Log
log_truncate_on_rotation configuration parameter, Where to Log
longitude, Cube-Based Earth Distances
looks_like_number
in PL/Perl, Utility Functions in PL/Perl
loop, Simple Loops
in PL/pgSQL, Simple Loops
lower, String Functions and Operators, Range/Multirange Functions and Operators
and locales, Behavior
lower_inc, Range/Multirange Functions and Operators
lower_inf, Range/Multirange Functions and Operators
lo_close, Closing a Large Object Descriptor
lo_compat_privileges configuration parameter, Previous PostgreSQL Versions
lo_creat, Creating a Large Object, Server-Side Functions
lo_create, Creating a Large Object
lo_export, Exporting a Large Object, Server-Side Functions
lo_from_bytea, Server-Side Functions
lo_get, Server-Side Functions
lo_import, Importing a Large Object, Server-Side Functions
lo_import_with_oid, Importing a Large Object
lo_lseek, Seeking in a Large Object
lo_lseek64, Seeking in a Large Object
lo_open, Opening an Existing Large Object
lo_put, Server-Side Functions
lo_read, Reading Data from a Large Object
lo_tell, Obtaining the Seek Position of a Large Object
lo_tell64, Obtaining the Seek Position of a Large Object
lo_truncate, Truncating a Large Object
lo_truncate64, Truncating a Large Object
lo_unlink, Removing a Large Object, Server-Side Functions
lo_write, Writing Data to a Large Object
lpad, String Functions and Operators
lseg, Line Segments, Geometric Functions and Operators
LSN, WAL Internals
ltree, ltree
ltree2text, Operators and Functions
ltrim, String Functions and Operators, Binary String Functions and Operators

M

MAC address (see macaddr)
MAC address (EUI-64 format) (see macaddr)
macaddr (data type), macaddr
macaddr8 (data type), macaddr8
macaddr8_set7bit, Network Address Functions and Operators
macOS, macOS
installation on, macOS
IPC configuration, Shared Memory and Semaphores
shared library, Compiling and Linking Dynamically-Loaded Functions
magic block, Dynamic Loading
maintenance, Routine Database Maintenance Tasks
maintenance_io_concurrency configuration parameter, Asynchronous Behavior
maintenance_work_mem configuration parameter, Memory
make, Requirements
makeaclitem, System Information Functions and Operators
make_date, Date/Time Functions and Operators
make_interval, Date/Time Functions and Operators
make_time, Date/Time Functions and Operators
make_timestamp, Date/Time Functions and Operators
make_timestamptz, Date/Time Functions and Operators
make_valid, Functions and Operators
MANPATH, Environment Variables
masklen, Network Address Functions and Operators
materialized view, Materialized Views
implementation through rules, Materialized Views
materialized views, pg_matviews
max, Aggregate Functions
max_connections configuration parameter, Connection Settings
max_files_per_process configuration parameter, Kernel Resource Usage
max_function_args configuration parameter, Preset Options
max_identifier_length configuration parameter, Preset Options
max_index_keys configuration parameter, Preset Options
max_locks_per_transaction configuration parameter, Lock Management
max_logical_replication_workers configuration parameter, Subscribers
max_parallel_maintenance_workers configuration parameter, Asynchronous Behavior
max_parallel_workers configuration parameter, Asynchronous Behavior
max_parallel_workers_per_gather configuration parameter, Asynchronous Behavior
max_pred_locks_per_page configuration parameter, Lock Management
max_pred_locks_per_relation configuration parameter, Lock Management
max_pred_locks_per_transaction configuration parameter, Lock Management
max_prepared_transactions configuration parameter, Memory
max_replication_slots configuration parameter, Sending Servers
max_slot_wal_keep_size configuration parameter, Sending Servers
max_stack_depth configuration parameter, Memory
max_standby_archive_delay configuration parameter, Standby Servers
max_standby_streaming_delay configuration parameter, Standby Servers
max_sync_workers_per_subscription configuration parameter, Subscribers
max_wal_senders configuration parameter, Sending Servers
max_wal_size configuration parameter, Checkpoints
max_worker_processes configuration parameter, Asynchronous Behavior
md5, String Functions and Operators, Binary String Functions and Operators
MD5, Password Authentication
median, Aggregate Expressions
(see also percentile)
memory context
in SPI, Memory Management
memory overcommit, Linux Memory Overcommit
metaphone, Metaphone
min, Aggregate Functions
MinGW, MinGW/Native Windows
installation on, MinGW/Native Windows
min_dynamic_shared_memory configuration parameter, Memory
min_parallel_index_scan_size configuration parameter, Planner Cost Constants
min_parallel_table_scan_size configuration parameter, Planner Cost Constants
min_scale, Mathematical Functions and Operators
min_wal_size configuration parameter, Checkpoints
mod, Mathematical Functions and Operators
mode
statistical, Aggregate Functions
monitoring, Monitoring Database Activity
database activity, Monitoring Database Activity
MOVE, MOVE
moving-aggregate mode, Moving-Aggregate Mode
multirange (function), Range/Multirange Functions and Operators
multirange type, Range Types
Multiversion Concurrency Control, Introduction
MultiXactId, Multixacts and Wraparound
MVCC, Introduction

N

name, Identifiers and Key Words
qualified, Creating a Schema
syntax of, Identifiers and Key Words
unqualified, The Schema Search Path
NaN (see not a number)
natural join, Joined Tables
negation, Logical Operators
NetBSD
IPC configuration, Shared Memory and Semaphores
shared library, Compiling and Linking Dynamically-Loaded Functions
start script, Starting the Database Server
netmask, Network Address Functions and Operators
network, Network Address Types, Network Address Functions and Operators
data types, Network Address Types
nextval, Sequence Manipulation Functions
NFS, NFS
nlevel, Operators and Functions
non-durable, Non-Durable Settings
nonblocking connection, Database Connection Control Functions, Asynchronous Command Processing
nonrepeatable read, Transaction Isolation
normalize, String Functions and Operators
normalized, String Functions and Operators
normal_rand, normal_rand
NOT (operator), Logical Operators
not a number
floating point, Floating-Point Types
numeric (data type), Arbitrary Precision Numbers
NOT IN, Subquery Expressions, Row and Array Comparisons
not-null constraint, Not-Null Constraints
notation, Calling Functions
functions, Calling Functions
notice processing, Notice Processing
in libpq, Notice Processing
notice processor, Notice Processing
notice receiver, Notice Processing
NOTIFY, Asynchronous Notification, NOTIFY
in libpq, Asynchronous Notification
NOTNULL, Comparison Functions and Operators
now, Date/Time Functions and Operators
npoints, Geometric Functions and Operators
nth_value, Window Functions
ntile, Window Functions
null value
with check constraints, Check Constraints
comparing, Comparison Functions and Operators
default value, Default Values
in DISTINCT, DISTINCT
in libpq, Retrieving Query Result Information
in PL/Perl, PL/Perl Functions and Arguments
in PL/Python, Null, None
with unique constraints, Unique Constraints
NULLIF, NULLIF
number
constant, Numeric Constants
numeric, Numeric Constants
numeric (data type), Arbitrary Precision Numbers
numnode, Text Search Functions and Operators, Manipulating Queries
num_nonnulls, Comparison Functions and Operators
num_nulls, Comparison Functions and Operators
NVL, COALESCE

O

object identifier, Object Identifier Types
data type, Object Identifier Types
object-oriented database, Concepts
obj_description, System Information Functions and Operators
octet_length, String Functions and Operators, Binary String Functions and Operators, Bit String Functions and Operators
OFFSET, LIMIT and OFFSET
oid, Object Identifier Types
OID
in libpq, Retrieving Other Result Information
oid2name, oid2name
old_snapshot, old_snapshot
old_snapshot_threshold configuration parameter, Asynchronous Behavior
ON CONFLICT, INSERT
ONLY, The FROM Clause
OOM, Linux Memory Overcommit
OpenBSD
IPC configuration, Shared Memory and Semaphores
shared library, Compiling and Linking Dynamically-Loaded Functions
start script, Starting the Database Server
OpenSSL, PostgreSQL Features
(see also SSL)
operator, Operators, Operator Precedence, Functions and Operators, Logical Operators, Operators, User-Defined Operators
invocation, Operator Invocations
logical, Logical Operators
precedence, Operator Precedence
syntax, Operators
type resolution in an invocation, Operators
user-defined, User-Defined Operators
operator class, Operator Classes and Operator Families, Index Methods and Operator Classes
operator family, Operator Classes and Operator Families, Operator Classes and Operator Families
optimization information, Function Optimization Information, Operator Optimization Information
for functions, Function Optimization Information
for operators, Operator Optimization Information
OR (operator), Logical Operators
Oracle, Porting from Oracle PL/SQL
porting from PL/SQL to PL/pgSQL, Porting from Oracle PL/SQL
ORDER BY, Querying a Table, Sorting Rows (ORDER BY)
and locales, Behavior
ordered-set aggregate, Aggregate Expressions
built-in, Aggregate Functions
ordering operator, System Dependencies on Operator Classes
ordinality, Set Returning Functions
outer join, Joined Tables
output function, User-Defined Types
OVER clause, Window Function Calls
overcommit, Linux Memory Overcommit
OVERLAPS, Date/Time Functions and Operators
overlay, String Functions and Operators, Binary String Functions and Operators, Bit String Functions and Operators
overloading, Function Overloading
functions, Function Overloading
operators, User-Defined Operators
owner, Privileges

P

pageinspect, pageinspect
pages_per_range storage parameter, Index Storage Parameters
page_checksum, General Functions
page_header, General Functions
palloc, Writing Code
PAM, PostgreSQL Features, PAM Authentication
parallel query, Parallel Query
parallel_leader_participation configuration parameter, Asynchronous Behavior
parallel_setup_cost configuration parameter, Planner Cost Constants
parallel_tuple_cost configuration parameter, Planner Cost Constants
parallel_workers storage parameter, Storage Parameters
parameter
syntax, Positional Parameters
parenthesis, Value Expressions
parse_ident, String Functions and Operators
partition pruning, Partition Pruning
partitioned table, Table Partitioning
partitioning, Table Partitioning
password, Role Attributes
authentication, Password Authentication
of the superuser, Creating a Database Cluster
password file, The Password File
passwordcheck, passwordcheck
password_encryption configuration parameter, Authentication
path, Geometric Functions and Operators
for schemas, Statement Behavior
PATH, Environment Variables
path (data type), Paths
pattern matching, Pattern Matching
patterns
in psql and pg_dump, Patterns
pclose, Geometric Functions and Operators
peer, Peer Authentication
percentile
continuous, Aggregate Functions
discrete, Aggregate Functions
percent_rank, Window Functions
hypothetical, Aggregate Functions
performance, Performance Tips
perl, Requirements
Perl, PL/Perl — Perl Procedural Language
permission (see privilege)
pfree, Writing Code
PGAPPNAME, Environment Variables
pgbench, pgbench
PGcancel, Canceling Queries in Progress
PGCHANNELBINDING, Environment Variables
PGCLIENTENCODING, Environment Variables
PGconn, Database Connection Control Functions
PGCONNECT_TIMEOUT, Environment Variables
pgcrypto, pgcrypto
PGDATA, Creating a Database Cluster
PGDATABASE, Environment Variables
PGDATESTYLE, Environment Variables
PGEventProc, Event Callback Procedure
PGGEQO, Environment Variables
PGGSSENCMODE, Environment Variables
PGGSSLIB, Environment Variables
PGHOST, Environment Variables
PGHOSTADDR, Environment Variables
PGKRBSRVNAME, Environment Variables
PGLOCALEDIR, Environment Variables
PGOPTIONS, Environment Variables
PGPASSFILE, Environment Variables
PGPASSWORD, Environment Variables
PGPORT, Environment Variables
pgp_armor_headers, pgp_armor_headers
pgp_key_id, pgp_key_id()
pgp_pub_decrypt, pgp_pub_decrypt()
pgp_pub_decrypt_bytea, pgp_pub_decrypt()
pgp_pub_encrypt, pgp_pub_encrypt()
pgp_pub_encrypt_bytea, pgp_pub_encrypt()
pgp_sym_decrypt, pgp_sym_decrypt()
pgp_sym_decrypt_bytea, pgp_sym_decrypt()
pgp_sym_encrypt, pgp_sym_encrypt()
pgp_sym_encrypt_bytea, pgp_sym_encrypt()
PGREQUIREPEER, Environment Variables
PGREQUIRESSL, Environment Variables
PGresult, Main Functions
pgrowlocks, pgrowlocks, Overview
PGSERVICE, Environment Variables
PGSERVICEFILE, Environment Variables
PGSSLCERT, Environment Variables
PGSSLCOMPRESSION, Environment Variables
PGSSLCRL, Environment Variables
PGSSLCRLDIR, Environment Variables
PGSSLKEY, Environment Variables
PGSSLMAXPROTOCOLVERSION, Environment Variables
PGSSLMINPROTOCOLVERSION, Environment Variables
PGSSLMODE, Environment Variables
PGSSLROOTCERT, Environment Variables
PGSSLSNI, Environment Variables
pgstatginindex, Functions
pgstathashindex, Functions
pgstatindex, Functions
pgstattuple, pgstattuple, Functions
pgstattuple_approx, Functions
PGSYSCONFDIR, Environment Variables
PGTARGETSESSIONATTRS, Environment Variables
PGTZ, Environment Variables
PGUSER, Environment Variables
pgxs, Extension Building Infrastructure
pg_advisory_lock, Advisory Lock Functions
pg_advisory_lock_shared, Advisory Lock Functions
pg_advisory_unlock, Advisory Lock Functions
pg_advisory_unlock_all, Advisory Lock Functions
pg_advisory_unlock_shared, Advisory Lock Functions
pg_advisory_xact_lock, Advisory Lock Functions
pg_advisory_xact_lock_shared, Advisory Lock Functions
pg_aggregate, pg_aggregate
pg_am, pg_am
pg_amcheck, pg_amcheck
pg_amop, pg_amop
pg_amproc, pg_amproc
pg_archivecleanup, pg_archivecleanup
pg_attrdef, pg_attrdef
pg_attribute, pg_attribute
pg_authid, pg_authid
pg_auth_members, pg_auth_members
pg_available_extensions, pg_available_extensions
pg_available_extension_versions, pg_available_extension_versions
pg_backend_memory_contexts, pg_backend_memory_contexts
pg_backend_pid, System Information Functions and Operators
pg_backup_start_time, Backup Control Functions
pg_basebackup, pg_basebackup
pg_blocking_pids, System Information Functions and Operators
pg_buffercache, pg_buffercache
pg_buffercache_pages, pg_buffercache
pg_cancel_backend, Server Signaling Functions
pg_cast, pg_cast
pg_checksums, pg_checksums
pg_class, pg_class
pg_client_encoding, String Functions and Operators
pg_collation, pg_collation
pg_collation_actual_version, Database Object Management Functions
pg_collation_is_visible, System Information Functions and Operators
PG_COLOR, When Color is Used
PG_COLORS, Configuring the Colors
pg_column_compression, Database Object Management Functions
pg_column_size, Database Object Management Functions
pg_config, pg_config, pg_config
with ecpg, Processing Embedded SQL Programs
with libpq, Building libpq Programs
with user-defined C functions, Writing Code
pg_conf_load_time, System Information Functions and Operators
pg_constraint, pg_constraint
pg_controldata, pg_controldata
pg_control_checkpoint, System Information Functions and Operators
pg_control_init, System Information Functions and Operators
pg_control_recovery, System Information Functions and Operators
pg_control_system, System Information Functions and Operators
pg_conversion, pg_conversion
pg_conversion_is_visible, System Information Functions and Operators
pg_copy_logical_replication_slot, Replication Management Functions
pg_copy_physical_replication_slot, Replication Management Functions
pg_create_logical_replication_slot, Replication Management Functions
pg_create_physical_replication_slot, Replication Management Functions
pg_create_restore_point, Backup Control Functions
pg_ctl, Creating a Database Cluster, Starting the Database Server, pg_ctl
pg_current_logfile, System Information Functions and Operators
pg_current_snapshot, System Information Functions and Operators
pg_current_wal_flush_lsn, Backup Control Functions
pg_current_wal_insert_lsn, Backup Control Functions
pg_current_wal_lsn, Backup Control Functions
pg_current_xact_id, System Information Functions and Operators
pg_current_xact_id_if_assigned, System Information Functions and Operators
pg_cursors, pg_cursors
pg_database, Template Databases, pg_database
pg_database_size, Database Object Management Functions
pg_db_role_setting, pg_db_role_setting
pg_ddl_command, Pseudo-Types
pg_default_acl, pg_default_acl
pg_depend, pg_depend
pg_describe_object, System Information Functions and Operators
pg_description, pg_description
pg_drop_replication_slot, Replication Management Functions
pg_dump, pg_dump
pg_dumpall, pg_dumpall
use during upgrade, Upgrading Data via pg_dumpall
pg_enum, pg_enum
pg_event_trigger, pg_event_trigger
pg_event_trigger_ddl_commands, Capturing Changes at Command End
pg_event_trigger_dropped_objects, Processing Objects Dropped by a DDL Command
pg_event_trigger_table_rewrite_oid, Handling a Table Rewrite Event
pg_event_trigger_table_rewrite_reason, Handling a Table Rewrite Event
pg_export_snapshot, Snapshot Synchronization Functions
pg_extension, pg_extension
pg_extension_config_dump, Extension Configuration Tables
pg_filenode_relation, Database Object Management Functions
pg_file_rename, adminpack
pg_file_settings, pg_file_settings
pg_file_sync, adminpack
pg_file_unlink, adminpack
pg_file_write, adminpack
pg_foreign_data_wrapper, pg_foreign_data_wrapper
pg_foreign_server, pg_foreign_server
pg_foreign_table, pg_foreign_table
pg_freespace, Functions
pg_freespacemap, pg_freespacemap
pg_function_is_visible, System Information Functions and Operators
pg_get_catalog_foreign_keys, System Information Functions and Operators
pg_get_constraintdef, System Information Functions and Operators
pg_get_expr, System Information Functions and Operators
pg_get_functiondef, System Information Functions and Operators
pg_get_function_arguments, System Information Functions and Operators
pg_get_function_identity_arguments, System Information Functions and Operators
pg_get_function_result, System Information Functions and Operators
pg_get_indexdef, System Information Functions and Operators
pg_get_keywords, System Information Functions and Operators
pg_get_object_address, System Information Functions and Operators
pg_get_ruledef, System Information Functions and Operators
pg_get_serial_sequence, System Information Functions and Operators
pg_get_statisticsobjdef, System Information Functions and Operators
pg_get_triggerdef, System Information Functions and Operators
pg_get_userbyid, System Information Functions and Operators
pg_get_viewdef, System Information Functions and Operators
pg_get_wal_replay_pause_state, Recovery Control Functions
pg_group, pg_group
pg_has_role, System Information Functions and Operators
pg_hba.conf, The pg_hba.conf File
pg_hba_file_rules, pg_hba_file_rules
pg_ident.conf, User Name Maps
pg_identify_object, System Information Functions and Operators
pg_identify_object_as_address, System Information Functions and Operators
pg_import_system_collations, Database Object Management Functions
pg_index, pg_index
pg_indexam_has_property, System Information Functions and Operators
pg_indexes, pg_indexes
pg_indexes_size, Database Object Management Functions
pg_index_column_has_property, System Information Functions and Operators
pg_index_has_property, System Information Functions and Operators
pg_inherits, pg_inherits
pg_init_privs, pg_init_privs
pg_isready, pg_isready
pg_is_in_backup, Backup Control Functions
pg_is_in_recovery, Recovery Control Functions
pg_is_other_temp_schema, System Information Functions and Operators
pg_is_wal_replay_paused, Recovery Control Functions
pg_jit_available, System Information Functions and Operators
pg_language, pg_language
pg_largeobject, pg_largeobject
pg_largeobject_metadata, pg_largeobject_metadata
pg_last_committed_xact, System Information Functions and Operators
pg_last_wal_receive_lsn, Recovery Control Functions
pg_last_wal_replay_lsn, Recovery Control Functions
pg_last_xact_replay_timestamp, Recovery Control Functions
pg_listening_channels, System Information Functions and Operators
pg_locks, pg_locks
pg_logdir_ls, adminpack
pg_logical_emit_message, Replication Management Functions
pg_logical_slot_get_binary_changes, Replication Management Functions
pg_logical_slot_get_changes, Replication Management Functions
pg_logical_slot_peek_binary_changes, Replication Management Functions
pg_logical_slot_peek_changes, Replication Management Functions
pg_log_backend_memory_contexts, Server Signaling Functions
pg_lsn, pg_lsn Type
pg_ls_archive_statusdir, Generic File Access Functions
pg_ls_dir, Generic File Access Functions
pg_ls_logdir, Generic File Access Functions
pg_ls_tmpdir, Generic File Access Functions
pg_ls_waldir, Generic File Access Functions
pg_matviews, pg_matviews
pg_mcv_list_items, Inspecting MCV Lists
pg_my_temp_schema, System Information Functions and Operators
pg_namespace, pg_namespace
pg_notification_queue_usage, System Information Functions and Operators
pg_notify, pg_notify
pg_opclass, pg_opclass
pg_opclass_is_visible, System Information Functions and Operators
pg_operator, pg_operator
pg_operator_is_visible, System Information Functions and Operators
pg_opfamily, pg_opfamily
pg_opfamily_is_visible, System Information Functions and Operators
pg_options_to_table, System Information Functions and Operators
pg_partitioned_table, pg_partitioned_table
pg_partition_ancestors, Database Object Management Functions
pg_partition_root, Database Object Management Functions
pg_partition_tree, Database Object Management Functions
pg_policies, pg_policies
pg_policy, pg_policy
pg_postmaster_start_time, System Information Functions and Operators
pg_prepared_statements, pg_prepared_statements
pg_prepared_xacts, pg_prepared_xacts
pg_prewarm, pg_prewarm
pg_prewarm.autoprewarm configuration parameter, Configuration Parameters
pg_prewarm.autoprewarm_interval configuration parameter, Configuration Parameters
pg_proc, pg_proc
pg_promote, Recovery Control Functions
pg_publication, pg_publication
pg_publication_rel, pg_publication_rel
pg_publication_tables, pg_publication_tables
pg_range, pg_range
pg_read_binary_file, Generic File Access Functions
pg_read_file, Generic File Access Functions
pg_receivewal, pg_receivewal
pg_receivexlog, pg_receivexlog renamed to pg_receivewal (see pg_receivewal)
pg_recvlogical, pg_recvlogical
pg_relation_filenode, Database Object Management Functions
pg_relation_filepath, Database Object Management Functions
pg_relation_size, Database Object Management Functions
pg_reload_conf, Server Signaling Functions
pg_relpages, Functions
pg_replication_origin, pg_replication_origin
pg_replication_origin_advance, Replication Management Functions
pg_replication_origin_create, Replication Management Functions
pg_replication_origin_drop, Replication Management Functions
pg_replication_origin_oid, Replication Management Functions
pg_replication_origin_progress, Replication Management Functions
pg_replication_origin_session_is_setup, Replication Management Functions
pg_replication_origin_session_progress, Replication Management Functions
pg_replication_origin_session_reset, Replication Management Functions
pg_replication_origin_session_setup, Replication Management Functions
pg_replication_origin_status, pg_replication_origin_status
pg_replication_origin_xact_reset, Replication Management Functions
pg_replication_origin_xact_setup, Replication Management Functions
pg_replication_slots, pg_replication_slots
pg_replication_slot_advance, Replication Management Functions
pg_resetwal, pg_resetwal
pg_resetxlog, pg_resetxlog renamed to pg_resetwal (see pg_resetwal)
pg_restore, pg_restore
pg_rewind, pg_rewind
pg_rewrite, pg_rewrite
pg_roles, pg_roles
pg_rotate_logfile, Server Signaling Functions
pg_rules, pg_rules
pg_safe_snapshot_blocking_pids, System Information Functions and Operators
pg_seclabel, pg_seclabel
pg_seclabels, pg_seclabels
pg_sequence, pg_sequence
pg_sequences, pg_sequences
pg_service.conf, The Connection Service File
pg_settings, pg_settings
pg_shadow, pg_shadow
pg_shdepend, pg_shdepend
pg_shdescription, pg_shdescription
pg_shmem_allocations, pg_shmem_allocations
pg_shseclabel, pg_shseclabel
pg_size_bytes, Database Object Management Functions
pg_size_pretty, Database Object Management Functions
pg_sleep, Delaying Execution
pg_sleep_for, Delaying Execution
pg_sleep_until, Delaying Execution
pg_snapshot_xip, System Information Functions and Operators
pg_snapshot_xmax, System Information Functions and Operators
pg_snapshot_xmin, System Information Functions and Operators
pg_start_backup, Backup Control Functions
pg_statio_all_indexes, Viewing Statistics, pg_statio_all_indexes
pg_statio_all_sequences, Viewing Statistics, pg_statio_all_sequences
pg_statio_all_tables, Viewing Statistics, pg_statio_all_tables
pg_statio_sys_indexes, Viewing Statistics
pg_statio_sys_sequences, Viewing Statistics
pg_statio_sys_tables, Viewing Statistics
pg_statio_user_indexes, Viewing Statistics
pg_statio_user_sequences, Viewing Statistics
pg_statio_user_tables, Viewing Statistics
pg_statistic, Single-Column Statistics, pg_statistic
pg_statistics_obj_is_visible, System Information Functions and Operators
pg_statistic_ext, Extended Statistics, pg_statistic_ext
pg_statistic_ext_data, Extended Statistics, pg_statistic_ext
pg_stats, Single-Column Statistics, pg_stats
pg_stats_ext, pg_stats_ext
pg_stats_ext_exprs, pg_stats_ext_exprs
pg_stat_activity, Viewing Statistics, pg_stat_activity
pg_stat_all_indexes, Viewing Statistics, pg_stat_all_indexes
pg_stat_all_tables, Viewing Statistics, pg_stat_all_tables
pg_stat_archiver, Viewing Statistics, pg_stat_archiver
pg_stat_bgwriter, Viewing Statistics, pg_stat_bgwriter
pg_stat_clear_snapshot, Statistics Functions
pg_stat_database, Viewing Statistics, pg_stat_database
pg_stat_database_conflicts, Viewing Statistics, pg_stat_database_conflicts
pg_stat_file, Generic File Access Functions
pg_stat_get_activity, Statistics Functions
pg_stat_get_backend_activity, Statistics Functions
pg_stat_get_backend_activity_start, Statistics Functions
pg_stat_get_backend_client_addr, Statistics Functions
pg_stat_get_backend_client_port, Statistics Functions
pg_stat_get_backend_dbid, Statistics Functions
pg_stat_get_backend_idset, Statistics Functions
pg_stat_get_backend_pid, Statistics Functions
pg_stat_get_backend_start, Statistics Functions
pg_stat_get_backend_userid, Statistics Functions
pg_stat_get_backend_wait_event, Statistics Functions
pg_stat_get_backend_wait_event_type, Statistics Functions
pg_stat_get_backend_xact_start, Statistics Functions
pg_stat_get_snapshot_timestamp, Statistics Functions
pg_stat_gssapi, Viewing Statistics, pg_stat_gssapi
pg_stat_progress_analyze, Viewing Statistics, ANALYZE Progress Reporting
pg_stat_progress_basebackup, Viewing Statistics, Base Backup Progress Reporting
pg_stat_progress_cluster, Viewing Statistics, CLUSTER Progress Reporting
pg_stat_progress_copy, Viewing Statistics, COPY Progress Reporting
pg_stat_progress_create_index, Viewing Statistics, CREATE INDEX Progress Reporting
pg_stat_progress_vacuum, Viewing Statistics, VACUUM Progress Reporting
pg_stat_replication, Viewing Statistics, pg_stat_replication
pg_stat_replication_slots, Viewing Statistics, pg_stat_replication_slots
pg_stat_reset, Statistics Functions
pg_stat_reset_replication_slot, Statistics Functions
pg_stat_reset_shared, Statistics Functions
pg_stat_reset_single_function_counters, Statistics Functions
pg_stat_reset_single_table_counters, Statistics Functions
pg_stat_reset_slru, Statistics Functions
pg_stat_slru, Viewing Statistics, pg_stat_slru
pg_stat_ssl, Viewing Statistics, pg_stat_ssl
pg_stat_statements, pg_stat_statements
function, Functions
pg_stat_statements_info, The pg_stat_statements_info View
pg_stat_statements_reset, Functions
pg_stat_subscription, Viewing Statistics, pg_stat_subscription
pg_stat_sys_indexes, Viewing Statistics
pg_stat_sys_tables, Viewing Statistics
pg_stat_user_functions, Viewing Statistics, pg_stat_user_functions
pg_stat_user_indexes, Viewing Statistics
pg_stat_user_tables, Viewing Statistics
pg_stat_wal, Viewing Statistics, pg_stat_wal
pg_stat_wal_receiver, Viewing Statistics, pg_stat_wal_receiver
pg_stat_xact_all_tables, Viewing Statistics
pg_stat_xact_sys_tables, Viewing Statistics
pg_stat_xact_user_functions, Viewing Statistics
pg_stat_xact_user_tables, Viewing Statistics
pg_stop_backup, Backup Control Functions
pg_subscription, pg_subscription
pg_subscription_rel, pg_subscription_rel
pg_surgery, pg_surgery
pg_switch_wal, Backup Control Functions
pg_tables, pg_tables
pg_tablespace, pg_tablespace
pg_tablespace_databases, System Information Functions and Operators
pg_tablespace_location, System Information Functions and Operators
pg_tablespace_size, Database Object Management Functions
pg_table_is_visible, System Information Functions and Operators
pg_table_size, Database Object Management Functions
pg_temp, Statement Behavior
securing functions, Writing SECURITY DEFINER Functions Safely
pg_terminate_backend, Server Signaling Functions
pg_test_fsync, pg_test_fsync
pg_test_timing, pg_test_timing
pg_timezone_abbrevs, pg_timezone_abbrevs
pg_timezone_names, pg_timezone_names
pg_total_relation_size, Database Object Management Functions
pg_transform, pg_transform
pg_trgm, pg_trgm
pg_trgm.similarity_threshold configuration parameter, GUC Parameters
pg_trgm.strict_word_similarity_threshold configuration parameter, GUC Parameters
pg_trgm.word_similarity_threshold configuration parameter, GUC Parameters
pg_trigger, pg_trigger
pg_trigger_depth, System Information Functions and Operators
pg_try_advisory_lock, Advisory Lock Functions
pg_try_advisory_lock_shared, Advisory Lock Functions
pg_try_advisory_xact_lock, Advisory Lock Functions
pg_try_advisory_xact_lock_shared, Advisory Lock Functions
pg_ts_config, pg_ts_config
pg_ts_config_is_visible, System Information Functions and Operators
pg_ts_config_map, pg_ts_config_map
pg_ts_dict, pg_ts_dict
pg_ts_dict_is_visible, System Information Functions and Operators
pg_ts_parser, pg_ts_parser
pg_ts_parser_is_visible, System Information Functions and Operators
pg_ts_template, pg_ts_template
pg_ts_template_is_visible, System Information Functions and Operators
pg_type, pg_type
pg_typeof, System Information Functions and Operators
pg_type_is_visible, System Information Functions and Operators
pg_upgrade, pg_upgrade
pg_user, pg_user
pg_user_mapping, pg_user_mapping
pg_user_mappings, pg_user_mappings
pg_verifybackup, pg_verifybackup
pg_views, pg_views
pg_visibility, pg_visibility
pg_visible_in_snapshot, System Information Functions and Operators
pg_waldump, pg_waldump
pg_walfile_name, Backup Control Functions
pg_walfile_name_offset, Backup Control Functions
pg_wal_lsn_diff, Backup Control Functions
pg_wal_replay_pause, Recovery Control Functions
pg_wal_replay_resume, Recovery Control Functions
pg_xact_commit_timestamp, System Information Functions and Operators
pg_xact_commit_timestamp_origin, System Information Functions and Operators
pg_xact_status, System Information Functions and Operators
pg_xlogdump, pg_xlogdump renamed to pg_waldump (see pg_waldump)
phantom read, Transaction Isolation
phraseto_tsquery, Text Search Functions and Operators, Parsing Queries
pi, Mathematical Functions and Operators
PIC, Compiling and Linking Dynamically-Loaded Functions
PID
determining PID of server process
in libpq, Connection Status Functions
pipelining, Pipeline Mode
in libpq, Pipeline Mode
PITR, Backup and Restore
PITR standby, High Availability, Load Balancing, and Replication
pkg-config, PostgreSQL Features
with ecpg, Processing Embedded SQL Programs
with libpq, Building libpq Programs
PL/Perl, PL/Perl — Perl Procedural Language
PL/PerlU, Trusted and Untrusted PL/Perl
PL/pgSQL, PL/pgSQL — SQL Procedural Language
PL/Python, PL/Python — Python Procedural Language
PL/SQL (Oracle), Porting from Oracle PL/SQL
porting to PL/pgSQL, Porting from Oracle PL/SQL
PL/Tcl, PL/Tcl — Tcl Procedural Language
plainto_tsquery, Text Search Functions and Operators, Parsing Queries
plan_cache_mode configuration parameter, Other Planner Options
plperl.on_init configuration parameter, Configuration
plperl.on_plperlu_init configuration parameter, Configuration
plperl.on_plperl_init configuration parameter, Configuration
plperl.use_strict configuration parameter, Configuration
plpgsql.check_asserts configuration parameter, Checking Assertions
plpgsql.variable_conflict configuration parameter, Variable Substitution
pltcl.start_proc configuration parameter, PL/Tcl Configuration
pltclu.start_proc configuration parameter, PL/Tcl Configuration
point, Points, Geometric Functions and Operators
point-in-time recovery, Backup and Restore
policy, Row Security Policies
polygon, Polygons, Geometric Functions and Operators
polymorphic function, Polymorphic Types
polymorphic type, Polymorphic Types
popcount (see bit_count)
popen, Geometric Functions and Operators
populate_record, hstore Operators and Functions
port, Parameter Key Words
port configuration parameter, Connection Settings
position, String Functions and Operators, Binary String Functions and Operators, Bit String Functions and Operators
POSTGRES, The Berkeley POSTGRES Project
postgres, Architectural Fundamentals, Starting the Database Server, Creating a Database, postgres
postgres user, The PostgreSQL User Account
Postgres95, Postgres95
postgresql.auto.conf, Parameter Interaction via the Configuration File
postgresql.conf, Parameter Interaction via the Configuration File
postgres_fdw, postgres_fdw
postmaster, postmaster
post_auth_delay configuration parameter, Developer Options
power, Mathematical Functions and Operators
PQbackendPID, Connection Status Functions
PQbinaryTuples, Retrieving Query Result Information
with COPY, Functions Associated with the COPY Command
PQcancel, Canceling Queries in Progress
PQclear, Main Functions
PQclientEncoding, Control Functions
PQcmdStatus, Retrieving Other Result Information
PQcmdTuples, Retrieving Other Result Information
PQconndefaults, Database Connection Control Functions
PQconnectdb, Database Connection Control Functions
PQconnectdbParams, Database Connection Control Functions
PQconnectionNeedsPassword, Connection Status Functions
PQconnectionUsedPassword, Connection Status Functions
PQconnectPoll, Database Connection Control Functions
PQconnectStart, Database Connection Control Functions
PQconnectStartParams, Database Connection Control Functions
PQconninfo, Database Connection Control Functions
PQconninfoFree, Miscellaneous Functions
PQconninfoParse, Database Connection Control Functions
PQconsumeInput, Asynchronous Command Processing
PQcopyResult, Miscellaneous Functions
PQdb, Connection Status Functions
PQdescribePortal, Main Functions
PQdescribePrepared, Main Functions
PQencryptPassword, Miscellaneous Functions
PQencryptPasswordConn, Miscellaneous Functions
PQendcopy, Obsolete Functions for COPY
PQenterPipelineMode, Functions Associated with Pipeline Mode
PQerrorMessage, Connection Status Functions
PQescapeBytea, Escaping Strings for Inclusion in SQL Commands
PQescapeByteaConn, Escaping Strings for Inclusion in SQL Commands
PQescapeIdentifier, Escaping Strings for Inclusion in SQL Commands
PQescapeLiteral, Escaping Strings for Inclusion in SQL Commands
PQescapeString, Escaping Strings for Inclusion in SQL Commands
PQescapeStringConn, Escaping Strings for Inclusion in SQL Commands
PQexec, Main Functions
PQexecParams, Main Functions
PQexecPrepared, Main Functions
PQexitPipelineMode, Functions Associated with Pipeline Mode
PQfformat, Retrieving Query Result Information
with COPY, Functions Associated with the COPY Command
PQfinish, Database Connection Control Functions
PQfireResultCreateEvents, Miscellaneous Functions
PQflush, Asynchronous Command Processing
PQfmod, Retrieving Query Result Information
PQfn, The Fast-Path Interface
PQfname, Retrieving Query Result Information
PQfnumber, Retrieving Query Result Information
PQfreeCancel, Canceling Queries in Progress
PQfreemem, Miscellaneous Functions
PQfsize, Retrieving Query Result Information
PQftable, Retrieving Query Result Information
PQftablecol, Retrieving Query Result Information
PQftype, Retrieving Query Result Information
PQgetCancel, Canceling Queries in Progress
PQgetCopyData, Functions for Receiving COPY Data
PQgetisnull, Retrieving Query Result Information
PQgetlength, Retrieving Query Result Information
PQgetline, Obsolete Functions for COPY
PQgetlineAsync, Obsolete Functions for COPY
PQgetResult, Asynchronous Command Processing
PQgetssl, Connection Status Functions
PQgetSSLKeyPassHook_OpenSSL, Database Connection Control Functions
PQgetvalue, Retrieving Query Result Information
PQhost, Connection Status Functions
PQhostaddr, Connection Status Functions
PQinitOpenSSL, SSL Library Initialization
PQinitSSL, SSL Library Initialization
PQinstanceData, Event Support Functions
PQisBusy, Asynchronous Command Processing
PQisnonblocking, Asynchronous Command Processing
PQisthreadsafe, Behavior in Threaded Programs
PQlibVersion, Miscellaneous Functions
(see also PQserverVersion)
PQmakeEmptyPGresult, Miscellaneous Functions
PQnfields, Retrieving Query Result Information
with COPY, Functions Associated with the COPY Command
PQnotifies, Asynchronous Notification
PQnparams, Retrieving Query Result Information
PQntuples, Retrieving Query Result Information
PQoidStatus, Retrieving Other Result Information
PQoidValue, Retrieving Other Result Information
PQoptions, Connection Status Functions
PQparameterStatus, Connection Status Functions
PQparamtype, Retrieving Query Result Information
PQpass, Connection Status Functions
PQping, Database Connection Control Functions
PQpingParams, Database Connection Control Functions
PQpipelineStatus, Functions Associated with Pipeline Mode
PQpipelineSync, Functions Associated with Pipeline Mode
PQport, Connection Status Functions
PQprepare, Main Functions
PQprint, Retrieving Query Result Information
PQprotocolVersion, Connection Status Functions
PQputCopyData, Functions for Sending COPY Data
PQputCopyEnd, Functions for Sending COPY Data
PQputline, Obsolete Functions for COPY
PQputnbytes, Obsolete Functions for COPY
PQregisterEventProc, Event Support Functions
PQrequestCancel, Canceling Queries in Progress
PQreset, Database Connection Control Functions
PQresetPoll, Database Connection Control Functions
PQresetStart, Database Connection Control Functions
PQresStatus, Main Functions
PQresultAlloc, Miscellaneous Functions
PQresultErrorField, Main Functions
PQresultErrorMessage, Main Functions
PQresultInstanceData, Event Support Functions
PQresultMemorySize, Miscellaneous Functions
PQresultSetInstanceData, Event Support Functions
PQresultStatus, Main Functions
PQresultVerboseErrorMessage, Main Functions
PQsendDescribePortal, Asynchronous Command Processing
PQsendDescribePrepared, Asynchronous Command Processing
PQsendFlushRequest, Functions Associated with Pipeline Mode
PQsendPrepare, Asynchronous Command Processing
PQsendQuery, Asynchronous Command Processing
PQsendQueryParams, Asynchronous Command Processing
PQsendQueryPrepared, Asynchronous Command Processing
PQserverVersion, Connection Status Functions
PQsetClientEncoding, Control Functions
PQsetdb, Database Connection Control Functions
PQsetdbLogin, Database Connection Control Functions
PQsetErrorContextVisibility, Control Functions
PQsetErrorVerbosity, Control Functions
PQsetInstanceData, Event Support Functions
PQsetnonblocking, Asynchronous Command Processing
PQsetNoticeProcessor, Notice Processing
PQsetNoticeReceiver, Notice Processing
PQsetResultAttrs, Miscellaneous Functions
PQsetSingleRowMode, Retrieving Query Results Row-by-Row
PQsetSSLKeyPassHook_OpenSSL, Database Connection Control Functions
PQsetTraceFlags, Control Functions
PQsetvalue, Miscellaneous Functions
PQsocket, Connection Status Functions
PQsslAttribute, Connection Status Functions
PQsslAttributeNames, Connection Status Functions
PQsslInUse, Connection Status Functions
PQsslStruct, Connection Status Functions
PQstatus, Connection Status Functions
PQtrace, Control Functions
PQtransactionStatus, Connection Status Functions
PQtty, Connection Status Functions
PQunescapeBytea, Escaping Strings for Inclusion in SQL Commands
PQuntrace, Control Functions
PQuser, Connection Status Functions
predicate locking, Serializable Isolation Level
PREPARE, PREPARE
PREPARE TRANSACTION, PREPARE TRANSACTION
prepared statements, DEALLOCATE, EXECUTE, EXPLAIN, PREPARE
creating, PREPARE
executing, EXECUTE
removing, DEALLOCATE
showing the query plan, EXPLAIN
preparing a query
in PL/pgSQL, Plan Caching
in PL/Python, Database Access Functions
in PL/Tcl, Database Access from PL/Tcl
pre_auth_delay configuration parameter, Developer Options
primary key, Primary Keys
primary_conninfo configuration parameter, Standby Servers
primary_slot_name configuration parameter, Standby Servers
privilege, Privileges, Schemas and Privileges, Rules and Privileges, Rules and Privileges
querying, System Information Functions and Operators
with rules, Rules and Privileges
for schemas, Schemas and Privileges
with views, Rules and Privileges
procedural language, Procedural Languages, Writing a Procedural Language Handler
externally maintained, Procedural Languages
handler for, Writing a Procedural Language Handler
procedure, User-Defined Procedures
user-defined, User-Defined Procedures
procedures
output parameter, SQL Procedures with Output Parameters
promote_trigger_file configuration parameter, Standby Servers
protocol, Frontend/Backend Protocol
frontend-backend, Frontend/Backend Protocol
ps, Standard Unix Tools
to monitor activity, Standard Unix Tools
psql, Accessing a Database, psql
Python, PL/Python — Python Procedural Language

Q

qualified name, Creating a Schema
query, Querying a Table, Queries
query plan, Using EXPLAIN
query tree, The Query Tree
querytree, Text Search Functions and Operators, Manipulating Queries
quotation marks
and identifiers, Identifiers and Key Words
escaping, String Constants
quote_all_identifiers configuration parameter, Previous PostgreSQL Versions
quote_ident, String Functions and Operators
in PL/Perl, Utility Functions in PL/Perl
use in PL/pgSQL, Executing Dynamic Commands
quote_literal, String Functions and Operators
in PL/Perl, Utility Functions in PL/Perl
use in PL/pgSQL, Executing Dynamic Commands
quote_nullable, String Functions and Operators
in PL/Perl, Utility Functions in PL/Perl
use in PL/pgSQL, Executing Dynamic Commands

R

radians, Mathematical Functions and Operators
radius, Geometric Functions and Operators
RADIUS, RADIUS Authentication
RAISE
in PL/pgSQL, Reporting Errors and Messages
random, Mathematical Functions and Operators
random_page_cost configuration parameter, Planner Cost Constants
range table, The Query Tree
range type, Range Types
exclude, Constraints on Ranges
indexes on, Indexing
range_agg, Aggregate Functions
range_intersect_agg, Aggregate Functions
range_merge, Range/Multirange Functions and Operators
rank, Window Functions
hypothetical, Aggregate Functions
read committed, Read Committed Isolation Level
read-only transaction, Statement Behavior
setting, SET TRANSACTION
setting default, Statement Behavior
readline, Requirements
real, Floating-Point Types
REASSIGN OWNED, REASSIGN OWNED
record, Pseudo-Types
recovery.conf, recovery.conf file merged into postgresql.conf
recovery.signal, Archive Recovery
recovery_end_command configuration parameter, Archive Recovery
recovery_init_sync_method configuration parameter, Error Handling
recovery_min_apply_delay configuration parameter, Standby Servers
recovery_target configuration parameter, Recovery Target
recovery_target_action configuration parameter, Recovery Target
recovery_target_inclusive configuration parameter, Recovery Target
recovery_target_lsn configuration parameter, Recovery Target
recovery_target_name configuration parameter, Recovery Target
recovery_target_time configuration parameter, Recovery Target
recovery_target_timeline configuration parameter, Recovery Target
recovery_target_xid configuration parameter, Recovery Target
rectangle, Boxes
RECURSIVE, CREATE VIEW
in common table expressions, Recursive Queries
in views, CREATE VIEW
referential integrity, Foreign Keys, Foreign Keys
REFRESH MATERIALIZED VIEW, REFRESH MATERIALIZED VIEW
regclass, Object Identifier Types
regcollation, Object Identifier Types
regconfig, Object Identifier Types
regdictionary, Object Identifier Types
regexp_match, String Functions and Operators, POSIX Regular Expressions
regexp_matches, String Functions and Operators, POSIX Regular Expressions
regexp_replace, String Functions and Operators, POSIX Regular Expressions
regexp_split_to_array, String Functions and Operators, POSIX Regular Expressions
regexp_split_to_table, String Functions and Operators, POSIX Regular Expressions
regnamespace, Object Identifier Types
regoper, Object Identifier Types
regoperator, Object Identifier Types
regproc, Object Identifier Types
regprocedure, Object Identifier Types
regression intercept, Aggregate Functions
regression slope, Aggregate Functions
regression test, Installation Procedure
regression tests, Regression Tests
regrole, Object Identifier Types
regr_avgx, Aggregate Functions
regr_avgy, Aggregate Functions
regr_count, Aggregate Functions
regr_intercept, Aggregate Functions
regr_r2, Aggregate Functions
regr_slope, Aggregate Functions
regr_sxx, Aggregate Functions
regr_sxy, Aggregate Functions
regr_syy, Aggregate Functions
regtype, Object Identifier Types
regular expression, SIMILAR TO Regular Expressions, POSIX Regular Expressions
(see also pattern matching)
regular expressions
and locales, Behavior
reindex, Routine Reindexing
REINDEX, REINDEX
reindexdb, reindexdb
relation, Concepts
relational database, Concepts
RELEASE SAVEPOINT, RELEASE SAVEPOINT
remove_temp_files_after_crash configuration parameter, Developer Options
repeat, String Functions and Operators
repeatable read, Repeatable Read Isolation Level
replace, String Functions and Operators
replication, High Availability, Load Balancing, and Replication
Replication Origins, Replication Progress Tracking
Replication Progress Tracking, Replication Progress Tracking
replication slot
logical replication, Replication Slots
streaming replication, Replication Slots
reporting errors
in PL/pgSQL, Reporting Errors and Messages
RESET, RESET
restartpoint, WAL Configuration
restart_after_crash configuration parameter, Error Handling
restore_command configuration parameter, Archive Recovery
RESTRICT, Dependency Tracking
with DROP, Dependency Tracking
foreign key action, Foreign Keys
RETURN NEXT
in PL/pgSQL, RETURN NEXT and RETURN QUERY
RETURN QUERY
in PL/pgSQL, RETURN NEXT and RETURN QUERY
RETURNING, Returning Data from Modified Rows
RETURNING INTO, Executing a Command with a Single-Row Result
in PL/pgSQL, Executing a Command with a Single-Row Result
reverse, String Functions and Operators
REVOKE, Privileges, REVOKE
right, String Functions and Operators
right join, Joined Tables
role, Database Roles, Role Membership, Predefined Roles
applicable, applicable_roles
enabled, enabled_roles
membership in, Role Membership
privilege to create, Role Attributes
privilege to initiate replication, Role Attributes
ROLLBACK, ROLLBACK
rollback
psql, Variables
ROLLBACK PREPARED, ROLLBACK PREPARED
ROLLBACK TO SAVEPOINT, ROLLBACK TO SAVEPOINT
ROLLUP, GROUPING SETS, CUBE, and ROLLUP
round, Mathematical Functions and Operators
routine, User-Defined Procedures
routine maintenance, Routine Database Maintenance Tasks
row, Concepts, Table Basics
ROW, Row Constructors
row estimation, Row Estimation Examples
multivariate, Multivariate Statistics Examples
planner, Row Estimation Examples
row type, Composite Types
constructor, Row Constructors
row-level security, Row Security Policies
row-wise comparison, Row and Array Comparisons
row_number, Window Functions
row_security configuration parameter, Statement Behavior
row_security_active, System Information Functions and Operators
row_to_json, Processing and Creating JSON Data
rpad, String Functions and Operators
rtrim, String Functions and Operators, Binary String Functions and Operators
rule, The Rule System, Views and the Rule System, How SELECT Rules Work, Materialized Views, Rules on INSERT, UPDATE, and DELETE, Rules on INSERT, UPDATE, and DELETE, Rules on INSERT, UPDATE, and DELETE, Rules Versus Triggers
and materialized views, Materialized Views
and views, Views and the Rule System
for DELETE, Rules on INSERT, UPDATE, and DELETE
for INSERT, Rules on INSERT, UPDATE, and DELETE
for SELECT, How SELECT Rules Work
compared with triggers, Rules Versus Triggers
for UPDATE, Rules on INSERT, UPDATE, and DELETE

S

SAVEPOINT, SAVEPOINT
savepoints, RELEASE SAVEPOINT, ROLLBACK TO SAVEPOINT, SAVEPOINT
defining, SAVEPOINT
releasing, RELEASE SAVEPOINT
rolling back, ROLLBACK TO SAVEPOINT
scalar (see expression)
scale, Mathematical Functions and Operators
schema, Schemas, Creating a Schema, The Public Schema, Overview
creating, Creating a Schema
current, The Schema Search Path, System Information Functions and Operators
public, The Public Schema
removing, Creating a Schema
SCRAM, Password Authentication
search path, The Schema Search Path
current, System Information Functions and Operators
object visibility, System Information Functions and Operators
search_path configuration parameter, The Schema Search Path, Statement Behavior
use in securing functions, Writing SECURITY DEFINER Functions Safely
SECURITY LABEL, SECURITY LABEL
sec_to_gc, Cube-Based Earth Distances
seg, seg
segment_size configuration parameter, Preset Options
SELECT, Querying a Table, Queries, SELECT Output Columns, SELECT
determination of result type, SELECT Output Columns
select list, Select Lists
SELECT INTO, Executing a Command with a Single-Row Result, SELECT INTO
in PL/pgSQL, Executing a Command with a Single-Row Result
semaphores, Shared Memory and Semaphores
sepgsql, sepgsql
sepgsql.debug_audit configuration parameter, GUC Parameters
sepgsql.permissive configuration parameter, GUC Parameters
sequence, Sequence Manipulation Functions
and serial type, Serial Types
sequential scan, Planner Method Configuration
seq_page_cost configuration parameter, Planner Cost Constants
serial, Serial Types
serial2, Serial Types
serial4, Serial Types
serial8, Serial Types
serializable, Serializable Isolation Level
Serializable Snapshot Isolation, Introduction
serialization anomaly, Transaction Isolation, Serializable Isolation Level
server log, Error Reporting and Logging, Log File Maintenance
log file maintenance, Log File Maintenance
Server Name Indication, Parameter Key Words
server spoofing, Preventing Server Spoofing
server_encoding configuration parameter, Preset Options
server_version configuration parameter, Preset Options
server_version_num configuration parameter, Preset Options
session_preload_libraries configuration parameter, Shared Library Preloading
session_replication_role configuration parameter, Statement Behavior
session_user, System Information Functions and Operators
SET, Configuration Settings Functions, SET
SET CONSTRAINTS, SET CONSTRAINTS
set difference, Combining Queries (UNION, INTERSECT, EXCEPT)
set intersection, Combining Queries (UNION, INTERSECT, EXCEPT)
set operation, Combining Queries (UNION, INTERSECT, EXCEPT)
set returning functions, Set Returning Functions
functions, Set Returning Functions
SET ROLE, SET ROLE
SET SESSION AUTHORIZATION, SET SESSION AUTHORIZATION
SET TRANSACTION, SET TRANSACTION
set union, Combining Queries (UNION, INTERSECT, EXCEPT)
SET XML OPTION, Statement Behavior
setseed, Mathematical Functions and Operators
setval, Sequence Manipulation Functions
setweight, Text Search Functions and Operators, Manipulating Documents
setweight for specific lexeme(s), Text Search Functions and Operators
set_bit, Binary String Functions and Operators, Bit String Functions and Operators
set_byte, Binary String Functions and Operators
set_config, Configuration Settings Functions
set_limit, Functions and Operators
set_masklen, Network Address Functions and Operators
sha224, Binary String Functions and Operators
sha256, Binary String Functions and Operators
sha384, Binary String Functions and Operators
sha512, Binary String Functions and Operators
shared library, Shared Libraries, Compiling and Linking Dynamically-Loaded Functions
shared memory, Shared Memory and Semaphores
shared_buffers configuration parameter, Memory
shared_memory_type configuration parameter, Memory
shared_preload_libraries, Shared Memory and LWLocks
shared_preload_libraries configuration parameter, Shared Library Preloading
shobj_description, System Information Functions and Operators
SHOW, Configuration Settings Functions, SHOW, Streaming Replication Protocol
show_limit, Functions and Operators
show_trgm, Functions and Operators
shutdown, Shutting Down the Server
SIGHUP, Parameter Interaction via the Configuration File, The pg_hba.conf File, User Name Maps
SIGINT, Shutting Down the Server
sign, Mathematical Functions and Operators
signal
backend processes, Server Signaling Functions
significant digits, Locale and Formatting
SIGQUIT, Shutting Down the Server
SIGTERM, Shutting Down the Server
SIMILAR TO, SIMILAR TO Regular Expressions
similarity, Functions and Operators
sin, Mathematical Functions and Operators
sind, Mathematical Functions and Operators
single-user mode, Options for Single-User Mode
sinh, Mathematical Functions and Operators
skeys, hstore Operators and Functions
sleep, Delaying Execution
slice, hstore Operators and Functions
sliced bread (see TOAST)
slope, Geometric Functions and Operators
SLRU, pg_stat_slru
smallint, Integer Types
smallserial, Serial Types
Solaris, Solaris
installation on, Solaris
shared library, Compiling and Linking Dynamically-Loaded Functions
start script, Starting the Database Server
SOME, Aggregate Functions, Subquery Expressions, Row and Array Comparisons
sort, intarray Functions and Operators
sorting, Sorting Rows (ORDER BY)
sort_asc, intarray Functions and Operators
sort_desc, intarray Functions and Operators
soundex, Soundex
SP-GiST (see index)
SPI, Server Programming Interface, spi
examples, spi
spi_commit
in PL/Perl, Database Access from PL/Perl
SPI_commit, SPI_commit
SPI_commit_and_chain, SPI_commit
SPI_connect, SPI_connect
SPI_connect_ext, SPI_connect
SPI_copytuple, SPI_copytuple
spi_cursor_close
in PL/Perl, Database Access from PL/Perl
SPI_cursor_close, SPI_cursor_close
SPI_cursor_fetch, SPI_cursor_fetch
SPI_cursor_find, SPI_cursor_find
SPI_cursor_move, SPI_cursor_move
SPI_cursor_open, SPI_cursor_open
SPI_cursor_open_with_args, SPI_cursor_open_with_args
SPI_cursor_open_with_paramlist, SPI_cursor_open_with_paramlist
SPI_cursor_parse_open, SPI_cursor_parse_open
SPI_exec, SPI_exec
SPI_execp, SPI_execp
SPI_execute, SPI_execute
SPI_execute_extended, SPI_execute_extended
SPI_execute_plan, SPI_execute_plan
SPI_execute_plan_extended, SPI_execute_plan_extended
SPI_execute_plan_with_paramlist, SPI_execute_plan_with_paramlist
SPI_execute_with_args, SPI_execute_with_args
spi_exec_prepared
in PL/Perl, Database Access from PL/Perl
spi_exec_query
in PL/Perl, Database Access from PL/Perl
spi_fetchrow
in PL/Perl, Database Access from PL/Perl
SPI_finish, SPI_finish
SPI_fname, SPI_fname
SPI_fnumber, SPI_fnumber
spi_freeplan
in PL/Perl, Database Access from PL/Perl
SPI_freeplan, SPI_freeplan
SPI_freetuple, SPI_freetuple
SPI_freetuptable, SPI_freetuptable
SPI_getargcount, SPI_getargcount
SPI_getargtypeid, SPI_getargtypeid
SPI_getbinval, SPI_getbinval
SPI_getnspname, SPI_getnspname
SPI_getrelname, SPI_getrelname
SPI_gettype, SPI_gettype
SPI_gettypeid, SPI_gettypeid
SPI_getvalue, SPI_getvalue
SPI_is_cursor_plan, SPI_is_cursor_plan
SPI_keepplan, SPI_keepplan
SPI_modifytuple, SPI_modifytuple
SPI_palloc, SPI_palloc
SPI_pfree, SPI_pfree
spi_prepare
in PL/Perl, Database Access from PL/Perl
SPI_prepare, SPI_prepare
SPI_prepare_cursor, SPI_prepare_cursor
SPI_prepare_extended, SPI_prepare_extended
SPI_prepare_params, SPI_prepare_params
spi_query
in PL/Perl, Database Access from PL/Perl
spi_query_prepared
in PL/Perl, Database Access from PL/Perl
SPI_register_relation, SPI_register_relation
SPI_register_trigger_data, SPI_register_trigger_data
SPI_repalloc, SPI_repalloc
SPI_result_code_string, SPI_result_code_string
SPI_returntuple, SPI_returntuple
spi_rollback
in PL/Perl, Database Access from PL/Perl
SPI_rollback, SPI_rollback
SPI_rollback_and_chain, SPI_rollback
SPI_saveplan, SPI_saveplan
SPI_scroll_cursor_fetch, SPI_scroll_cursor_fetch
SPI_scroll_cursor_move, SPI_scroll_cursor_move
SPI_start_transaction, SPI_start_transaction
SPI_unregister_relation, SPI_unregister_relation
split_part, String Functions and Operators
SQL/CLI, SQL Conformance
SQL/Foundation, SQL Conformance
SQL/Framework, SQL Conformance
SQL/JRT, SQL Conformance
SQL/JSON path language, The SQL/JSON Path Language
SQL/MDA, SQL Conformance
SQL/MED, SQL Conformance
SQL/OLB, SQL Conformance
SQL/PSM, SQL Conformance
SQL/Schemata, SQL Conformance
SQL/XML, SQL Conformance
limits and conformance, XML Limits and Conformance to SQL/XML
sqrt, Mathematical Functions and Operators
ssh, Secure TCP/IP Connections with SSH Tunnels
SSI, Introduction
SSL, Secure TCP/IP Connections with SSL, SSL Support
in libpq, Connection Status Functions
with libpq, Parameter Key Words
ssl configuration parameter, SSL
sslinfo, sslinfo
ssl_ca_file configuration parameter, SSL
ssl_cert_file configuration parameter, SSL
ssl_cipher, Functions Provided
ssl_ciphers configuration parameter, SSL
ssl_client_cert_present, Functions Provided
ssl_client_dn, Functions Provided
ssl_client_dn_field, Functions Provided
ssl_client_serial, Functions Provided
ssl_crl_dir configuration parameter, SSL
ssl_crl_file configuration parameter, SSL
ssl_dh_params_file configuration parameter, SSL
ssl_ecdh_curve configuration parameter, SSL
ssl_extension_info, Functions Provided
ssl_issuer_dn, Functions Provided
ssl_issuer_field, Functions Provided
ssl_is_used, Functions Provided
ssl_key_file configuration parameter, SSL
ssl_library configuration parameter, Preset Options
ssl_max_protocol_version configuration parameter, SSL
ssl_min_protocol_version configuration parameter, SSL
ssl_passphrase_command configuration parameter, SSL
ssl_passphrase_command_supports_reload configuration parameter, SSL
ssl_prefer_server_ciphers configuration parameter, SSL
ssl_version, Functions Provided
SSPI, SSPI Authentication
STABLE, Function Volatility Categories
standard deviation, Aggregate Functions
population, Aggregate Functions
sample, Aggregate Functions
standard_conforming_strings configuration parameter, Previous PostgreSQL Versions
standby server, High Availability, Load Balancing, and Replication
standby.signal, Archive Recovery, Standby Server Operation, Setting Up a Standby Server
for hot standby, Administrator's Overview
pg_basebackup --write-recovery-conf, Options
standby_mode (see standby.signal)
START TRANSACTION, START TRANSACTION
starts_with, String Functions and Operators
START_REPLICATION, Streaming Replication Protocol
statement_timeout configuration parameter, Statement Behavior
statement_timestamp, Date/Time Functions and Operators
statistics, Aggregate Functions, Statistics Used by the Planner, Extended Statistics, Updating Planner Statistics, The Statistics Collector
of the planner, Statistics Used by the Planner, Extended Statistics, Updating Planner Statistics
stats_temp_directory configuration parameter, Query and Index Statistics Collector
stddev, Aggregate Functions
stddev_pop, Aggregate Functions
stddev_samp, Aggregate Functions
STONITH, High Availability, Load Balancing, and Replication
storage parameters, Storage Parameters
Streaming Replication, High Availability, Load Balancing, and Replication
strict_word_similarity, Functions and Operators
string (see character string)
strings
backslash quotes, Previous PostgreSQL Versions
escape warning, Previous PostgreSQL Versions
standard conforming, Previous PostgreSQL Versions
string_agg, Aggregate Functions
string_to_array, String Functions and Operators
string_to_table, String Functions and Operators
strip, Text Search Functions and Operators, Manipulating Documents
strpos, String Functions and Operators
subarray, intarray Functions and Operators
subltree, Operators and Functions
subpath, Operators and Functions
subquery, Aggregate Functions, Scalar Subqueries, Subqueries, Subquery Expressions
subscript, Subscripts
substr, String Functions and Operators, Binary String Functions and Operators
substring, String Functions and Operators, Binary String Functions and Operators, Bit String Functions and Operators, SIMILAR TO Regular Expressions, POSIX Regular Expressions
subtransactions
in PL/Tcl, Explicit Subtransactions in PL/Tcl
sum, Aggregate Functions
superuser, Accessing a Database, Role Attributes
superuser_reserved_connections configuration parameter, Connection Settings
support functions
in_range, B-Tree Support Functions
suppress_redundant_updates_trigger, Trigger Functions
svals, hstore Operators and Functions
synchronize_seqscans configuration parameter, Previous PostgreSQL Versions
synchronous commit, Asynchronous Commit
Synchronous Replication, High Availability, Load Balancing, and Replication
synchronous_commit configuration parameter, Settings
synchronous_standby_names configuration parameter, Primary Server
syntax, SQL Syntax
SQL, SQL Syntax
syslog_facility configuration parameter, Where to Log
syslog_ident configuration parameter, Where to Log
syslog_sequence_numbers configuration parameter, Where to Log
syslog_split_messages configuration parameter, Where to Log
system catalog, The System Catalog Schema
schema, The System Catalog Schema
systemd, PostgreSQL Features, Starting the Database Server
RemoveIPC, systemd RemoveIPC

T

table, Concepts, Table Basics, Modifying Tables
creating, Table Basics
inheritance, Inheritance
modifying, Modifying Tables
partitioning, Table Partitioning
removing, Table Basics
renaming, Renaming a Table
Table Access Method, Table Access Method Interface Definition
TABLE command, SELECT
table expression, Table Expressions
table function, Table Functions, xmltable
XMLTABLE, xmltable
table sampling method, Writing a Table Sampling Method
tableam
Table Access Method, Table Access Method Interface Definition
tablefunc, tablefunc
tableoid, System Columns
TABLESAMPLE method, Writing a Table Sampling Method
tablespace, Tablespaces
default, Statement Behavior
temporary, Statement Behavior
table_am_handler, Pseudo-Types
tan, Mathematical Functions and Operators
tand, Mathematical Functions and Operators
tanh, Mathematical Functions and Operators
target list, The Query Tree
Tcl, PL/Tcl — Tcl Procedural Language
tcn, tcn
tcp_keepalives_count configuration parameter, Connection Settings
tcp_keepalives_idle configuration parameter, Connection Settings
tcp_keepalives_interval configuration parameter, Connection Settings
tcp_user_timeout configuration parameter, Connection Settings
template0, Template Databases
template1, Creating a Database, Template Databases
temp_buffers configuration parameter, Memory
temp_file_limit configuration parameter, Disk
temp_tablespaces configuration parameter, Statement Behavior
test, Regression Tests
test_decoding, test_decoding
text, Character Types, Network Address Functions and Operators
text search, Text Search Types, Text Search Types, Full Text Search, Preferred Index Types for Text Search
data types, Text Search Types
functions and operators, Text Search Types
indexes, Preferred Index Types for Text Search
text2ltree, Operators and Functions
threads, Behavior in Threaded Programs
with libpq, Behavior in Threaded Programs
tid, Object Identifier Types
time, Date/Time Types, Times
constants, Special Values
current, Current Date/Time
output format, Date/Time Output
(see also formatting)
time span, Date/Time Types
time with time zone, Date/Time Types, Times
time without time zone, Date/Time Types, Times
time zone, Time Zones, Locale and Formatting, POSIX Time Zone Specifications
conversion, AT TIME ZONE
input abbreviations, Date/Time Configuration Files
POSIX-style specification, POSIX Time Zone Specifications
time zone data, Build Process Details
time zone names, Locale and Formatting
timelines, Backup and Restore
TIMELINE_HISTORY, Streaming Replication Protocol
timeofday, Date/Time Functions and Operators
timeout
client authentication, Authentication
deadlock, Lock Management
timestamp, Date/Time Types, Time Stamps
timestamp with time zone, Date/Time Types, Time Stamps
timestamp without time zone, Date/Time Types, Time Stamps
timestamptz, Date/Time Types
TimeZone configuration parameter, Locale and Formatting
timezone_abbreviations configuration parameter, Locale and Formatting
TOAST, TOAST
and user-defined types, TOAST Considerations
per-column storage settings, Description
per-type storage settings, Description
versus large objects, Introduction
toast_tuple_target storage parameter, Storage Parameters
token, Lexical Structure
to_ascii, String Functions and Operators
to_char, Data Type Formatting Functions
and locales, Behavior
to_date, Data Type Formatting Functions
to_hex, String Functions and Operators
to_json, Processing and Creating JSON Data
to_jsonb, Processing and Creating JSON Data
to_number, Data Type Formatting Functions
to_regclass, System Information Functions and Operators
to_regcollation, System Information Functions and Operators
to_regnamespace, System Information Functions and Operators
to_regoper, System Information Functions and Operators
to_regoperator, System Information Functions and Operators
to_regproc, System Information Functions and Operators
to_regprocedure, System Information Functions and Operators
to_regrole, System Information Functions and Operators
to_regtype, System Information Functions and Operators
to_timestamp, Data Type Formatting Functions, Date/Time Functions and Operators
to_tsquery, Text Search Functions and Operators, Parsing Queries
to_tsvector, Text Search Functions and Operators, Parsing Documents
trace_locks configuration parameter, Developer Options
trace_lock_oidmin configuration parameter, Developer Options
trace_lock_table configuration parameter, Developer Options
trace_lwlocks configuration parameter, Developer Options
trace_notify configuration parameter, Developer Options
trace_recovery_messages configuration parameter, Developer Options
trace_sort configuration parameter, Developer Options
trace_userlocks configuration parameter, Developer Options
track_activities configuration parameter, Query and Index Statistics Collector
track_activity_query_size configuration parameter, Query and Index Statistics Collector
track_commit_timestamp configuration parameter, Sending Servers
track_counts configuration parameter, Query and Index Statistics Collector
track_functions configuration parameter, Query and Index Statistics Collector
track_io_timing configuration parameter, Query and Index Statistics Collector
track_wal_io_timing configuration parameter, Query and Index Statistics Collector
transaction, Transactions
transaction ID, Preventing Transaction ID Wraparound Failures
wraparound, Preventing Transaction ID Wraparound Failures
transaction isolation, Transaction Isolation
transaction isolation level, Transaction Isolation, Statement Behavior
read committed, Read Committed Isolation Level
repeatable read, Repeatable Read Isolation Level
serializable, Serializable Isolation Level
setting, SET TRANSACTION
setting default, Statement Behavior
transaction log (see WAL)
transaction_deferrable configuration parameter, Statement Behavior
transaction_isolation configuration parameter, Statement Behavior
transaction_read_only configuration parameter, Statement Behavior
transaction_timestamp, Date/Time Functions and Operators
transform_null_equals configuration parameter, Platform and Client Compatibility
transition tables, CREATE TRIGGER
(see also ephemeral named relation)
implementation in PLs, SPI_register_trigger_data
referencing from C trigger, Writing Trigger Functions in C
translate, String Functions and Operators
transparent huge pages, Memory
trigger, Pseudo-Types, Triggers, Writing Trigger Functions in C, Rules Versus Triggers, Trigger Functions, Trigger Functions
arguments for trigger functions, Overview of Trigger Behavior
constraint trigger, Description
for updating a derived tsvector column, Triggers for Automatic Updates
in C, Writing Trigger Functions in C
in PL/pgSQL, Trigger Functions
in PL/Python, Trigger Functions
in PL/Tcl, Trigger Functions in PL/Tcl
compared with rules, Rules Versus Triggers
triggered_change_notification, tcn
trigger_file (see promote_trigger_file)
trim, String Functions and Operators, Binary String Functions and Operators
trim_array, Array Functions and Operators
trim_scale, Mathematical Functions and Operators
true, Boolean Type
trunc, Mathematical Functions and Operators, Network Address Functions and Operators
TRUNCATE, TRUNCATE
trusted, Trusted and Untrusted PL/Perl
PL/Perl, Trusted and Untrusted PL/Perl
tsm_handler, Pseudo-Types
tsm_system_rows, tsm_system_rows
tsm_system_time, tsm_system_time
tsquery (data type), tsquery
tsquery_phrase, Text Search Functions and Operators, Manipulating Queries
tsvector (data type), tsvector
tsvector concatenation, Manipulating Documents
tsvector_to_array, Text Search Functions and Operators
tsvector_update_trigger, Trigger Functions
tsvector_update_trigger_column, Trigger Functions
ts_debug, Text Search Functions and Operators, Configuration Testing
ts_delete, Text Search Functions and Operators
ts_filter, Text Search Functions and Operators
ts_headline, Text Search Functions and Operators, Highlighting Results
ts_lexize, Text Search Functions and Operators, Dictionary Testing
ts_parse, Text Search Functions and Operators, Parser Testing
ts_rank, Text Search Functions and Operators, Ranking Search Results
ts_rank_cd, Text Search Functions and Operators, Ranking Search Results
ts_rewrite, Text Search Functions and Operators, Query Rewriting
ts_stat, Text Search Functions and Operators, Gathering Document Statistics
ts_token_type, Text Search Functions and Operators, Parser Testing
tuple_data_split, Heap Functions
txid_current, System Information Functions and Operators
txid_current_if_assigned, System Information Functions and Operators
txid_current_snapshot, System Information Functions and Operators
txid_snapshot_xip, System Information Functions and Operators
txid_snapshot_xmax, System Information Functions and Operators
txid_snapshot_xmin, System Information Functions and Operators
txid_status, System Information Functions and Operators
txid_visible_in_snapshot, System Information Functions and Operators
type (see data type)
type cast, Numeric Constants, Type Casts

U

UESCAPE, Identifiers and Key Words, String Constants with Unicode Escapes
unaccent, unaccent, Functions
Unicode escape, String Constants with Unicode Escapes
in identifiers, Identifiers and Key Words
in string constants, String Constants with Unicode Escapes
Unicode normalization, String Functions and Operators
UNION, Combining Queries (UNION, INTERSECT, EXCEPT), UNION, CASE, and Related Constructs
determination of result type, UNION, CASE, and Related Constructs
uniq, intarray Functions and Operators
unique constraint, Unique Constraints
unistr, String Functions and Operators
Unix domain socket, Parameter Key Words
unix_socket_directories configuration parameter, Connection Settings
unix_socket_group configuration parameter, Connection Settings
unix_socket_permissions configuration parameter, Connection Settings
unknown, Pseudo-Types
UNLISTEN, UNLISTEN
unnest, Array Functions and Operators
for multirange, Range/Multirange Functions and Operators
for tsvector, Text Search Functions and Operators
unqualified name, The Schema Search Path
updatable views, Updatable Views
UPDATE, Updates, Updating Data, Returning Data from Modified Rows, UPDATE
RETURNING, Returning Data from Modified Rows
update_process_title configuration parameter, Process Title
updating, Updating Data
upgrading, Upgrading a PostgreSQL Cluster
upper, String Functions and Operators, Range/Multirange Functions and Operators
and locales, Behavior
upper_inc, Range/Multirange Functions and Operators
upper_inf, Range/Multirange Functions and Operators
UPSERT, INSERT
URI, Connection Strings
user, System Information Functions and Operators, Database Roles
current, System Information Functions and Operators
user mapping, Foreign Data
User name maps, User Name Maps
user_catalog_table storage parameter, Storage Parameters
UUID, UUID Type, UUID Type, PostgreSQL Features
generating, UUID Type
uuid-ossp, uuid-ossp
uuid_generate_v1, uuid-ossp Functions
uuid_generate_v1mc, uuid-ossp Functions
uuid_generate_v3, uuid-ossp Functions

V

vacuum, Routine Vacuuming
VACUUM, VACUUM
vacuumdb, vacuumdb
vacuumlo, vacuumlo
vacuum_cost_delay configuration parameter, Cost-based Vacuum Delay
vacuum_cost_limit configuration parameter, Cost-based Vacuum Delay
vacuum_cost_page_dirty configuration parameter, Cost-based Vacuum Delay
vacuum_cost_page_hit configuration parameter, Cost-based Vacuum Delay
vacuum_cost_page_miss configuration parameter, Cost-based Vacuum Delay
vacuum_defer_cleanup_age configuration parameter, Primary Server
vacuum_failsafe_age configuration parameter, Statement Behavior
vacuum_freeze_min_age configuration parameter, Statement Behavior
vacuum_freeze_table_age configuration parameter, Statement Behavior
vacuum_index_cleanup storage parameter, Storage Parameters
vacuum_multixact_failsafe_age configuration parameter, Statement Behavior
vacuum_multixact_freeze_min_age configuration parameter, Statement Behavior
vacuum_multixact_freeze_table_age configuration parameter, Statement Behavior
vacuum_truncate storage parameter, Storage Parameters
value expression, Value Expressions
VALUES, VALUES Lists, UNION, CASE, and Related Constructs, VALUES
determination of result type, UNION, CASE, and Related Constructs
varchar, Character Types
variadic function, SQL Functions with Variable Numbers of Arguments
variance, Aggregate Functions
population, Aggregate Functions
sample, Aggregate Functions
var_pop, Aggregate Functions
var_samp, Aggregate Functions
version, Accessing a Database, System Information Functions and Operators, Upgrading a PostgreSQL Cluster
compatibility, Upgrading a PostgreSQL Cluster
view, Views, Views and the Rule System, Materialized Views, Cooperation with Views
implementation through rules, Views and the Rule System
materialized, Materialized Views
updating, Cooperation with Views
Visibility Map, Visibility Map
VM (see Visibility Map)
void, Pseudo-Types
VOLATILE, Function Volatility Categories
volatility, Function Volatility Categories
functions, Function Volatility Categories
VPATH, Installation Procedure, Extension Building Infrastructure

W

WAL, Reliability and the Write-Ahead Log
wal_block_size configuration parameter, Preset Options
wal_buffers configuration parameter, Settings
wal_compression configuration parameter, Settings
wal_consistency_checking configuration parameter, Developer Options
wal_debug configuration parameter, Developer Options
wal_init_zero configuration parameter, Settings
wal_keep_size configuration parameter, Sending Servers
wal_level configuration parameter, Settings
wal_log_hints configuration parameter, Settings
wal_receiver_create_temp_slot configuration parameter, Standby Servers
wal_receiver_status_interval configuration parameter, Standby Servers
wal_receiver_timeout configuration parameter, Standby Servers
wal_recycle configuration parameter, Settings
wal_retrieve_retry_interval configuration parameter, Standby Servers
wal_segment_size configuration parameter, Preset Options
wal_sender_timeout configuration parameter, Sending Servers
wal_skip_threshold configuration parameter, Settings
wal_sync_method configuration parameter, Settings
wal_writer_delay configuration parameter, Settings
wal_writer_flush_after configuration parameter, Settings
warm standby, High Availability, Load Balancing, and Replication
websearch_to_tsquery, Text Search Functions and Operators
WHERE, The WHERE Clause
where to log, Where to Log
WHILE
in PL/pgSQL, WHILE
width, Geometric Functions and Operators
width_bucket, Mathematical Functions and Operators
window function, Window Functions, Window Function Calls, Window Function Processing, Window Functions
built-in, Window Functions
invocation, Window Function Calls
order of execution, Window Function Processing
WITH, WITH Queries (Common Table Expressions), SELECT
in SELECT, WITH Queries (Common Table Expressions), SELECT
WITH CHECK OPTION, CREATE VIEW
WITHIN GROUP, Aggregate Expressions
witness server, High Availability, Load Balancing, and Replication
word_similarity, Functions and Operators
work_mem configuration parameter, Memory
wraparound
of multixact IDs, Multixacts and Wraparound
of transaction IDs, Preventing Transaction ID Wraparound Failures

X

xid, Object Identifier Types
xid8, Object Identifier Types
xmax, System Columns
xmin, System Columns
XML, XML Type
XML export, Mapping Tables to XML
XML Functions, XML Functions
XML option, Creating XML Values, Statement Behavior
xml2, xml2
xmlagg, xmlagg, Aggregate Functions
xmlbinary configuration parameter, Statement Behavior
xmlcomment, xmlcomment
xmlconcat, xmlconcat
xmlelement, xmlelement
XMLEXISTS, XMLEXISTS
xmlforest, xmlforest
xmloption configuration parameter, Statement Behavior
xmlparse, Creating XML Values
xmlpi, xmlpi
xmlroot, xmlroot
xmlserialize, Creating XML Values
xmltable, xmltable
xml_is_well_formed, xml_is_well_formed
xml_is_well_formed_content, xml_is_well_formed
xml_is_well_formed_document, xml_is_well_formed
XPath, xpath
xpath_exists, xpath_exists
xpath_table, xpath_table
XQuery regular expressions, Differences from XQuery (LIKE_REGEX)
xslt_process, xslt_process

Y

yacc, Requirements

Z

zero_damaged_pages configuration parameter, Developer Options
zlib, Requirements, Anti-Features
Prev | Up
Bibliography | Home


PostgreSQL 14.4 Documentation
Next

PostgreSQL 14.4 Documentation

The PostgreSQL Global Development Group

CHM制作者：中启乘数科技（杭州）有限公司

Copyright © 1996–2022 The PostgreSQL Global Development Group

Legal Notice

Table of Contents

Preface
1. What Is PostgreSQL?
2. A Brief History of PostgreSQL
3. Conventions
4. Further Information
5. Bug Reporting Guidelines
I. Tutorial
1. Getting Started
2. The SQL Language
3. Advanced Features
II. The SQL Language
4. SQL Syntax
5. Data Definition
6. Data Manipulation
7. Queries
8. Data Types
9. Functions and Operators
10. Type Conversion
11. Indexes
12. Full Text Search
13. Concurrency Control
14. Performance Tips
15. Parallel Query
III. Server Administration
16. Installation from Binaries
17. Installation from Source Code
18. Installation from Source Code on Windows
19. Server Setup and Operation
20. Server Configuration
21. Client Authentication
22. Database Roles
23. Managing Databases
24. Localization
25. Routine Database Maintenance Tasks
26. Backup and Restore
27. High Availability, Load Balancing, and Replication
28. Monitoring Database Activity
29. Monitoring Disk Usage
30. Reliability and the Write-Ahead Log
31. Logical Replication
32. Just-in-Time Compilation (JIT)
33. Regression Tests
IV. Client Interfaces
34. libpq — C Library
35. Large Objects
36. ECPG — Embedded SQL in C
37. The Information Schema
V. Server Programming
38. Extending SQL
39. Triggers
40. Event Triggers
41. The Rule System
42. Procedural Languages
43. PL/pgSQL — SQL Procedural Language
44. PL/Tcl — Tcl Procedural Language
45. PL/Perl — Perl Procedural Language
46. PL/Python — Python Procedural Language
47. Server Programming Interface
48. Background Worker Processes
49. Logical Decoding
50. Replication Progress Tracking
VI. Reference
I. SQL Commands
II. PostgreSQL Client Applications
III. PostgreSQL Server Applications
VII. Internals
51. Overview of PostgreSQL Internals
52. System Catalogs
53. Frontend/Backend Protocol
54. PostgreSQL Coding Conventions
55. Native Language Support
56. Writing a Procedural Language Handler
57. Writing a Foreign Data Wrapper
58. Writing a Table Sampling Method
59. Writing a Custom Scan Provider
60. Genetic Query Optimizer
61. Table Access Method Interface Definition
62. Index Access Method Interface Definition
63. Generic WAL Records
64. B-Tree Indexes
65. GiST Indexes
66. SP-GiST Indexes
67. GIN Indexes
68. BRIN Indexes
69. Hash Indexes
70. Database Physical Storage
71. System Catalog Declarations and Initial Contents
72. How the Planner Uses Statistics
73. Backup Manifest Format
VIII. Appendixes
A. PostgreSQL Error Codes
B. Date/Time Support
C. SQL Key Words
D. SQL Conformance
E. Release Notes
F. Additional Supplied Modules
G. Additional Supplied Programs
H. External Projects
I. The Source Code Repository
J. Documentation
K. PostgreSQL Limits
L. Acronyms
M. Glossary
N. Color Support
O. Obsolete or Renamed Features
Bibliography
Index
Next
Preface


Legal Notice

PostgreSQL is Copyright © 1996–2022 by the PostgreSQL Global Development Group.

Postgres95 is Copyright © 1994–5 by the Regents of the University of California.

Permission to use, copy, modify, and distribute this software and its documentation for any purpose, without fee, and without a written agreement is hereby granted, provided that the above copyright notice and this paragraph and the following two paragraphs appear in all copies.

IN NO EVENT SHALL THE UNIVERSITY OF CALIFORNIA BE LIABLE TO ANY PARTY FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOST PROFITS, ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS DOCUMENTATION, EVEN IF THE UNIVERSITY OF CALIFORNIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

THE UNIVERSITY OF CALIFORNIA SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE SOFTWARE PROVIDED HEREUNDER IS ON AN “AS-IS” BASIS, AND THE UNIVERSITY OF CALIFORNIA HAS NO OBLIGATIONS TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.