What happens when you press Run

BigQuery is a serverless data warehouse. You do not choose machine sizes or manage indexes. You write SQL, and Google's infrastructure spreads the work across many workers. Understanding a little of how it does that explains almost every cost and performance tip in this course.

Storage is columnar

Tables are stored by column, compressed, in a format called Capacitor. When a query references three columns of a fifty column table, BigQuery reads only those three. This is the first and most important cost lever: the columns you select decide most of what you pay.

There are no indexes in the usual sense

Instead of B-tree indexes, BigQuery reduces work with partitioning (splitting a table, usually by date) and clustering (sorting data within partitions by up to four columns). Queries that filter on these columns can skip large parts of the table. We cover both in section two.

How you pay

There are two main models.

• On demand: you pay per amount of data processed by each query. The first terabyte processed each month is free, which is generous for learning.
• Capacity (editions): your organisation buys compute capacity measured in slots, and queries share it. Cost depends on the capacity reserved rather than bytes scanned.

Many organisations use on demand pricing for analysts, and that is the model we optimise for here. Even under capacity pricing, scanning less data makes queries faster and leaves more capacity for colleagues, so the same habits pay off.

The number to watch

Before you run a query, the editor shows an estimate such as "This query will process 3.2 GB when run". After it runs, the job information shows bytes processed and bytes billed. Get into the habit of looking at that number every time. By the end of this section you will be able to predict it before the editor tells you.

Getting set up

Open the BigQuery console, create a project and use the sandbox if you do not want to add billing. Public datasets live in the bigquery-public-data project, and we use several of them throughout the course.