BigQuery - adapter.get_relation() not finding a table that exists

jaboydev · August 14, 2025, 2:57am

The problem I’m having

BigQuery adapter.get_relation() not finding a table. When checking the manifest the project and data locations all match expected, and there are no casing issues.

The context of why I’m trying to do this

I’m unable to add a comment to the existing thread, but I figured I’d post here just incase anyone else comes across this problem.

The issue/solution:

The issue was happening in some environments for us, but we were unable to reproduce it at will. After significant digging, we noticed this line in the big-query adapter. We were not cleaning up our test runs, and in an enterprise setting across many many pipeline executions we had >100k tables in our elementary schema and in that situation the request fails silently as the adapter assumes that all tables exist in the cache.

github.com/dbt-labs/dbt-bigquery

dbt/adapters/bigquery/impl.py

0995665e4


      
          all_tables = client.list_tables(
              dataset_ref,
              # BigQuery paginates tables by alphabetizing them, and using
              # the name of the last table on a page as the key for the
              # next page. If that key table gets dropped before we run
              # list_relations, then this will 404. So, we avoid this
              # situation by making the page size sufficiently large.
              # see: https://github.com/dbt-labs/dbt/issues/726
              # TODO: cache the list of relations up front, and then we
              #       won't need to do this
              max_results=100000,
          )
          
          # This will 404 if the dataset does not exist. This behavior mirrors
          # the implementation of list_relations for other adapters
          try:
              return [self._bq_table_to_relation(table) for table in all_tables]  # type: ignore[misc]
          except google.api_core.exceptions.NotFound:
              return []
          except google.api_core.exceptions.Forbidden as exc:
              logger.debug("list_relations_without_caching error: {}".format(str(exc)))

Hope this helps someone, it took us way to long to hunt down.

Topic		Replies	Views
adapter.get_relation Not Working with BigQuery Help bigquery , dbt-cloud	3	574	July 10, 2024
Dbt test error: BigQueryRelation has invalid value {'database': Undefined, 'schema': Undefined, 'identifier': Undefined} Archive	0	2136	September 8, 2021
adapter.get_columns_in_relation() returning a list of empty strings Help	0	342	July 10, 2024
adapter.get_relation compiling strangely Help	2	3555	December 28, 2023
dbt fails to create schema in bigquery after 1.5 upgrade Help bigquery , dbt-core	2	1528	May 10, 2023

BigQuery - adapter.get_relation() not finding a table that exists

The problem I’m having

The context of why I’m trying to do this

The issue/solution:

Related topics