
Querying ISK from Databricks serverless through OpenSharing
Databricks serverless has historically been unable to query ISK tables due to restrictions on custom storage endpoints and Spark configuration. OpenSharing provides a new way to expose ISK data through Unity Catalog, enabling direct SQL queries without changing the underlying storage format.
November 19, 2026
Background
ISK exposes each Kafka topic as an Iceberg table through two endpoints: an Iceberg REST catalog for table metadata, and an S3-compatible endpoint that serves the Parquet data files. A client needs both, the catalog URI and the S3 endpoint URL with credentials.
Databricks offers these ways to read Iceberg tables:
- Classic compute with custom Spark configuration. Add the Iceberg Spark runtime, set the REST catalog URI and point fs.s3a.endpoint at ISK. This is how ISK has been used from Databricks so far.
- SQL warehouses and serverless compute. Custom Spark configuration and JARs are not allowed, so the setup above cannot be applied.
- Unity Catalog external locations. Storage must be S3 or Cloudflare R2. There is no setting for a custom S3 endpoint.
- Catalog federation (foreign Iceberg tables). Supported sources are Hive metastore, AWS Glue, Snowflake, Salesforce and a few other named systems. There is no generic Iceberg REST catalog source, and the table data has to sit in a Unity Catalog external location.
Only the first accepts the ISK combination, as a result, ISK tables do not appear in Unity Catalog today and can not be queried from SQL warehouses or serverless compute.
OpenSharing
Databricks released OpenSharing in June 2026 as the successor to Delta Sharing announcement. It's an open protocol that enables secure, direct access to shared data across platforms. Most importantly, Unity Catalog can mount a share from any server that implements the protocol, and its tables can be queried from SQL warehouses and serverless compute docs. The recipient installs the share from a credential file that holds the server URL and a bearer token.
The protocol is a small REST API. The server lists shares, schemas and tables, returns a table's schema, and for a query returns the table's data files as HTTPS URLs. The reader downloads the files from those URLs, so it needs no storage credential or endpoint setting. MinIO uses it for AIStor Tables, its built-in Iceberg tables: AIStor translates the Iceberg metadata into OpenSharing responses at request time, the same approach ISK takes below MinIO.
How it hangs together
ISK gets an OpenSharing endpoint next to its Iceberg REST catalog. The data files are served by the existing file endpoint.

A query runs in three steps:
- Unity Catalog requests the table's schema and file list from the sharing endpoint. The query's filters are passed along as hints.
- ISK returns the schema and one entry per data file: an expiring signed HTTPS URL, the file size, partition values, row count and column statistics.
- The warehouse downloads each Parquet file over HTTPS.
Although OpenSharing supports both Iceberg and Delta formats, Databricks currently only supports Delta through OpenSharing. This means ISK must translate its Iceberg metadata into Delta-compatible responses for Databricks to query the data. The sharing endpoint builds each response from the topic's Iceberg metadata: the Iceberg schema is converted to a Delta schema, and Iceberg data files are already Parquet files.
Unity Catalog requests the file list on every query and does not cache it, so a query via ISK gets fully up to date data from the Kafka topic.
CDC topics
ISK serves a CDC topic as data files plus Iceberg positional deletes (a list per data file of deleted row positions). The sharing protocol carries deletes as Delta deletion vectors (a compressed bitmap of row positions per data file). Since internally ISK already holds these positions as bitmaps it is trivial to convert them to deletion vectors for Open Sharing instead of positional delete files.
With each request Databricks sends the response formats and reader features it supports. For tables with deletes ISK responds in Delta format with the deletion-vector feature; other tables can use the Parquet format.
Prototype
We built a minimal OpenSharing server in about a day and served test tables with the same layout as ISK topics. Using a serverless SQL warehouse, we installed the share through the Unity Catalog API, created a catalog from it and queried each table with SQL.
| Check | Result |
|---|---|
| Query from a serverless SQL warehouse | Correct row counts and types, including nested structs and timestamps |
| Partition column also stored in the data files, as ISK writes them | Read correctly |
| Table with deletion vectors, inline and as separate files | Deleted rows hidden; counts match |
| Data transfer | One HTTPS request per data file, from the warehouse to the server |
Limits
Current limits on the Databricks side:
- Delta format only. Databricks documents that Iceberg-only tables are not supported, and it did not call the Iceberg endpoint that OpenSharing defines. ISK has to respond in Delta format; but at least the underlying Parquet files can be used as is.
- Pruning is up to the server. Databricks sends the query's filters as hints, including filters on non-partition columns such as kafka_offset, and re-applies them to the rows it reads. It downloads every file in the response and doesn't use the file statistics to prune.
- Catalog Explorer shows tables without columns. Unity Catalog fetches the schema at query time and does not store it.
- New tables only appear after a sync of a few minutes.
- Shared volumes are not available from non-Databricks providers yet. The protocol defines them, but Databricks has not enabled them for this case.
Next steps
Remaining work to make this production-ready:
- Schema types. Some Iceberg types have no direct Delta equivalent, such as time and uuid. Timestamps without a time zone need Delta's timestampNtz reader feature.
- Pruning. File stats based pruning has to be applied on file list generation - basically like server side planning. Server has to evaluate the filter hints against each file's stats and metadata, and leave out files that cannot match from file list response.
- Statistics. Right now they seem to be unused by Databricks reader for any kind of planning optimization / file pruning - but potentially that may change in the future. It is reasonably trivial to convert from Iceberg file stats (row count, min/max etc).
- Signed URLs. Issue and verify expiring file URLs on ISK's file endpoint, and refresh them for long-running queries.
- Deletes. Convert CDC position deletes to deletion vectors.
One interesting opportunity is that ISK can decide which data files Databricks needs to read based on the query. Rather than sending every file, ISK can use partition information, statistics and potentially indexes like Bloom filters to identify only the relevant data. This could significantly reduce the amount of data transferred and improve query performance, particularly for lookups and range queries.
OpenSharing also opens the door to combining live Kafka data with historical data in a single Databricks view. ISK would serve the live data through OpenSharing, while Databricks reads the historical Iceberg or Delta tables directly. Users could then query both together as if they were a single table.
Conclusion
OpenSharing removes a key barrier to using ISK with Databricks, making it possible to query live Kafka data directly from serverless SQL warehouses without custom Spark configuration or changes to the underlying storage.
Our one-day prototype demonstrates that the approach works, including support for CDC deletes. While there's still work to make it production-ready, it opens up new possibilities for faster, more efficient queries and combining live and historical data in a single view.
Ultimately, this brings ISK closer to making Kafka data as easy to access and query as any other table in Databricks.