Downloading Direct Data Files

This section outlines how to send requests to Direct Data API to retrieve the desired data from your Vault. You can send individual requests to Direct Data API using the Vault Postman Collection or use the Shell script we've provided to download the latest available Direct Data file. Learn more about the structure of each Direct Data file in Direct Data File Structure.

Sending Requests to Direct Data API

You can retrieve and download Direct Data files by sending requests to Direct Data API. Before calling the API, you must enable Direct Data in your Vault. In limited release Vaults, enable this feature in your Vault by navigating to Admin > Settings > General Settings and selecting Enable Direct Data API. In general release Vaults, enable this feature by contacting Veeva Support. The ability for Admins to enable this feature will be available to general release Vaults in 25R2.

Once enabled, Vault generates the first Full file and you can successfully send requests to the following endpoints:

Authentication: Use one of the authentication endpoints to authenticate your account prior to sending requests to Direct Data API.
Retrieve Available Direct Data Files: Use this endpoint to list all Direct Data files available for download. For example, Full files and Incremental files. To filter the files returned, you can include query parameters in your request or use attributes in the response. Learn more about the returned files in Direct Data File Structure.
Download Direct Data File: After listing the files available to download, you can use this endpoint to download a specific Direct Data file. For example, you can use this endpoint to download the first Full file, and then each subsequent Incremental file.

Learn more about these endpoints in our video walkthrough and the Vault API Reference.

First Full File

When Direct Data API is enabled for the first time, Vault extracts a Full file and publishes it at 1:00 AM UTC. The time it takes to prepare the first Full file depends on when the feature was enabled and how much data is present in your Vault. Depending on what day Direct Data API was enabled in your Vault, the first Full file may not be available until the following day. Veeva can help provide an estimate for when the first Full file will be available after the feature is enabled in your Vault. Direct Data API returns a standard error message until the first Full file is generated.

Filtering Files

Direct Data API provides the following options to help filter the Direct Data files available for download:

Filter by File Type

When using the Retrieve Available Direct Data Files endpoint, the extract_type query parameter allows you to filter based on file type. For example, you can filter to retrieve only Incremental, Full, or Log files. This is helpful for integration processes which only need certain file types.

Filter by Time

When using the Retrieve Available Direct Data Files endpoint, you can use the start_time and stop_time query parameters to filter on a specific window of time for which data was captured in the Direct Data file. For example, you may want to retrieve all Incremental files generated within an hour timeframe, from 2025-01-15T07:00:00Z to 2025-01-15T08:00:00Z.

All Full files have a start time of 00:00 Jan 1, 2000. To include Full files in your results when filtering by time, you must use 00:00 Jan 1, 2000 or earlier as your start_time. Otherwise, Direct Data API only returns Incremental files for the specified window of time.

Filter out Files with No Changes

When using the Retrieve Available Direct Data Files endpoint, each returned file includes a record_count attribute that can help locate empty files. This attribute provides the total number of records for all extracts in the file. If this count is zero (0), it means no data changes were captured in the time interval for which the file was produced. Therefore, there is no need to send a request to Direct Data API to download a file with record_count=0.

As an integration best practice, we recommend designing your integration to check for files with record_count=0 and, if true, skip the API call to retrieve said files. This results in faster integrations because it eliminates the need to spend time processing empty files.

Requesting Files at Fixed Times

Direct Data API publishes Direct Data files at fixed times in a day. For example, for a data extraction window of 13:00 to 13:15 UTC, the corresponding Incremental file will be published at 13:30 UTC. This means an integration can check and request the file at 13:30 UTC or after. This reduces the number of requests to the API to check if a file is available and makes your code simpler to manage.

Retry Date for Failures

Ensuring the reliable delivery of Direct Data files is paramount. If Direct Data API fails to publish a file for a specific time interval, the response includes an error object containing the file’s name, the next_retry time in UTC, and a standard message. If Direct Data API successfully publishes the file at the next_retry time, the response then includes the metadata for that file, thus replacing the error object.