Skip to content

Data export

Overview

Data Export is a feature that extracts the event data being loaded into Hive Analytics every hour and uploads it to cloud storage.

The data that Data Export provides is raw event data. It is provided so that you can build your own database, or process the data into the form you want and use it for analysis according to your purpose.

Hive Analytics provides the file conversion and transfer of the data, but you must register the cloud storage with the service of the cloud you use.

Note

Because data is provided per event, split transfer by project is not supported.

What can you do?

Data analysts

  • Load the data collected by Hive Analytics directly into your company's DB to process it into the form you want and analyze it in depth.
  • Connect the raw data to your own BI tools or analysis environment to build custom dashboards.

Developers

  • Automatically load Hive event data into your own data pipeline.
  • Integrate with AWS S3 or GCP Cloud Storage to use the latest event data every hour directly from cloud storage.

Quick start

If you are setting up data export for the first time, follow these steps to complete the cloud storage integration.

  1. Create a bucket dedicated to data export in the cloud storage you will use (AWS S3 or GCP Cloud Storage) and prepare an authentication key.
  2. Go to the Analytics console > Data > Data Export settings page.
  3. Select the events (logs) to export. (Up to 30)
  4. Select the storage (AWS S3 / GCP Cloud Storage) and enter the bucket name.
  5. Select the data type (CSV / JSON).
  6. Register the authentication key.

Note

For how to create a bucket and issue an authentication key for each cloud storage, see Full features.

Full features

Data export logic

data_export_01.png

Event data stored in BigQuery is converted into files every hour according to the data export cycle and uploaded to the registered cloud storage.

Data criteria

  • The selected event data is queried and transferred as files to cloud storage.
  • Data is extracted every hour according to the transfer cycle, based on UTC.
    • Example: At 01:00 (UTC) on September 1, 2023, the data from 00:00:00 to 00:59:59 (UTC) on September 1, 2023 is extracted and transferred
    • The partitioning criterion for the dateTime attribute is set to the query date -1 day.
    • Example: When the data from 00:00:00 to 00:59:59 (UTC) on September 1, 2023 is extracted, 00:00:00 on August 30, 2023
    • If the datetime value is earlier than the query time -1 day, the data is not included in the export data.
  • Data is queried by the time it was entered into BigQuery.
    • Based on the bigqueryRegistTimestamp attribute
    • Sample query for data extraction
SELECT *
FROM bigquery_table
WHERE bigqueryRegistTimestamp BETWEEN '2023-09-01 00:00:00' and '2023-09-01 00:59:59'
and dateTime >= '2023-08-31 00:00:00'

Configure data export

data_export_02.png

Select events

Select the events (logs) to export.

  • You can search for and select events by entering part of the event name.
  • You can select up to 30 events.

Select storage

You must use cloud storage as the storage for saving data.

Supported clouds:

  • AWS S3
  • GCP Google Cloud Storage

Location (bucket name)

Enter the bucket name of the storage.

  • If the AWS S3 bucket name is s3://s3_bucket_name → enter only s3_bucket_name
  • If the Google Cloud Storage bucket name is gs://google_bucket_name → enter only google_bucket_name

Data type

Two data types are provided.

  • CSV
  • JSON
  • All files are encoded in UTF-8.

File upload cycle

Every hour, data for a one-hour range is extracted and uploaded.

  • The time range is extracted based on the bigqueryRegistTimestamp attribute value. (UTC)
    • Example: Data extraction and upload start at 15:00 (UTC): the data from 05:00:00 to 05:59:59 in the bigqueryRegistTimestamp attribute is extracted.
  • The completion time may vary depending on the number of files and the upload size.

Register authentication keys

Permission is required to upload data to cloud storage. You must register an authentication key or authentication key file that has permission to save data. How you register the authentication key differs by cloud service.

  • S3 — Register the ACCESS_KEY and ACCESS_SECRET_KEY values data_export_03.png
  • GCS — Register the authentication key file data_export_04.png

Configure cloud storage

GCP - Google Cloud Storage

The following settings are required to export data to Google Cloud.

  1. On the Google Cloud console page, go to Cloud Storage.

    data_export_05.png

  2. Create a bucket to use exclusively for data export.

    • Once set, the bucket name cannot be changed. If necessary, you must delete the existing bucket and create a new one.
    • We recommend creating it as a bucket dedicated to data export.

  3. You must create a service key to provide to data export and grant it write permission for the bucket.

    1. On the console page, go to IAM & Admin → the Service accounts menu.
    2. Click Create service account to create a new account.
      • You can create the ID used for the account with any name you want. (Example: hive_data_transfer_account@projectId.iam.gserviceaccount.com) data_export_06.png
      • After you create the account, go to the Keys tab and create a key for the service.
      • Create a JSON key file with Add key → Create new key.
      • Download the created key file and keep it safe.
    3. Go back to Cloud Storage and go to the Permissions tab of the bucket you created. data_export_07.png
      • On the Permissions tab, go to Grant access → Add principals and enter the newly created service account ID.
      • In Assign roles, add the two Cloud Storage → Storage Object Creator, Storage Object Viewer permissions, and then click OK.

  4. After all settings are complete, register the service key file on the data export settings page of Hive Analytics.

AWS - S3

The following settings are required to export data to AWS.

  1. On the AWS console page, go to Storage → S3. data_export_08.png
  2. Create a bucket dedicated to data export.
    • Once set, the bucket name cannot be changed. If necessary, you must delete the existing bucket and create a new one.
    • We recommend using it only as a bucket dedicated to data export.
  3. You must create an account for data export.
    • This user must be used only as an account dedicated to data export. Create a new IAM user.
  4. Create an access key for the created account. For related information, see Managing Access Keys for IAM Users - Creating Access Keys.
    • Keep the access key in a safe place.
  5. Add an inline policy to the created account.
    • Create the policy by referring to the "To embed an inline policy for a user group (console)" section.
    • Select the JSON tab to create the policy, and paste the following JSON code.
    • For YOUR-BUCKET-NAME-HERE, enter the name of the bucket you created.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetBucketLocation", "s3:ListBucket"],
      "Resource": ["arn:aws:s3:::YOUR-BUCKET-NAME-HERE"]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": ["arn:aws:s3:::YOUR-BUCKET-NAME-HERE/*"]
    }
  ]
}
  1. After you complete all tasks, add the stored access key to the [Analytics console > Data > Data Export] settings.

File storage format

Data storage directory structure

General file path format:

withhive/data_export/build_type/YYYY/MM/DD/event_name/event_name_YYYY_MM_DD_UUID.file_extension
  • Build type: Has two values, sandbox and live. If you configure it in sandbox, the data is saved under sandbox.
  • YYYY/MM/DD: The reference year/month/day of the extracted data. (UTC)
  • UUID: A random value that prevents overwriting caused by duplicate file names.
  • File extension: Depends on the selected file type.
File type Compressed Final file name
json V withhive/data_export/build_type/YYYY/MM/DD/event_name/event_name_YYYY_MM_DD_UUID.json.gzip
csv V withhive/data_export/build_type/YYYY/MM/DD/event_name/event_name_YYYY_MM_DD_UUID.csv.gzip

File extensions

  • csv.gzip: A file that consists of data with fields separated by commas ( , ). Encryption cannot be set when the file is compressed (not supported).
  • json.gzip: A file that consists of data text structured in JavaScript object syntax. It is separated line by line, and it is a JSON file compressed with gzip. Encryption cannot be set when the file is compressed (not supported).

Notes & tips

  • No retroactive transfer of past data: Data export works from the time you register it. Past data collected before registration is not sent retroactively.
  • Maximum number of events: You can select up to 30 events.
  • Limit on the total size of extracted data per event: If the extracted data exceeds 500 MB, it is excluded from the transfer. The data actually transferred is a file compressed to about 15%.
  • Event — Define the events and attributes to export