Skip to content

How to send Log Batches

Note

The.log batches can be transmitted in 2 ways.

Direct file upload to Google Cloud Storage

GCS bucket list

Operating Environment
Bucket Address
Description
Live Service gs://snapshot_upload_live Bucket for uploading log batch data for service operating environment
Sandbox gs://snapshot_upload_sandbox Bucket for uploading log batch data for sandbox environment

(Mandatory) File naming convention

  • CompanyName_CategoryValue/YYYY/MM/dd/CompanyName_CategoryValue_YYYY_MM_dd_HH_mm_ss_uuidRandom.log
    • It is mandatory to follow the file naming convention.
    • The date and uuid in the file name are for checking the storage date and preventing duplication.
  • The file extension is ".log".
  • Example)
    • com2us_nbasuperstars_team/2023/03/24/com2us_nbasuperstars_team_2023-03-24_05_55_00_c4c9c6e0-4093-4e29-853f-2d0d9608ca3b.log

File format

  • Files should be composed of logs for each table and data should not be mixed with different categories.
  • Newline-separated json file (sample)
{"dateTime":"2021-10-11 11:27:15", "category":"game_name_login_log", "timezone":"UTC+09:00", "channel":"HIVE", "user_id":10053833008, "app_id":"com.com2us.hivesdk.android.google.global.normal", "did":5102491016, "level":0, "client_ip":"172.XX.XXX.XX", "device_name":"TINNO U319AA", "company":"C2S", "server_id":"GLOBAL-2", "server_uid":256489411, "last_login_date":"2021-10-10 08:33:11", "os_version":"11", "is_emulator":0, "game_language":"en"}
{"dateTime":"2021-10-11 11:27:15", "category":"game_name_login_log", "timezone":"UTC+09:00", "channel":"HIVE", "user_id":10053833008, "app_id":"com.com2us.hivesdk.android.google.global.normal", "did":5102491016, "level":0, "client_ip":"172.XX.XXX.XX", "device_name":"TINNO U319AA", "company":"C2S", "server_id":"GLOBAL-2", "server_uid":256489411, "last_login_date":"2021-10-10 08:33:11", "os_version":"11", "is_emulator":0, "game_language":"en"}
{"dateTime":"2021-10-11 11:27:15", "category":"game_name_login_log", "timezone":"UTC+09:00", "channel":"HIVE", "user_id":10053833008, "app_id":"com.com2us.hivesdk.android.google.global.normal", "did":5102491016, "level":0, "client_ip":"172.XX.XXX.XX", "device_name":"TINNO U319AA", "company":"C2S", "server_id":"GLOBAL-2", "server_uid":256489411, "last_login_date":"2021-10-10 08:33:11", "os_version":"11", "is_emulator":0, "game_language":"en"}
  • Files should not be compressed.
  • File size should be controlled to not exceed 1G.
  • When uploading multiple files, it’s better to have fewer files with larger sizes.
  • Uploading too many files at once can cause errors (less than 10 per second).

GCS file upload guide

Log transmission via FluentD

Note

Log batch cannot be transferred to the server receiving via streaming FluentD.

For detailed information about using FluentD, refer to fluentd method page.

Fluentd transmission precautions

  • Chunk file - limit is 10Mbyte. Files exceeding this limit are automatically deleted, so take care.
  • If line-by-line transmission (less than 10Mbyte) is not possible, it is recommended to use the google cloud storage file upload.

Dedicated server information for Log batch

Server Type URL Address
Live Server 01 analytics-hivelog-snap-01.withhive.com
Live Server 02 analytics-hivelog-snap-02.withhive.com
Sandbox Server sandbox-analytics-hivelog.withhive.com


Fluentd tag rule

  • Tag creation rule
    • CompanyName_Table Name(category value)_snapshot
  • Example
    • Use the tag created by combining the company name and category value, including_snapshot
      • com2us_logname_snapshot

References

  • Regardless of the method of transmission, log batch files are stored in GCS for a certain period.
    • Retention period: 90 days
    • Files older than 90 days are deleted and cannot be recovered. (Data in BigQuery is still queryable)
  • Files are kept for purposes such as reloading data into BigQuery later or verifying data omissions.