How to send log batches

Direct file upload to Google Cloud Storage¶

• To upload.log batch files to GCS, you must first apply for BigQuery permissions.

GCS bucket list¶

(Mandatory) File naming convention¶

• CompanyName_CategoryValue/YYYY/MM/dd/CompanyName_CategoryValue_YYYY_MM_dd_HH_mm_ss_uuidRandom.log
  • It is mandatory to follow the file naming convention.
  • The date and uuid in the file name are for checking the storage date and preventing duplication.
• The file extension is ".log".
• Example)
  • com2us_nbasuperstars_team/2023/03/24/com2us_nbasuperstars_team_2023-03-24_05_55_00_c4c9c6e0-4093-4e29-853f-2d0d9608ca3b.log

File format¶

• Files should be composed of logs for each table and data should not be mixed with different categories.
• Newline-separated json file (sample)

{"dateTime":"2021-10-11 11:27:15", "category":"game_name_login_log", "timezone":"UTC+09:00", "channel":"HIVE", "user_id":10053833008, "app_id":"com.com2us.hivesdk.android.google.global.normal", "did":5102491016, "level":0, "client_ip":"172.XX.XXX.XX", "device_name":"TINNO U319AA", "company":"C2S", "server_id":"GLOBAL-2", "server_uid":256489411, "last_login_date":"2021-10-10 08:33:11", "os_version":"11", "is_emulator":0, "game_language":"en"}
{"dateTime":"2021-10-11 11:27:15", "category":"game_name_login_log", "timezone":"UTC+09:00", "channel":"HIVE", "user_id":10053833008, "app_id":"com.com2us.hivesdk.android.google.global.normal", "did":5102491016, "level":0, "client_ip":"172.XX.XXX.XX", "device_name":"TINNO U319AA", "company":"C2S", "server_id":"GLOBAL-2", "server_uid":256489411, "last_login_date":"2021-10-10 08:33:11", "os_version":"11", "is_emulator":0, "game_language":"en"}
{"dateTime":"2021-10-11 11:27:15", "category":"game_name_login_log", "timezone":"UTC+09:00", "channel":"HIVE", "user_id":10053833008, "app_id":"com.com2us.hivesdk.android.google.global.normal", "did":5102491016, "level":0, "client_ip":"172.XX.XXX.XX", "device_name":"TINNO U319AA", "company":"C2S", "server_id":"GLOBAL-2", "server_uid":256489411, "last_login_date":"2021-10-10 08:33:11", "os_version":"11", "is_emulator":0, "game_language":"en"}

• Files should not be compressed.
• File size should be controlled to not exceed 1G.
• When uploading multiple files, it’s better to have fewer files with larger sizes.
• Uploading too many files at once can cause errors (less than 10 per second).

GCS file upload guide¶

• https://cloud.google.com/storage/docs/uploading-objects?hl=ko#storage-upload-object-python
• Immediately after uploading the file, refreshing may make it appear as if the file has disappeared.
  • This is because the file is moved immediately after processing, and re-uploading the same file can cause data to be duplicated in BigQuery, so please take note.

Log transmission via FluentD¶

For detailed information about using FluentD, refer to fluentd method page.

Fluentd transmission precautions¶

• Chunk file - limit is 10Mbyte. Files exceeding this limit are automatically deleted, so take care.
• If line-by-line transmission (less than 10Mbyte) is not possible, it is recommended to use the google cloud storage file upload.

Dedicated server information for Log batch¶

Fluentd tag rule¶

• Tag creation rule
  • CompanyName_Table Name(category value)_snapshot
• Example
  • Use the tag created by combining the company name and category value, including_snapshot
    • com2us_logname_snapshot

References¶

• Regardless of the method of transmission, log batch files are stored in GCS for a certain period.
  • Retention period: 90 days
  • Files older than 90 days are deleted and cannot be recovered. (Data in BigQuery is still queryable)
• Files are kept for purposes such as reloading data into BigQuery later or verifying data omissions.