How to send Log Batches
Note
The.log batches can be transmitted in 2 ways.
Direct file upload to Google Cloud Storage¶
- To upload.log batch files to GCS, you must first apply for BigQuery permissions.
GCS bucket list¶
Operating Environment | Bucket Address | Description |
---|---|---|
Live Service | gs://snapshot_upload_live | Bucket for uploading log batch data for service operating environment |
Sandbox | gs://snapshot_upload_sandbox | Bucket for uploading log batch data for sandbox environment |
(Mandatory) File naming convention¶
- CompanyName_CategoryValue/YYYY/MM/dd/CompanyName_CategoryValue_YYYY_MM_dd_HH_mm_ss_uuidRandom.log
- It is mandatory to follow the file naming convention.
- The date and uuid in the file name are for checking the storage date and preventing duplication.
- The file extension is ".log".
- Example)
- com2us_nbasuperstars_team/2023/03/24/com2us_nbasuperstars_team_2023-03-24_05_55_00_c4c9c6e0-4093-4e29-853f-2d0d9608ca3b.log
File format¶
- Files should be composed of logs for each table and data should not be mixed with different categories.
- Newline-separated json file (sample)
{"dateTime":"2021-10-11 11:27:15", "category":"game_name_login_log", "timezone":"UTC+09:00", "channel":"HIVE", "user_id":10053833008, "app_id":"com.com2us.hivesdk.android.google.global.normal", "did":5102491016, "level":0, "client_ip":"172.XX.XXX.XX", "device_name":"TINNO U319AA", "company":"C2S", "server_id":"GLOBAL-2", "server_uid":256489411, "last_login_date":"2021-10-10 08:33:11", "os_version":"11", "is_emulator":0, "game_language":"en"}
{"dateTime":"2021-10-11 11:27:15", "category":"game_name_login_log", "timezone":"UTC+09:00", "channel":"HIVE", "user_id":10053833008, "app_id":"com.com2us.hivesdk.android.google.global.normal", "did":5102491016, "level":0, "client_ip":"172.XX.XXX.XX", "device_name":"TINNO U319AA", "company":"C2S", "server_id":"GLOBAL-2", "server_uid":256489411, "last_login_date":"2021-10-10 08:33:11", "os_version":"11", "is_emulator":0, "game_language":"en"}
{"dateTime":"2021-10-11 11:27:15", "category":"game_name_login_log", "timezone":"UTC+09:00", "channel":"HIVE", "user_id":10053833008, "app_id":"com.com2us.hivesdk.android.google.global.normal", "did":5102491016, "level":0, "client_ip":"172.XX.XXX.XX", "device_name":"TINNO U319AA", "company":"C2S", "server_id":"GLOBAL-2", "server_uid":256489411, "last_login_date":"2021-10-10 08:33:11", "os_version":"11", "is_emulator":0, "game_language":"en"}
- Files should not be compressed.
- File size should be controlled to not exceed 1G.
- When uploading multiple files, it’s better to have fewer files with larger sizes.
- Uploading too many files at once can cause errors (less than 10 per second).
GCS file upload guide¶
- https://cloud.google.com/storage/docs/uploading-objects?hl=ko#storage-upload-object-python
- Immediately after uploading the file, refreshing may make it appear as if the file has disappeared.
- This is because the file is moved immediately after processing, and re-uploading the same file can cause data to be duplicated in BigQuery, so please take note.
Log transmission via FluentD¶
Note
Log batch cannot be transferred to the server receiving via streaming FluentD.
For detailed information about using FluentD, refer to fluentd method page.
Fluentd transmission precautions¶
- Chunk file - limit is 10Mbyte. Files exceeding this limit are automatically deleted, so take care.
- If line-by-line transmission (less than 10Mbyte) is not possible, it is recommended to use the google cloud storage file upload.
Dedicated server information for Log batch¶
Server Type | URL Address |
---|---|
Live Server 01 | analytics-hivelog-snap-01.withhive.com |
Live Server 02 | analytics-hivelog-snap-02.withhive.com |
Sandbox Server | sandbox-analytics-hivelog.withhive.com |
Fluentd tag rule¶
- Tag creation rule
- CompanyName_Table Name(category value)_snapshot
- Example
- Use the tag created by combining the company name and category value, including_snapshot
- com2us_logname_snapshot
- Use the tag created by combining the company name and category value, including_snapshot
References¶
- Regardless of the method of transmission, log batch files are stored in GCS for a certain period.
- Retention period: 90 days
- Files older than 90 days are deleted and cannot be recovered. (Data in BigQuery is still queryable)
- Files are kept for purposes such as reloading data into BigQuery later or verifying data omissions.