SORACOM Harvest Data Download format change notice
When using SORACOM Harvest to capture and store data from your devices, you can easily view and download the data directly from the User Console. This update introduces several changes when downloading data in order to improve precision and support additional features.
This update will take effect on September 21.
Note that these changes apply only to the following options in the Soracom User Console:
- Downloading data using the "Download content as JSON" option
- Downloading data using the "Download chart data as JSON" option
- Downloading data using the "Download chart data as CSV" option
This update does not affect the response format returned when retrieving data using the Harvest Data APIs.
Date and time format
Currently, the date and time that corresponds to each data entry is output using the format Sep, 16, 2021 13:45, with minute precision.
With this update, the date and time format will be changed to the ISO 8601 format, and will include both millisecond precision and time zone information, in order to provide more precision and to be easier for various applications to process. The new output will use the format 2021-09-16T13:45:59.123-08:00.
An exception to this is the "Download chart data as CSV" option, where the original date and time format will also be included in order to maintain compatibility with spreadsheet applications:
- The
__timecolumn will follow the original date and time format, and will be updated to include second precision using the formatSep, 16, 2021 13:45:59. - A
__iso8601Timecolumn will be added to the CSV and will use the ISO 8601 format of2021-09-16T13:45:59.123-08:00.
Simple Location Tracking data
Simple Location Tracking (in Japanese) is an optional service for plan-KM1 SIMs that determines the approximate geographic location of a device based on its cellular connection, without the device needing its own GPS or Wi-Fi positioning functionality.
While the longitude and latitude data provided by Simple Location Tracking is automatically combined with other data uploaded by a device to Harvest Data (as well as other Soracom services), this location data was not included when using the "Downloading content as JSON" option.
With this update, the location data follow the same format as the Harvest Data API response format (in Japanese), and will be included as a metadata property for each data entry.
Remove contentType property
With this update, the contentType property previously included when downloading data using the "Download chart data as JSON" and "Download chart data as CSV" options will no longer be included.
Data format samples
When downloading data using the "Download content as JSON" option, the new format will look like this:
[
{
"resourceType":"Subscriber",
"resourceId":"295050012345678",
"time":"2021-09-16T13:45:59.123-08:00",
"contentType":"application/json",
"content":{
"clickType":2,
"clickTypeName":"DOUBLE",
"batteryLevel":1,
"binaryParserEnabled":true
//...other data from device
},
"metadata":{
"locationQueryResult":"success",
"location":{
"lat": "47.6062",
"lon": "-122.3321"
}
}
},
// ...other data entries
]When downloading data using the "Download chart data as JSON" option, the new format will look like this:
[
{
"resourceType":"Subscriber",
"resourceId":"295050012345678",
"time":"2021-09-16T13:45:59.123-08:00",
"chartData":{
"clickType":2,
"batteryLevel":1,
"$metadata.location.lat":"47.6062",
"$metadata.location.lon":"-122.3321"
//...other data from device
}
},
// ...other data entries
]When downloading data using the "Download chart data as CSV" option, the new format will look like this:
__resourceType,__resourceId,__time,__iso8601Time,$metadata.location.lat,$metadata.location.lon,batteryLevel,clickType "Subscriber","295050012345678","Sep, 16, 2021 13:45:59","2021-09-16T13:45:59.123-08:00","47.6062","-122.3321","1","2" ...
