Rawdata

Rawdata consists of a list of observations within a queried time range.
The rawdata API allows you to retrieve data, with a high degree of flexibility.
You can transform fields to your liking, perform small calculations, and resample data.

For large data sets, the request can be delegated to a job.
This will significantly improve the chances of retrieving the data set, circumventing timeouts and potential network issues.

The rawdata api has several query params that can be passed to specify the data need.

Fields

Get a list of all rawdata fields

/resources/rawdata/keys

{
  "keys": {
    "io_exp_in1": {
      "short_desc": "exp in1",
      "long_desc": "I/Os expander input 1",
      "type": "boolean",
      "group": "IO related"
    },
    "code": {
      "short_desc": "EVC",
      "long_desc": "GPS Event code reported by 'EV' msg",
      "type": "number",
      "group": "Event identification"
    },
    "ecu_distance": {
      "short_desc": "VDist",
      "long_desc": "Vehicle dashboard odometer reported by ECUMon [m]",
      "type": "number",
      "group": "ECU monitor accessory related"
    },
    "ecu_idle_fuel": {
      "short_desc": "IFuel",
      "long_desc": "Fuel consumed in IDLE. Rep. by ECUMon [L x10]",
      "type": "number",
      "group": "ECU monitor accessory related"
    },
    "cf_rssi": {
      "short_desc": "rssi",
      "long_desc": "GSM Cell rssi (0-31)",
      "type": "number",
      "group": "Communication related"
    },
    "pid": {
      "short_desc": "Pegasus ID",
      "long_desc": "Pegasus site ID",
      "type": "number",
      "group": "Site identification"
    }
  }
}

The rawdata api allows you to select fields from the pegasus' events table.
Simply pass the fields parameter to the request with the fields you want.

To view a list of all the fields available to you, perform an empty GET request on the rawdata resource

By default all rawdata requests come with

• system_time
• event_time
• id
• vid

To select fields, simply pass them to fields parameter

/rawdata?fields=comdelay,speed,cf_rssi,hdop,dev_dist...

You may also pass a JSON array to the fields parameter, this allows for higher flexibility when doing complex transformations.

/rawdata?fields=["dev_ign","dev_idle","cf_rssi","hdop","dev_dist"]

For a list of the rawdata fields please refer to the

Master Fields List

For a more detailed list of the rawdata fields please refer to:

Pegasus API Fields

Sets

you can pass sets, which are a group of fields (ex. basic, counters, network, photo ...) as part of the fields parameter preceeded by $

/rawdata?fields=$basic,$network

Exclude

If you want to bring some parts of a set or sets, it may be better to include the whole set, and then exclude what you dont want.
In order to do this you can pass the excludes parameter.

/rawdata?fields=$basic&excludes=speed,mph

The excludes paramater also supports regular expressions, this allows you to exclude multiple fields at once:
For example, bring all device counters and exclude all ecu counters

/rawdata?fields=$counters&exclude=$ecu.*^

Instead of:

/rawdata?fields=dev_dist,dev_idle,dev_ign,dev_ospeed,dev_orpm...

Transformations

In addition to selecting fields, you can rename and create new fields.

A rawdata transformation that renames rssi, latitude, and longitude

https://api.pegasusgateway.com/rawdata?vehicles=197&duration=P1D&fields=latitude:@lat,longitude:@lon,rssi:@cf_rssi,speed,head

{
  "head": 9,
  "vid": 197,
  "event_time": "2018-10-16T22:12:20",
  "system_time": "2018-10-16T22:12:24.412584",
  "longitude": -75.49467,
  "latitude": 10.35501,
  "rssi": 28,
  "speed": 0,
  "id": 5014678489
}

Renaming Fields

To rename a field, simply pass the name you want, and the field it is renaming referenced with @ and separated by a :

renamed_field:@field

odometer:@dev_dist

You can do as many of these as you want.

/rawdata?fields=rssi:@cf_rssi,satellites:@sv,gps_quality:@hdop

Building fields

Rawdata request with a simple operator

https://api.pegasusgateway.com/rawdata?vehicles=1674&from=2018-10-05&to=2018-10-09&fields=ratio:@lat/@lon,lat,lon

{
  "system_time": "2018-10-05T00:06:40.998478",
  "ratio": -0.3205509899,
  "vid": 1674,
  "event_time": "2018-10-05T00:06:21",
  "lon": -80.16553,
  "lat": 25.69714,
  "id": 5014187981
}

Rawdata request that has more operators and manipulations

https://api.pegasusgateway.com/rawdata?vehicles=1674&from=2018-10-06&to=2018-10-07&fields=kph:@speed,mph,vehicle_battery_volts:@ecu_battery/1000,communication_delay:(@system_epoch-@event_epoch),centimeters_per_second:@mph_44.704,speed_in_knots:@mph_0.868976

{
  "system_time": "2018-10-06T15:51:02.014862",
  "vid": 1674,
  "event_time": "2018-10-06T15:50:58",
  "communication_delay": 4.0148599148,
  "mph": 7,
  "speed_in_knots": 6.082832,
  "centimeters_per_second": 312.928,
  "kph": 11.26538,
  "vehicle_battery_volts": null,
  "id": 5014260288
}

You can also use basic arithmetic operations to build new fields:

Divide the latitude by the longitude

ratio:@lat/@lon

or subtract epoch timestamps for the time the event was received by the time the event was generated.

comdelay:@system_epoch-@event_epoch

Remember, you can also use numbers in the transformations, for example to get the km reported by dev_dist:

km:@dev_dist/1000

vehicle_battery_volts:@ecu_battery/1000

centimeters_per_second:@mph*44.704

knots:@mph*0.868976

Casting Fields

You can also concatenate strings

/rawdata?fields:["custom:'text_to_prepend'%2B@field__str"]

Example:

/rawdata?fields:["google:'https://google.com/maps?q=' %2B @lat__str %2B ',' %2B @lon__str"]

Note that the + sign is URL encoded in the request as %2B, this is required in order for the parser to work.

/rawdata?vehicles=2600&duration=P1D&fields=%5B%22$basic%22,%22google:%27https://google.com/maps?q=%27%20%2B%20@latstr%20%2B%20%27,%27%20%2B%20@lonstr%22%5D&distance=km&volume=liter&time=second&speed=kph&order=vid,event_time,system_time&tz=America/New_York

If you get a validation error, make sure the castings match! How?
Simply append one of

• __str
• __int
• __float

Rawdata request that casts the speed as a string

https://api.pegasusgateway.com/rawdata?vehicles=1674&from=2018-10-05&to=2018-10-09&fields=speed_as_string:@speed__str,speed&speed=kph

{
  "system_time": "2018-10-06T16:36:54.534108",
  "vid": 1674,
  "event_time": "2018-10-06T16:36:45",
  "speed_as_string": "20.92142",
  "speed": 20.92142,
  "id": 5014261769
}

Example combining everything above

Results in the daily total hours with the engine on, distance traveled, last location, etc.

/rawdata?vehicles=1673&fields=["DATE_LAST_REPORTED...

{
  "events": [
    {
      "vid": 1673,
      "event_time": "2019-01-09T00:00:00",
      "LAST_LAT": 25.74485,
      "HOURS_IDLING": 0,
      "LAST_LON": -80.22526,
      "LOCATION": "https://google.com/maps?q=25.74485,-80.22526",
      "HOURS_OF_OPERATION": 0.6,
      "DISTANCE_TRAVELED_KM": 22.2,
      "DATE_LAST_REPORTED": 1547075815000
    },
    {
      "vid": 1673,
      "event_time": "2019-01-10T00:00:00",
      "LAST_LAT": 25.78423,
      "HOURS_IDLING": 0,
      "LAST_LON": -80.29391,
      "LOCATION": "https://google.com/maps?q=25.78423,-80.29391",
      "HOURS_OF_OPERATION": 0.4,
      "DISTANCE_TRAVELED_KM": 11.1,
      "DATE_LAST_REPORTED": 1547116807000
    }
  ]
}

Filtering

Rawdata request filtered by events with valid positions

https://api.pegasusgateway.com/rawdata?vehicles=197&from=2018-10-30T08:00:00&to=2018-10-30T23:59:59&fields=mph,valid_position&filter=valid_position

{
  "system_time": "2018-10-30T12:03:08.701770",
  "vid": 197,
  "event_time": "2018-10-30T12:03:05",
  "mph": 1,
  "valid_position": true,
  "id": 5002929550
}

Rawdata request filtered by events with valid positions and mph > 30

https://api.pegasusgateway.com/rawdata?vehicles=197&from=2018-10-30T08:00:00&to=2018-10-30T23:59:59&fields=mph,valid_position&filter=valid_position+and+mph>30

{
  "system_time": "2018-10-30T12:03:08.701770",
  "vid": 197,
  "event_time": "2018-10-30T12:03:05",
  "mph": 1,
  "valid_position": true,
  "id": 5002929550
}

Rawdata request with a filter that shows events after 8 with speeds greater than 20 and valid positions

https://api.pegasusgateway.com/rawdata?vehicles=197&from=2018-10-10T08:00:00&to=2018-10-11T23:59:59&fields=mph,valid_position,event_hour&filter=system_hour>8+and+(speed>20+and+valid_position)

{
  "system_time": "2018-10-10T09:14:59.506056",
  "vid": 197,
  "event_time": "2018-10-10T09:14:57",
  "mph": 28,
  "valid_position": true,
  "id": 5014414518,
  "event_hour": 9
}

The rawdata allows you to filter fields that meet a certain criteria.
Using the filter field on the request, you can pass complex boolean logic that meets your criteria.

You can apply boolean logic to filter the desired results, available operators:

>, < , >= , <=, ==, !=, and, or, not

Some examples:

valid_position and speed > 30

label == "spd" and mph < 50

(lat < -4 and lat > -5) and ( lon < -74 and lon > -75 )

system_hour == 7 and (system_minute > 0 and system_minute < 30)

speed > 50

A more complex example:

(event_wkday == 3) and cf_rssi < 10 and ( event_hour > 7 and event_hour < 17)

Which translates to:
get all events on wednesdays between 7AM and 5PM with an cf_rssi value less than 10 (low GSM signal)

&filter=valid_position and hdop < 3

Codes

You can further filter by looking at individual event codes generated, event codes depend on the managed configuration of the device.

Rawdata request of only event codes 4 and 47
https://api.pegasusgateway.com/rawdata?vehicles=1674&from=2018-09-30T08:00:00&to=2018-09-30T23:59:59&fields=$basic&filter=speed+%3E+10&codes=4,47

{
  "ecu_battery": null,
  "head": 7,
  "code": 47,
  "hdop": 1.16,
  "event_time": "2018-09-30T14:14:46",
  "ecu_distance": null,
  "vid": 1674,
  "lon": -80.16431,
  "sv": 7,
  "mph": 16,
  "label": "agglnchng",
  "valid_position": true,
  "type": 10,
  "lat": 25.69373,
  "system_time": "2018-09-30T14:14:49.760305",
  "speed": 16,
  "id": 5014039405,
  "device_id": 357042060068252
}

Label

GET labels
/labels

{
  ...
  "spd": {
    "edited_at": "2017-07-21T13:56:49.184650+00:00",
    "en": "Speeding:The vehicle has exceeded the speed limit continuously for more than the threshold configured",
    "created_at": "2017-07-21T13:56:49.184766+00:00",
    "label": "spd",
    "es": "Exceso de velocidad:El vehiculo ha excedido el limite de velocidad continuamente por mas del umbral configurado"
  },
  ...
}

Filtering by labels allows you ask for all the specific events across your fleet, no matter the configuration of the device. Labels are shared for every configuration, so you can ask for all the aggressive lane changes and speeding events for all your vehicles, regardless of the configurations of the devices.

For more information about labels please visit: labels

Rawdata request from group 500 with speeding and aggressive lane change events
https://api.pegasusgateway.com/rawdata?groups=500&from=2018-09-30T08:00:00&to=2018-09-30T23:59:59&fields=$basic&filter=speed>10&labels=agglnchng,spd

{
  "head": 191,
  "code": 47,
  "hdop": 0.75,
  "event_time": "2018-09-30T11:40:30",
  "type": 10,
  "vid": 1673,
  "lon": -80.20623,
  "sv": 12,
  "mph": 22,
  "label": "agglnchng",
  "valid_position": true,
  "lat": 25.82955,
  "system_time": "2018-09-30T11:40:36.382065",
  "speed": 22,
  "id": 5014035611,
  "device_id": 357042060068740
}

Types

Rawdata request for maximum speed (metric==2) in driving metric events (types=12)

https://api.pegasusgateway.com/rawdata?vehicles=1674&from=2018-10-01T00:00:00&to=2018-10-10T23:59:59&fields=metric,metric_value&filter=metric==2&order=metric&types=12&speed=mph

{
  "system_time": "2018-10-01T16:10:13.464688",
  "metric_value": 20,
  "vid": 1674,
  "metric": 2,
  "event_time": "2018-10-01T16:08:47",
  "id": 5014071812
}

Events reported by the devices are classified by types, the most common type is 10, which means it's a regular GPS event.
There are other useful types for example 12, which gives you the driving metric values of the vehicles (Max Acceleration, Deceleration and Speed per trip)
In order to filter by types simply state:

types=12

Ordering

Rawdata request ordered by code (increasing), and mph (decreasing)

https://api.pegasusgateway.com/rawdata?groups=500&fields=$basic&duration=PT4H&order=code,-mph

{
  "head": 102,
  "code": 3,
  "hdop": 1.48,
  "event_time": "2018-10-17T23:24:02",
  "type": 10,
  "vid": 1483,
  "lon": -80.44694,
  "sv": 4,
  "mph": 0,
  "label": "ignoff",
  "valid_position": true,
  "lat": 25.66764,
  "system_time": "2018-10-17T23:24:05.677751",
  "speed": 0,
  "id": 5014716545,
  "device_id": 356612026170920
}

You can also order the resulting data set however you choose.
Simply pass the order field to the request

order=mph

This will order the events by mph in ascending order. but what about ascending?

Simple just append a -, to the field you want to order

order=-mph

What about multilevel ordering? Sure thing, no problem:

order=vid,-mph

This will return all events sorted by vehicle id (vid) and then by speed in descending order

Limiting

Rawdata request that limits the results to the first two and last 3 events

https://api.pegasusgateway.com/rawdata?groups=500&fields=$basic&duration=PT4H&order=code,-mph&filter=code%3E1%20and%20code%3C10&head=2&tail=3

{
  "head": 280,
  "code": 8,
  "hdop": 0.64,
  "event_time": "2018-10-17T22:43:00",
  "type": 10,
  "vid": 909,
  "lon": -83.85399,
  "sv": 13,
  "mph": 0,
  "label": "slwtrfc",
  "valid_position": true,
  "lat": 10.25766,
  "system_time": "2018-10-17T22:43:03.888619",
  "speed": 0,
  "id": 5014715284,
  "device_id": 356612026180218
}

By default the rawdata request will return all the events that match your criteria.

What if you want to see the first x events. Simply pass the head argument to the request

head=100

Will return the top 100 events from the set.

Want to see the last n events? pass tail as the argument.

tail=100

Want both? How about the first 100 and bottom 50 events?

head=100&tail=50

You can even pass negative values to hide the first or last rows, for example, tail=-1 removes the first row, and head=-1 removes the last row.

Resampling

Returns the difference in all counters (max-min), resampled at 1 day intervals and grouped by vids from November 1 . This gives you the total distance traveled, ignition on time, etc. per day per vehicle

https://api.pegasusgateway.com/rawdata?vehicles=617,654&fields=$counters&from=2015-11-01T00:00:00&to=2015-11-10T00:00:00&filter=(valid_position+and+hdop+<+3)&how=$counters:diff&freq=1D&group_by=vid&resample=event_time

{
  "dev_dist": 72542,
  "vid": 909,
  "ecu_dist": 72421,
  "event_time": "2015-11-09T00:00:00",
  "dev_idle": 21069,
  "dev_orpm": 8151,
  "dev_ospeed": 0,
  "ecu_ifuel": 19.4,
  "ecu_tfuel": 212.4,
  "ecu_eusage": 36900,
  "ecu_eidle": 14400,
  "dev_ign": 40823
}

Returns the mean value of all comdelays and the mean value of the RSSI (signal strength) sampled at 1 day intervals, grouped by vehicle IDs

https://api.pegasusgateway.com/rawdata?vehicles=617,654&fields=$basic,comdelay,rssi_avg:@cf_rssi&from=2015-11-1T00:00:00&to=2015-11-10T00:00:00&filter=(comdelay+<+3600)&how=comdelay:mean,rssi_avg:mean&freq=1D&group_by=vid&resample=event_time

{
  "comdelay": 155.2830875629,
  "rssi_avg": 15.835443038,
  "event_time": "2015-11-01T00:00:00",
  "vid": 617
}

Resampling is an important part of data analysis, timed data can be normalized to evenly timed chunks.

Events that pass through the Pegasus Gateway naturally contain time noticermation.

These fields are:

• event_time : time reported by the device.
• system_time: time the event was processed by the gateway.

The Rawdata API allows you to resample data based on either one of these.

There are 5 fields important to resampling.

• resample : timed field to use in resampling (event_time or system_time)
• freq : Frequency you want to resample to (ex. by day, by month, every 30 seconds, etc)
• how : Method to use for resampling, defaults to mean (ex. sum, diff, min, max ...)
• fill : How to fill gaps in the data set. ffill, pad, bfill
• group_by : group events by field(s) before performing resample. (useful for multiple vehicles)

Resampling also strips the event id from the set.

resample

Used to activate resampling.
Can only be event_time or system_time.

freq

Establishes the frequency by which to resample.
Must meet the format:

• NF

where N is equal to a number greater than 0.
and F can be one of:

• M : Months
• D : Days
• H : Hours
• T : Minutes
• S : Seconds

how

Determines how to resample the data, based on the frequency established.
This must be one of.

• sum
• mean : Average of values
• std : Standard deviation of values
• sem : MISSING
• max
• min
• median
• first
• last
• diff : range of values, max-min

You can also resample different fields by different methods.
Simply pass the fields as a csv, with the transformation you want separated by :

field:method

how=dev_dist:diff,cf_rssi:mean,hdop:mean,hdop:max,vdop:min

want to apply the same resample method to a set? Simply:

how=$counters:diff

group_by

You can also group the events before performing a resample. this is useful if you want to keep events tied to a specific field, such as the vehicles id (vid).

group_by:vid

You can additionally group by more than one field:

group_by:vid,label

Output Formats

Export as HTML

https://api.pegasusgateway.com/rawdata?vehicles=197&from=2015-10-30T08:00:00&to=2015-10-30T23:59:59&fields=vid,lat,lon&export=html

By default, events come in a JSON envelope under the events property.

The rawdata API also allows you to export in other formats:

• JSON (default)
• CSV
• TSV
• HTML table

Simply pass the export parameter on the request

export=html

Need another format? Send us a message on our Gitter or create an issue on our github

Asynchronous Requests

An asynchronous requests that generates a job id to be queried later

https://api.pegasusgateway.com/rawdata?vehicles=197&from=2015-09-30T08:00:00&to=2015-09-30T23:59:59&fields=ecu_battery,ecu_distance&filter=speed>10&async=true

{
  "message": "Job created",
  "job_id": 28813
}

While the job is being processed in the background you can see the progress under /jobs/:job_id

https://api.pegasusgateway.com/jobs/28813

{
  "control": "run",
  "stdout": "",
  "app": "pegasus2.0",
  "notice": null,
  "id": 28813,
  "user_id": 1,
  "state": "queued",
  "params": {
    "all": false,
    "event_groups": null,
    "labels": null,
    "raw": false,
    "export": null
  ...
}

Once the job is finished you can see the data here

https://api.pegasusgateway.com/jobs/28813/data.json

[
  {
    "code": 1,
    "device_id": 356612022409637,
    "ecu_battery": 13350,
    "ecu_distance": 269302153,
    "event_time": "2015-09-30T14:59:06",
    "hdop": 0.94,
    "head": 225,
    "id": 5002464983,
    "label": "",
    "lat": 4.61529,
    "lon": -74.10331,
    "mph": 11,
    "speed": 11,
    "sv": 8,
    "system_time": "2015-09-30T14:59:09.565434",
    "type": 10,
    "valid_position": true,
    "vid": 197
  }
]

Results returned by the rawdata API can get big, and we mean big.
This can bring up all sorts of issues when using via a web browser.

These include but are not limited to:

• Connection issues (timeouts, disconections)
• Lack of processing power on client machine (try parsing a 500MB JSON file on chrome)

In order to circumvent these issues, and guarantee that data is always retreived as requested. You can leave the query and subsequent processes (transformations, filtering and resampling) running in the background.

The async flag delegates the rawdata request to a Job to ensure the data is stored and can be safely used and reused

Simply set the async flag on the request

async=1

The request immediately returns a job ID
you can query the job ID to see the progress of the rawdata request
/jobs/:jobid

When the job finishes you can see the by accesing the jobs data
Data is stored depending on your export format.

For JSON (default)
/jobs/:jobid/data.json

For CSV, TSV, and HTML
/jobs/:jobid/data.{export_format}