Deduplicate mode

Specify a field in an incoming Events and only emit the Event if that field contains a unique value.

Features 

  • Define how many historical events should be used to determine if value in field is unique.

  • Specify a period of time to check for duplicates.

  • Specify the field in an incoming Event to use as a measure of uniqueness.

  • If an Event is deemed unique, it will be emitted. If it is deemed a duplicate, it will be ignored and a corresponding log entry will be created.

Configuration Options 

  • mode: 'deduplicate'

  • path: a JSON wrapped path the value of which should be used to determine the uniqueness of the event.

  • lookback: (Optional: must use either lookback or period) Number of past events (maximum of 1000) to examine for uniqueness.

  • period: (Optional: must use either lookback or period) Number of seconds to use for deduplication. When an event is received, subsequent events will not be emitted until this period has elapsed.

  • emit_duplicates: (Optional) Set to true if duplicate events should be emitted. 'unique_event': false will be included in emitted event.

  • tolerance: (Optional) The number of events with the same value to allow before deduplicating. Events of the same value will be treated as unique until the provided tolerance is reached.

Emitted Events 

{
  "unique_event": true
}

Example Configuration Options 

Check the last 100 received Events and only emit an Event if the value in the person.name field is unique.

{
  "mode": "deduplicate",
  "lookback": "100",
  "path": "<<person.name>>"
}
Was this helpful?