Deduplicate mode

Specify a field in an incoming Events and only emit the Event if that field contains a unique value.

Features 

  • Define how many historical events should be used to determine if value in field is unique.

  • Specify a period of time to check for duplicates.

  • Specify the field in an incoming Event to use as a measure of uniqueness.

  • If an Event is deemed unique, it will be emitted. If it is deemed a duplicate, it will be ignored and a corresponding log entry will be created.

Configuration Options 

  • mode: 'deduplicate'

  • path: a JSON wrapped path the value of which should be used to determine the uniqueness of the event.

  • lookback: (Optional: must use either lookback or period) Number of past events (maximum of 1000) to examine for uniqueness. Use a value of 0 to emit Event regardless of uniqueness.

  • period: (Optional: must use either lookback or period) Number of seconds to use for deduplication. When an event is received, subsequent events will not be emitted until this period has elapsed.

  • emit_duplicate: (Optional) Set to true if duplicate events should be emitted. 'unique_event': false will be included in emitted event.

Emitted Events 

{
  "unique_event": true
}

Example Configuration Options 

Check the last 100 received Events and only emit an Event if the value in the {{.person.name}} field is unique.

{
  "mode": "deduplicate",
  "lookback": "100",
  "path": "{{.person.name}}"
}