NSA App Waterslide seems to be useful for anons.
WaterSlide is an event-at-a-time architecture for processing metadata. It is
designed to take in a set of streaming events from multiple sources,
process them through a set of modules ("kids"), and return meaningful outputs.
The user specifies a directed processing graph ("pipeline") of kids used to
process data, both raw content (e.g., files, binary structures) and metadata
about content. Kids can be filters, aggregators, annotators, decoders,
translators, and collectors. Code is only executed when data is made available
to a processing operation.
WaterSlide can be used as a streaming MapReduce framework for complex event
processing. It is designed to efficiently process data by minimizing copies,
grouping data, and reusing memory. It contains specially designed data
structures intended to explore event correlation on a massive scale with data
that is fragmented across process and systems. As with most stream processing
frameworks, many WaterSlide processing functions favor efficient approximate
computation over less-efficient exact computations.
WaterSlide can be used for a variety of purposes. It is used to generate
interesting metadata from live data streams.
sauce:
https://github.com/waterslideLTS/waterslide
from:
https://code.nsa.gov/