Contract No.: 285248 Strategic Objective

Storm cluster

• 
  Master node: it runs a daemon called Nimbus. This process is responsible for distributing code around the cluster, assigning tasks to machines and monitoring for failures.
  
• 
  Worker nodes: each worker node runs a daemon called Supervisor. This process listens for work assigned to its machine and starts and stops worker processes as necessary based on the tasks the master node has assigned to it.
  

Each worker executes a subset of what is called topology. A topology in Storm's terminology is a graph of computation. It consists of a set of data sources (spout) and data operations (bolts) connected with stream grouping. Each node (spout/bolt) in a topology contains processing logic and links between nodes (stream grouping) indicate how data should be passed around between nodes. Consequently, there are two types of nodes or abstractions in a topology:

• 
  Spout: it is a source of streams in the topology (for example, reading data from a file or listening for incoming data in port). It provides the interface to make this process reliable, ensuring to resend a tuple (which is an ordered list of data items) in case Storm fails to process it.
  
• 
  Bolt: it is responsible for all the data processing (from filtering to joins, aggregations, reading and writing to files or databases, etc) that takes place in the topology. A bolt consumes any number of input streams (from a spout or other bolt) and produce new output streams (emit tuples to another bolt).
  

A stream grouping tells a topology how to send tuples among sets of bolt's task. It is important to remark that spouts and bolts will be executed in parallel as many tasks across the cluster. Consequently, the flow of streams between them must be correctly configured to guarantee fully message processing according the application logic. There are several built-in stream groupings included in Storm but the two main used in the Service Level SIEM are:

• 
  Shuffle grouping: in this case, tuples are randomly distributed across the bolt's tasks in a way such that each bolt is guaranteed to get an equal number of tuples.
  
• 
  Fields grouping: in this case, the stream is partitioned by the fields specified in the grouping. For example, if the stream is grouped by the "user-id" field, tuples with the same "user-id" will always go to the same task, but tuples with different "user-id" may go to different tasks.
  

• 
  Guaranteed message processing: each tuple coming off the OSSIM agents will be fully processed in a distributed manner. Storm provides the capability to track the tree of messages that a tuple triggers in an efficient way so in case of any failure, the tuple can be resent from the spout.
  
• 
  Robust process management with the use of Storm, Zookeeper and Supervisor running together.
  
• 
  Fault detection and automatic reassignment: tasks in a running topology heartbeat to Nimbus to indicate that they are running smoothly. Nimbus monitors heartbeats and will reassign tasks that have timed out. Additionally, all the tasks throughout the cluster that were sending messages to the failed tasks quickly reconnect to the new location of the tasks.
  
• 
  Efficient message passing: messages are passed directly between tasks using ZeroMQ without intermediate queuing.
  

Service Level SIEM Topology

In order to run the Service Level SIEM in a Storm cluster to have high-performance and be able of processing and correlating pattern of events from a more complex and business perspective, the following topology has been defined:



Service Level SIEM Topology

• 
  AgentOSSIMSpout
  

This spout is listening in a predefined port (41000/tcp by default) for events coming from the OSSIM Agents. These events will be already normalized to the ossim event format. This Spout emits tuples with only a field called 'ossimEvent' that includes the event sent by the ossim-agent. The listening port can be configured in the conf/ServiceLevelSIEM.conf file.

• 
  GE_FilterBolt
  

This bolt process will make a first event preprocessing detecting the FI-WARE GE that generated them. With this purpose, the userdata1 field in the ossim event must include the name of the device that generated it. This process receives the output from the AgentOSSIMSpout and emits tuples with two fields: "FIWARE_GE" and "event".

• 
  PRRS_FilterBolt
  

At this step of the topology it is possible to have different bolt processes depending on the event processing or filtering required by each FI-WARE GE that sent the events. Like an example, the PRRS_FilterBolt is implemented to process events coming from the Context-based Security & Compliance GE. This bolt process included in the Storm cluster (with parallelism configurable by file) uses an eventSchema file to create output tuples only with the relevant fields for the correlation. The conf/schemas/eventSchema file includes a line that starts with the name of the Generic Enabler (for example "PRRS:") and then a list separated by commas of pairs: =. The ossim fields are recovered from the "event" field received in the incoming tuples and the output fields are the one emitted by this process to the following one in the topology. This bolt is connected to the previous GE_FilterBolt with a "fields grouping" through the first field "FIWARE_GE" so all the events coming from a same Generic Enabler can be correlated.

• 
  CorrelationBolt
  

This bolt is the process included in the Service Level SIEM topology that will perform the correlation of the tuples emitted by the previous filter bolt that contains only the relevant fields for the specific FI-WARE GE. In the case of the events coming from the Context-based Security & Compliance GE, they arrive with a field grouping based on the field "securitySpecs", but this is configurable depending on the event source. The language chosen for the correlation that will arise alarms from a business perspective in this Service Level SIEM topology is the Event Processing Language (EPL). For this reason, the Esper (http://esper.codehaus.org/) libraries are used in this correlation bolt process.

• 
  DBWriterBolt
  

This bolt process included in the Service Level SIEM topology is the one in charge of writing the alarms received from the correlation bolt into the OSSIM Database. The database information is read from the conf/ServiceLevelSIEM.conf file. In this case, a Shuffle Grouping is used because each tuple received from whatever correlation bolt represents an alarm to be written immediately in the database to be available for the Visualization Framework. In the current implementation, these alarms are stored in the OSSIM database in a table called sls_alarm with the following fields:

Field name	Description
sls_alarm_id	It is the primary key that identifies the alarm generated
rule_id	It identifies the type of alarm. All available type of alarms are stored in a table called sls_rule
msg	Short description of the alarm
timestamp	Timestamp when the alarm was stored in the database
firstEvent_id	Identifier of the first event included in the pattern that generated the alarm. The information about this event (for example timestamp, source_ip, destination_ip, etc.) is stored in the table Event with all the other events received by the OSSIM Agents.
lastEvent_id	Identifier of the last event included in the pattern that generated the alarm. The information about this event (for example timestamp, source_ip, destination_ip, etc.) is stored in the table Event with all the other events received by the OSSIM Agents.

In order to make it more flexible, an alarmScheme file is used to define the list of fields to be recovered from the tuples received from the Correlation Bolts. This file includes the list of those fields separated by commas. For example:

AlarmID, AlarmMSG, FirstEvent, LastEvent

Besides, the file sqlsAlarmsDB.conf is used to define the list of SQL commands to interact with the database. Currently it includes the following one:

SQLInsertSLSAlarm = INSERT INTO sls_alarm (rule_id, msg, timestamp, firstEvent_id, lastEvent_id) values ({0},''{1}'',now(),0x{2},0x{3});

Service Level Correlation Rules

The language chosen for the correlation rules that will arise alarms from a business perspective in this Service Level SIEM topology is the Event Processing Language (EPL). This language is quite simple because it is similar to the well-known SQL in the use of the select and where clause. But it also allows expressing rich event conditions or patterns, correlation and time windows. The EPL statements, variables and the correlation rules that will trigger alarms are defined through the following configuration files that will be used by the correlation bolt process included in the topology:

• 
  EPLVariables.conf
  

This file is used to define variables to be used in the EPL statements. They depends on the correlation to be performed to arise alarms from a service level or bussiness perspective. Each line represents a variable and it is composed by three parts separated by comas: For example:

var_timeout,integer,10

AlarmMSG,string,FI-WARE additional security level compromised in client environment

• 
  EPLStatements.conf
  

This file is used to define statements in EPL language to be used in the correlation process. It is important to remark that these statements will not generate alarms in the correlation process. They will be used for example to define the objects (events) to be used in a pattern of events in the correlation rule. They can be also used to define variables (instead the EPLVariables.conf file) or any other EPL statement required for the correlation. For example:

insert into PRRSEventA select * from filterBolt_default where (FIWARE_GE = "prrs") and (status = "non-compliance")

insert into PRRSEventB select * from filterBolt_default where (FIWARE_GE = "prrs") and (status = "undeployed")

insert into PRRSEventC select * from filterBolt_default where (FIWARE_GE = "prrs") and (status = "not-found")

insert into PRRSEventD select * from filterBolt_default where (FIWARE_GE = "prrs") and (status = "deployed")

• 
  ServiceLevelSIEM.conf
  

The Service Level SIEM configuration file includes a parameter to defined the correlation rule that will generate the alarm. This statement will have asocciated a listener in the correlation process what means that when the condition included in it happens, a tuple will be emitted to the following process in the Service Level SIEM topology to write that alarm in the database.

correlation_rule = select AlarmID, AlarmMSG, a.event_id as FirstEvent, c.event_id as LastEvent from pattern [every (a=PRRSEventA) -> b=PRRSEventB(client_ip=a.client_ip and sec_enabler=a.sec_

enabler) -> c=PRRSEventC(client_ip=a.client_ip and securitySpecs=a.securitySpecs) where timer:within(var_timeout sec)]

In our example this correlation rule is a pattern of events but it could have any other condition depending on the client and the alarm to be generated. It will use the variables and statements defined in files indicated in the EPLVariables and EPLStatements parameters, respectively.

The parameters alarmMSGToEmit, alarmIDToEmit, eventToEmit1 and eventToEmit2 are the name of the fields that will be emitted by the correlation process. Those names can be modified depending on the correlation rule defined but they must be the same included in the alarmSchema file used by the DBWriteBolt process.

More information about the EPL language can be found in: http://docs.oracle.com/cd/E13157_01/wlevs/docs30/epl_guide/overview.html

Download 1.78 Mb.

Share with your friends: