To make it easier to add Broccoli support to your build environment, the distribution comes with a shell script that provides the necessary compiler and linker flags (similarly to many other packages): broccoli-config. Use the --cflags option to obtain the compiler options and --libs to obtain the linker options.
If you use the autoconf/automake tools, we recommend something along the following lines for your configure script:
dnl ################################################## dnl # Check for Broccoli dnl ################################################## AC_ARG_WITH(broccoli-config, AC_HELP_STRING([--with-broccoli-config=FILE], [Use given broccoli-config]), [ brocfg="$withval" ], [ AC_PATH_GENERIC(broccoli,, brocfg="broccoli-config", AC_MSG_ERROR(Cannot find Broccoli: Is broccoli-config in path? Use more fertilizer?)) ]) broccoli_libs=`$brocfg --libs` broccoli_cflags=`$brocfg --cflags` AC_SUBST(broccoli_libs) AC_SUBST(broccoli_cflags) |
Often you will want to make existing applications Bro-aware, that is, instrument them so that they can send and receive Bro events at appropriate moments in the execution flow. This will involve modifying an existing code tree, so care needs to be taken to avoid unwanted side effects. By protecting the instrumented code with #ifdef/#endif statements you can still build the original application, using the instrumented source tree. The broccoli-config script helps you in doing so because it already adds -DBROCCOLI to the compiler flags reported when run with the --cflags option:
cpk25@localhost:/home/cpk25 > broccoli-config --cflags -I/usr/local/include -I/usr/local/include -DBROCCOLI |
So simply surround all inserted code with a preprocessor check for BROCCOLI and you will be able to build the original application as soon as BROCCOLI is not defined.
Time for some code. In the code snippets below we will introduce variables whenever context requires them and not necessarily when C requires them. The library does not require calling a global initialization function. In order to make the API known, include broccoli.h:
#ifdef BROCCOLI #include <broccoli.h> #endif |
![]() | A note on Broccoli's memory management philosophy: Broccoli generally does not release objects you allocate. The approach taken is "you clean up what you allocate." |
Broccoli declares a number of data types in broccoli.h that you should know about. The more complex ones are kept opaque, while you do get access to the fields in the simpler ones. The full list is as follows:
Simple unsigned types: uint, uint32, uint16 and uchar.
Connection handles: BroConn, kept opaque.
Bro events: BroEvent, kept opaque.
Buffer objects: BroBuf, kept opaque. See the separate section on buffer management for details.
Records: BroRecord, kept opaque. See the separate section on record handling for details.
Strings (character and binary): BroString, defined as follows:
typedef struct bro_string { int str_len; char *str_val; } BroString; |
BroStrings are mostly kept transparent for convenience; please have a look at the string API: bro_string_init(), bro_string_set(), bro_string_set_data(), bro_string_copy(), bro_string_cleanup(), and bro_string_free().
Ports: BroPort for network ports, defined as follows:
typedef struct bro_port { uint16 port_num; /* port number in host byte order */ int port_proto; /* IPPROTO_xxx */ } BroPort; |
Subnets: BroSubnet, defined as follows:
typedef struct bro_subnet { uint32 sn_net; /* IP address in network byte order */ uint32 sn_width; /* Length of prefix to consider. */ } BroSubnet; |
You need to obtain a connection handle before you can receive or send events. Connection handles are pointers to BroConn structures, and kept opaque. Use bro_connect() or bro_connect_str() to obtain such a handle, depending on what parameters are more convenient for you: the former accepts the IP address and port number as separate numerical arguments, the latter uses a single string to encode both, in "hostname:port" format. bro_disconnect() terminates a connection and releases all resources associated with it. You can create as many connections as you like, to one or more peers. You can obtain the file descriptor of a connection using bro_conn_get_fd().
char host_str = "bro.yourcompany.com"; int port = 1234; struct hostent *host; BroConn *bc; if (! (host = gethostbyname(host_str)) || ! (host->h_addr_list[0])) { /* Error handling -- could not resolve host */ } /* Connect to Bro */ if (! (bc = bro_connect((struct in_addr*) host->h_addr_list[0], htons(port)))) { /* Error handling - could not connect to Bro */ } /* Send and receive events ... */ /* Disconnect from Bro */ bro_disconnect(bc); |
Or simply use the string-based version:
char host_str = "bro.yourcompany.com:1234"; BroConn *bc; /* Connect to Bro */ if (! (bc = bro_connect_str(host_str))) { /* Error handling - could not connect to Bro */ } |
In order to send an event to the remote Bro agent, you first create an empty event structure with the name of the event, then add parameters to pass to the event handler at the remote agent, and then send off the event. You need to make sure the remote Bro is interested in receiving that type of event. We'll talk more about this in a moment.
Let's assume we want to request a report of all connections a remote Bro currently keeps state for that match a given destination port and host name and that have amassed more than a certain number of bytes. The idea is to send an event to the remote Bro that contains the query, identifiable through a request ID, and have the remote Bro answer us with remote_conn events containing the information we asked for. The definition of our requesting event could look as follows in the Bro policy:
event report_conns(req_id: int, dest_host: string, dest_port: port, min_size: count); |
First, create a new event:
BroEvent *ev; if (! (ev = bro_event_new("report_conns"))) { /* Error handling - could not allocate new event. */ } |
Now we need to add parameters to the event. The sequence and types must match the event handler declaration — check the Bro policy to make sure they match. The function to use for adding parameter values is bro_event_add_val() All values are passed as a pointer arguments and are copied internally, so the object you're pointing to stays unmodified at all times. You clean up what you allocate. In order to indicate the type of the value passed into the function, you need to pass a numerical type identifier along as well. Table 1 lists the value types that Broccoli supports along with the type identifier and data structures to point to.
Table 1. Types, type tags, and data structures for event parameters in Broccoli
Type | Type Tag | Data structure |
---|---|---|
Boolean | BRO_TYPE_BOOL | int |
Integer value | BRO_TYPE_INT | int |
Counter (nonnegative integers) | BRO_TYPE_COUNT | uint32 |
Floating-point number | BRO_TYPE_DOUBLE | double |
Timestamp | BRO_TYPE_TIME | double (see also bro_util_timeval_to_double() and bro_util_current_time()) |
Time interval | BRO_TYPE_INTERVAL | double |
Strings (text and binary) | BRO_TYPE_STRING | BroString (see also the family of bro_string_xxx() functions) |
Network ports | BRO_TYPE_PORT | BroPort, with the port number in host byte order |
IPv4 address | BRO_TYPE_IPADDR | uint32, in network byte order |
IPv4 network | BRO_TYPE_NET | uint32, in network byte order |
IPv4 subnet | BRO_TYPE_SUBNET | BroSubnet, with the sn_net member in network byte order |
Record | BRO_TYPE_RECORD | BroRecord (see also the family of bro_record_xxx() functions and the explanation below) |
Knowing these, we can now compose a request_connections event:
BroString dest_host; BroPort dest_port; uint32 min_size; int req_id = 0; bro_event_add_val(ev, BRO_TYPE_INT, &req_id); req_id++; bro_string_set(&dest_host, "desthost.destdomain.com"); bro_event_add_val(ev, BRO_TYPE_STRING, &dest_host); bro_string_cleanup(&dest_host); dest_port.dst_port = 80; dest_port.dst_proto = IPPROTO_TCP; bro_event_add_val(ev, BRO_TYPE_PORT, &dest_port); min_size = 1000; bro_event_add_val(ev, BRO_TYPE_COUNT, &min_size); |
All that's left to do now is to send off the event. For this, use bro_event_send() and pass it the connection handle and the event. The function returns TRUE when the event could be sent right away, and FALSE when this was not possible. This does not mean that there was an error — likely the connection was just not ready to send the event at this point. In the latter case, the event is queued internally and sent at a later point. Whenever you call bro_event_send(), Broccoli attempts to send as much of an existing event queue as possible. Again, the event is copied internally to make it easier for you to send the same event repeatedly. You clean up what you allocate.
bro_event_send(bc, ev); bro_event_free(ev); |
Two other functions may be useful to you: bro_event_queue_empty() tells you whether there currently are queued events, and bro_event_queue_flush() attempts to flush the current event queue and returns the number of events that do remain in the queue after the flush. Note: you do not normally need to call this function, queue flushing is attempted every time you send an event.
Before a remote Bro will accept your connection and your events, it needs to have its policy configured accordingly:
Load either listen-clear or listen-ssl, depending on whether you want to have cleartext or encrypted communication. The former is easier to set up but obviously not recommended for deployment. See below for details on how to set up encrypted communication via SSL.
Redefine the destinations table, adding an entry for the new connection. This involves coming up with a tag for the connection under which the connection can be found in the table (a creative one would be "broccoli"), the IP from which you will be connecting, the pattern of names of the events the Bro will accept from you, whether you want Bro to connect to your machine on startup or not, a retry timeout, and whether to use SSL. In the policy this could look as follows:
@load listen-clear redef Remote::destinations += { ["broping"] = [$host = 127.0.0.1, $events = /ping/, $connect=F, $retry = 60 secs, $ssl=F] }; |
This example is taken from broping.bro, the policy the remote Bro must run when you want to use the broping tool explained in the section on testing below.
Broccoli also supports records, i.e., values that pack more than one value together. In Broccoli, the way you handle records is somewhat similar to events: after creating an empty record (of opaque type BroRecord, you can iteratively add vals to it. The main difference here is that you must specify a field name with the val; each val in a record can be identified both by position (a numerical index starting from zero), and by field name. You can retrieve vals in a record by field index or field name. You can also reassign values.
There is currently no separate definition of record types. You defined the type of a record implicitly by the sequence of field names and the sequence of the types of the vals you put into the record.
All fields in a record must be assigned before it can be shipped.
The API you want to look at consists of bro_record_new(), bro_record_free(), bro_record_add_val(), bro_record_get_nth_val(), bro_record_get_named_val(), bro_record_get_nth_val(), and bro_record_get_named_val(). Whenever type identifiers are required, please refer to Table 1 again for the valid type tag – data structure combinations.
Receiving events is a little more work because you need to
let the remote Bro agent know that you would like to receive those events,
tell Broccoli what to do with the various arriving events,
and when instrumenting an existing application, find a spot in the code path suitable for extracting and processing arriving events.
When Broccoli receives an event, it tries to dispatch the event through a callback registry. Any callbacks that are registered for the arrived event's name are invoked with the parameters shipped with the event. In order to register a callback, use bro_event_registry_add() and pass it the connection handle, the name of the event for which you register the callback, and the callback itself that matches the signature of the BroEventFunc type. The callback's type is defined rather generically as follows:
typedef void (*BroEventFunc) (BroConn *bc, ...); |
As you can see, all it requires is a connection handle as its first argument. Broccoli will pass the connection handle of the connection on which the event arrived through to the callback. BroEventFuncs are variadic, because each callback you provide is directly invoked with pointers to the parameters of the event, in a useful format. All you need to know is what type to point to in order to receive the parameters in the right layout. Refer to Table 1 again for a summary of those types. Continuing our example, we will want to process the connection reports that contain the responses to our report_conns event. Let's assume those look as follows:
event remote_conn(req_id: int, conn: connection); |
The reply events contain the request ID so we can associate requests with replies, and a connection record (defined in bro.init in Bro. (It'd be nicer to report all replies in a single event but we'll ignore that for now.) For this event, our callback would look like this:
void remote_conn_cb(BroConn *bc, int *req_id, BroRecord *conn); |
Now we register the callback using bro_event_registry_add():
bro_event_registry_add(bc, "remote_conn", remote_conn_cb); |
If you have multiple events you are interested in, register each one in the same way. At this point, Broccoli knows how to handle these events when they arrive. What's left to do is to let the remote Bro agent know that you would like to receive the events for which you registered. To do so, call bro_event_registry_request():
bro_event_registry_request(bc); |
Broccoli also supports unrequesting events, however support for dynamic event requesting/unrequesting is not implemented in Bro yet.
You will often find that you would like to connect data with a BroConn. Broccoli provides an API that lets you associate data items with a connection handle through a string-based key–value registry. The functions of interest are bro_conn_data_set(), bro_conn_data_get(), and bro_conn_data_del(). You need to provide a string identifier for a data item and can then use that string to register, look up, and remove the associated data item. Note that there is currently no mechanism to trigger a destructor function for registered data items when the Bro connection is terminated. You therefore need to make sure that all data items that you do not have pointers to via some other means are properly released before calling bro_disconnect().
Imagine you have instrumented the mother of all server applications. Building it takes forever, and every now and then you need to change some of the parameters that your Broccoli code uses, such as the host names of the Bro agents to talk to. To allow you to do this quickly, Broccoli comes with support for configuration files. All you need to do is change the settings in the file and restart the application (we're considering adding support for volatile configuration items that are read from the file every time they are requested).
Configuration is done in single configuration file per installation. You can obtain the location of this config file by running broccoli-config --config. In the configuration file, a "#" anywhere starts a comment that runs to the end of the line. Configuration items are specified as key-value pairs, as can be seen in this default file:
# This is the Broccoli system-wide configuration file. # # Entries are of the form <identifier> <value>, where the identifier # is a sequence of letters, and value can be a string (including # whitespace), and floating point or integer numbers. Comments start # with a "#" and go to the end of the line. For boolean values, you # may also use "yes", "on", "true", "no", "off", or "false". # Strings may contain whitespace, but need to be surrounded by # double quotes '"'. # # Examples: # Foo/PeerName mybro.securesite.com Foo/PortNum 123 Bar/SomeFloat 1.23443543 Bar/SomeLongStr "Hello World" |
You can name identifiers any way you like, but to keep things organized it is recommended to keep a namespace hierarchy similar to the file system. In the code, you can query configuration items using bro_conf_get_str(), bro_conf_get_int(), and bro_conf_get_dbl().
Broccoli provides an API for dynamically allocatable, growable, shrinkable, and consumable buffers with BroBufs. You may or may not find this useful — Broccoli mainly provides this feature in broccoli.h because these buffers are used internally anyway and because they are typical case of something that people implement themselves over and over again, for example to collect a set of data before sending it through a file descriptor, etc.
The buffers work as follows. The structure implementing a buffer is called BroBuf. BroBufs are initialized to a default size when created via bro_buf_new(), and released using bro_buf_free(). Each BroBuf has a content pointer that points to an arbitrary location between the start of the buffer and the first byte after the last byte currently used in the buffer (see buf_off in the illustration below). The content pointer can seek to arbitrary locations, and data can be copied from and into the buffer, adjusting the content pointer accordingly. You can repeatedly append data to end of the buffer's used contents using bro_buf_append().
<---------------- allocated buffer space ------------> <======== used buffer space ========> ^ ^ ^ ^ | | | | `buf `buf_ptr `buf_off `buf_len |
Have a look at the following functions for the details: bro_buf_new(), bro_buf_free(), bro_buf_append(), bro_buf_get(), bro_buf_get_end(), bro_buf_get_size(), bro_buf_get_used_size(), bro_buf_ptr_get(), bro_buf_ptr_tell(), bro_buf_ptr_seek(), bro_buf_ptr_check(), and bro_buf_ptr_read().
This is TODO, but here's Robin's HOWTO:
How to create certificates to authorize Bro's SSL connections ============================================================= - Create a global CA key/certificate once: * Create some directory to store the CA stuff, and create a few things there: mkdir <ca-dir> cd <ca-dir> mkdir private newcerts cert crl chmod 700 private touch index.txt echo 01 >serial cp bro/openssl.conf . * Create a private CA key: openssl genrsa -des3 -out private/ca_key.pem * Self-sign it: openssl req -new -x509 -key private/ca_key.pem -out ca_cert.pem -days 1095 - For each Bro: * Create a private key (w/o password): openssl genrsa -out bro_key.pem * Create a certification request: openssl req -new -key bro_key.pem -out bro.csr * Create a certificate using the CA key: openssl ca -config openssl.cnf -in bro.csr -out bro_cert.pem * Verify that the certicate is ok: openssl verify -CAfile ca_cert.pem bro_cert.pem * Concat Bro key and certificate: cat bro_key.pem bro_cert.pem >bro.pem * Copy this and the CA certificate to the IDS machine: scp bro.pem ca_cert.pem ids:... * Redef Bro's variables to point to the files: redef ssl_ca_certificate = "...../ca_cert.pem"; redef ssl_private_key = "...../bro.pem"; * Remove the unnecessary stuff: rm bro_key.pem bro.csr bro_cert.pem bro.pem |
The Broccoli distribution comes with a few small test programs, located in the test/ directory of the tree. The most notable one is broping [1], a mini-version of ping. It sends "ping" events to a remote Bro agent, expecting "pong" events in return. The remote Bro agent needs to run the following policy for this to work (included in the distribution):
@load listen-clear global ping_log: file &redef; redef ping_log = open_log_file("ping"); event remote_connection_established(ip: addr, p: port) { request_remote_events(ip, p, /ping/); } event ping(src_time: time) { event pong(src_time, current_time()); } event pong(src_time: time, dst_time: time) { print ping_log, fmt("ping received, %f at src, %f at dest, one-way: %f", src_time, dst_time, dst_time-src_time); } |
The @load statement is necessary to load the listen-clear policy, which causes the remote agent to listen and respond to clear-text connection attempts. Have a look at the policy — you can configure what IP addresses and port number the agent will be listening on. In the next two lines, a log file is created to which a line will be written whenever a pong event is triggered in response to a received ping event. Next is a remote_connection_established event handler. This handler gets invoked whenever a Bro agent (broping in this case) connects to the agent. Here, we request delivery of all events whose names match the /ping/ regular expression, thus just the ping event. Next is the definition of the ping event handler. You can see that ping events contain a timestamp (the time at which broping sent out the event), and that it triggers a pong event with the same timestamp and the current time. What to do when said pong event is triggered comes next: a line is dumped to the log file.
So how do the pong events ever end up at the broping client? broping simply asks for delivery of all pong events from the remote agent, using the same mechanism as shown above, but hard-coded in C. After that, every triggered pong event is sent off to broping, along with the timestamp of the creation of the ping event and the timestamp of the creation of the pong event.
When running broping, you'll see something like this:
cpk25@localhost:/home/cpk25/devel/broccoli > ./test/broping pong event from 127.0.0.1: seq=1, time=0.004700/1.010303 s pong event from 127.0.0.1: seq=2, time=0.053777/1.010266 s pong event from 127.0.0.1: seq=3, time=0.006435/1.010284 s pong event from 127.0.0.1: seq=4, time=0.020278/1.010319 s pong event from 127.0.0.1: seq=5, time=0.004563/1.010187 s pong event from 127.0.0.1: seq=6, time=0.005685/1.010393 s |
[1] | Pronunciation is said to be somewhere on the continuum between "brooping" and "burping". |