README.md 7.5 KB
Newer Older
1 2
# SN - Sentinel networking library

3 4 5 6
[![pipeline status](https://gitlab.labs.nic.cz/turris/sentinel/sn/badges/master/pipeline.svg)](https://gitlab.labs.nic.cz/turris/sentinel/sn/commits/master)
[![coverage report](https://gitlab.labs.nic.cz/turris/sentinel/sn/badges/master/coverage.svg)](https://gitlab.labs.nic.cz/turris/sentinel/sn/commits/master)


7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89
This package implements a bunch of features that are common for all Sentinel
boxes:

- ZMQ networking
- Common configuration framework
- Logging in standardized way (necessary to handle log messages by TM - Turris
  Monitoring)
- Message queue for handling messages in safe and good-performing way

## Usage

There are 2 types of possible usage of this library:

1. `sn.SN` class - raw usage
2. `SNBox` class and it's non-abstract implementations `SNGeneratorBox`,
   `SNPipelineBox`, `SNTerminationBox` and `SNMultipleOutputPipelineBox`

### Which one is the best for me?

#### SNBox

`SNBox` provides safe implementation for straightforward box behavior like
`out-only`, `in-out` or `in-only` type of processing.

It handles all unexpected Exceptions from the box and tries to recover from this
types of error. Every box is restated by `systemd` but it could be slow and it
could unnecessarily drop messages from queues. `SNBox` safes every box from this
suffer.

It aims to be a programmer-friendly and provides naive and straightforward API.

Use `SNBox` for every box in pipeline.

#### sn.SN

On the other hand, `sn.SN` provides only the basic gateway for common
configuration. All additional features are used independently.

Use `sn.SN` for every box that is part of Sentinel Network but has non-trivial
requirements for communication pattern.


#### Examples

`SNBox` is used across all pipeline boxes and almost all boxes of DynFW.

`sn.SN` is used for `smash` (2 event loops can't work together) or DynFW
`publisher` (encryption at public socket).

## SNBox

Every box is implemented as a class that inherits from one of these non-abstract
`SNBox` implementations:

- `SNGeneratorBox` - `out-only` behavior - box generates messages from another
  source - e.g. feeders for DynFW
- `SNPipelineBox` - `in-out` behavior - box takes one message from network,
  makes some changes in message or adds some data and sends message to the next
  box - e.g. `geoip`
- `SNTerminationBox` - `in-only` behavior - box takes message from network and
  stores it into DB for example
- `SNMultipleOutputPipelineBox` - box is pretty similar to `SNPipelineBox` but
  is expected unlimited number of outgoing messages - e.g. DynFW
  `rules_collector`

### Usage

```python
class MyBox(SNPipelineBox):
    def setup(self):
        return {
            "my_resource": init_my_resource(),
        }

    def teardown(self):
        self.ctx.my_resource.destroy()

    def process(self, msg_type, payload):
        if msg_type == "msg/i/should/care/about":
            payload["new_data"] = add_some_interesting_field()

            return msg_type, payload

Michal Mladek's avatar
Michal Mladek committed
90
if __name__ == "__main__":
91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
    MyBox("by_box_name").run()
```

#### Setup

Optional. Box could allocate here some resources that will be needed. All
resources must be returned as a dictionary. Otherwise will be thrown
`SetupError` exception.

All resource initialized by `setup()` will be available in
`self.ctx.RESOURCE_NAME`.

Box should not use `self` for its data.

#### Teardown

Optional. Box is able to make safe cleanup in this function. All resources are
available in `self.ctx.RESOURCE_NAME`.

#### Process

Mandatory. Box obtains every message in 2 variables: `msg_type` (string with
message type identification - see Sentinel documentation for details) and
`payload` - Python dictionary containing the whole message.

`process` returns:
- `None` - there is no reasonable answer for message
- a `tuple` of `msg_type` and `payload` determines outgoing message

#### Before first request

Optional. There is one more function:

```python
    def before_first_request(self):
        payload = get_some_data()  # Dictionary!

        return "sentinel/my/type", payload
```

This function is executed after full initialization of the box and before start
of the message loop itself. Returns `None` or tuple `(msg_type, payload)` as in
case of `process()`.

Currently, used only for DynFW `rules_collector`.


#### Specialities

##### `SNGeneratorBox`

`process` do not accept `msg_type` and `payload` and `process` must be a
generator - uses `yield` keyword.

##### `SNTerminationBox`

Non-`None` return value is treated as a error.

##### `SNMultipleOutputPipelineBox`

Return value is a `list` of tuples:

```python
        return [
            ("sentinel/type/one", payload1),
            ("sentinel/type/two", payload2),
        ]
```

The same behavior is expected in `before_first_request` function.

#### Available resources

`self` provides some common resources for box:
- `self.name` - name of the box provided to the constructor
- `self.logger` - initialized and configured logger
- `self.args` - parsed command line arguments
- `self.ctx` - user data from `setup()` function

Box should not use any other `self` data.

#### Examples

Basic examples are provided in `dev/` directory.

176 177 178 179 180 181 182 183 184 185 186 187 188
#### Programmer documentation

There is also available documentation more suitable for programmers of lower levels.

It is available in `docs/` directory and could be generated with `sphinx` tool:

```
pip install .[docs]
cd docs/
make html
BROWSER _build/html/index.html
```

189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303
## sn.SN

```python
def main():
    ctx = sn.SN(zmq.Context.instance())

    socket_in, socket_out = ctx.get_socket("in", "out")

    while True:
        msg = socket_in.recv_multipart()
        msg_type, payload = sn.parse_msg(msg)

        payload["new_data"] = add_some_interesting_field()

        msg = sn.encode_msg(msg_type, payload)
        socket_out.send_multipart(msg)


if __name__ == "__main__":
    main()
```

This example says it all. Nothing more is provided.


### `get_socket`

Socket (or resource in internal terminology) could be requested as:
```python
socket_in, socket_out = ctx.get_socket("in", "out")
```

or:

```python
socket_in = ctx.get_socket("in")
socket_out = ctx.get_socket("out")
```

for simple resource gathering.

There is an option to force socket type:

```python
socket_in = ctx.get_socket(("in", "PULL"))
```

## Logging

There are 2 preconfigured handlers:
- syslog - INFO and higher severity
- file under rotation - DEBUG and higher severity

Expected usage in `sn.SN` script:

```python
import logging
import sn
logger = logging.getLogger("component_name")
logger.info("I'm running!")
```

In `SNBox` is everything prepared in `self.logger`.

Your logger will inherit INFO level from root logger.

There is shared command line argument `-v` or `--verbose` that turns on DEBUG
level for file handler.

Please, do not change logging format for syslog handler. It will be parsed by
TM. File handler is prefixed by current time, for better debugging.

## Argument parser

Some scripts may need additional arguments.

Use this approach:
1. Get common SN argparser
2. Add needed arguments

```python

import sn

def my_argparser():
    parser = sn.get_arg_parser()
    parser.add_argument('--my-cool-feature',
                        action='store_true',
                        help='Enable by cool feature'
                       )

    return parser

```
3. Add new enriched argparser to `sn.SN`/`SNBox`

```python
ctx = sn.SN(zmq.Context.instance(), argparser=my_argparser())
```

or

```python
MyBox("by_box_name", argparser=my_argparser()).run()
```
4. Get parsed arguments as:
```python
ctx.args.ARGUMENT_NAME
```

or

```python
self.args.ARGUMENT_NAME
```