The SDB (Simple Data Base) interface is designed to store and retrieve data that, unlike pillars and grains, is not necessarily minion-specific. It is a generic database interface for SaltStack.

We will show how we can make use of SDB for storing and retrieving passwords.

SDB Configuration

The SDB interface requires a profile to be set up in the master configuration file.

/etc/salt/master.d/passwords.conf

The configuration stanza includes the name/ID that the profile will be referred to as, a driver setting, and any other arguments that are necessary for the SDB module that will be used.

 pwd:
     driver: json
     data: /srv/salt/common/pwd.json

We will store the data in the JSON format and make use of the sdb execution module to get, set and delete values from this file. These three methods can be implemented as follows in the python script /srv/salt/_sdb/json.py

'''
SDB module for JSON

Like all sdb modules, the JSON module requires a configuration profile
to be configured in either the minion or, as in our implementation,
in the master configuration file (/etc/salt/master.d/passwords.conf).
This profile requires very little:

    .. code-block:: yaml

      pwd:
        driver: json
        data: /srv/salt/common/pwd.json

The ``driver`` refers to the json module and ``data`` is the path
to the JSON file that contains the data.

This file should be saved as salt/_sdb/json.py

.. code-block:: yaml

    user: sdb://pwd/user1

CLI Example:

    .. code-block:: bash

        sudo salt-run sdb.get sdb://pwd/user1
'''
from __future__ import absolute_import
from salt.exceptions import CommandExecutionError
import salt.utils
import json

__func_alias__ = {
    'set_': 'set'
}

def _read_json(profile):
    '''
    Return the content of a JSON file
    '''
    try:
        with salt.utils.fopen(profile['data'], 'r') as fp_:
            return json.load(fp_)
        except IOError as exc:
            raise CommandExecutionError(exc)
        except KeyError as exc:
            raise CommandExecutionError(
                '{0} needs to be configured'.format(exc))
        except ValueError as exc:
            raise CommandExecutionError(
                'There was an error with the JSON data: {0}'.format(exc))

def get(key, profile=None):
    '''
    Get a value from a JSON file
    '''
    json_data = _read_json(profile)
    return json_data.get(key, {})

def set_(key, value, profile=None):
    '''
    Set a key/value pair in a JSON file
    '''
    json_data = _read_json(profile)
    json_data[key] = value

    try:
        with salt.utils.fopen(profile['data'], 'w') as fp_:
            json.dump(json_data, fp_, indent=2, sort_keys=True)
    except IOError as exc:
        raise CommandExecutionError(exc)

    return get(key, profile)

You can now store the hashed passwords in the JSON data file

{
  "user1": "$5$tEpxpTHeP...0128tglwMKE.X9b88fO4x0",
  "user2": "$5$n4XiZajqf...P3BrvFM5hYq.UazR4dHxl8"
}

and quering SDB to get the hashed strings:

$ sudo salt-run sdb.get sdb://pwd/user1

Of course the SDB query can be coded in a pillar .sls file.

users:
  user1:
    fullname: User One
    uid: 2000
    gid: 1000
    password: {{ salt['sdb.get']('sdb://pwd/user1') }}

SaltStack (or Salt, for short) is a Python-based open-source configuration management software and remote execution engine. It supports the “Infrastructure as Code” approach (the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools) to deployment and cloud management.

I’ll describe in this short article how you can extend the Salt functionalities through the addition of a new execution module.

Just to clarify, you don’t need to be able to write Python or other code to use Salt in the normal usage case. Adding extensions to Salt is an “advanced”, but quite interesting topic.

A bit of theory

A Salt execution module is a Python (2.6+) or Cython module, though some specificities do exist, placed in a directory called _modules at the root of the Salt file server, usually /srv/salt.

An execution module usually define a __virtual__() function, to determine whether the requirements for that module are met, and the __virtualname__ string variable, that is used by the documentation build system to know the virtual name of a module without calling the __virtual__ function.

The following example from the official documentation should clarify this point.

A huge and consistent corpus of libraries and functions are packaged in the Salt framework. You can speed up and highly simplify the development of your modules by making use of those resources.

# Some examples of import:
import salt.utils
import salt.utils.itertools
import salt.utils.url
import salt.fileserver
from salt.utils.odict import OrderedDict

See the official documentation for more information, or… wait for a future article.

As an example, we can write a simple module returning some information about the CPU architecture of a Linux host.

Step by step development

We start by importing some Python and Salt libraries and by defining the  __virtualname__ variable and a __virtual__() function.

# Import Python libs
import logging
# Import Salt libs
from salt.exceptions import CommandExecutionError

__virtualname__ = 'cpuinfo'

def __virtual__():
    '''
    Only run on Linux systems
    '''
    if __grains__['kernel'] != 'Linux':
        return (False,
            'The {0} execution module cannot be loaded: '
            'only available on Linux systems.'.format(
            __virtualname__))
    return __virtualname__

As you can see, __virtual__() returns  False  when the operating system is not Linux. This means that the module cpuinfo will be only available for Linux minions and hidden otherwise.

Salt comes with an interface to derive information about the underlying system. This is called the grains interface, because it presents salt with grains of information. Grains are collected for the operating system, domain name, IP address, kernel, OS type, memory, and many other system properties. The __grains__ dictionary contains the grains data generated by the minion that is currently being worked with.

It’s time now to implement the logic of our module.

log = logging.getLogger(__name__)

def _verify_run(out):
    '''
    Crash to the log if command execution was not
    successful.
    '''
    if out.get('retcode', 0) and out['stderr']:
        log.debug('Return code: {0}'.format(
            out.get('retcode')))
        log.debug('Error output\n{0}'.format(
            out.get('stderr', 'N/A')))
        raise CommandExecutionError(out['stderr'])

def _lscpu():
    '''
    Get available CPU information.
    '''
    try:
        out = __salt__['cmd.run_all']("lscpu")
    except:
        return None

    _verify_run(out)

    data = dict()
    for descr, value in [elm.split(":", 1) \
        for elm in out['stdout'].split(os.linesep)]:
            data[descr.strip()] = value.strip()

    cpus = data.get('CPU(s)')
    sockets = data.get('Socket(s)')
    cores = data.get('Core(s) per socket')
    return (cpus, sockets, cores)

Note that the functions _lscpu() and _verify_run() have their names starting with an underscore (Python weak “internal use” indicator) and thus, by convention, will not be exported by Salt to the public interface.

The Salt method cmd.run_all is used here to execute an external binary (lscpu) and grasp its standard output and error.

The function _verify_run() aims to catch any system error and, when necessary, abort the module execution. This code snippet shows the usage of the Python exceptions in Salt. We raise here a CommandExecutionError exception, declared in the Salt library salt.exceptions if a system error has occurred.

To end our module we implement a function which just calls _lscpu() and parses the user command line arguments (if any), or the module extra arguments, when out module is called by another script. A CommandExecutionError exception is raised for any invalid argument passed to our function.

def lscpu(*args):
    (cpus, sockets, cores) = _lscpu()
    infos = {
        'cores': cores,
        'logicals': cpus,
        'sockets': sockets
    }

    if not args:
        return infos

    try:
        ret = dict((arg, infos[arg]) for arg in args)
    except:
        raise CommandExecutionError(
            'Invalid flag passed to {0}.proc'.format(
            __virtualname__))
    return ret

This function lscpu() is public and will be available on all the Linux minions managed by Salt. Any public method that you define in a module can be invoked by prefixing its name with the corresponding virtual module (cpuinfo in our case):

salt '*' cpuinfo.lscpu

or, if you just need the number of logical CPUs:

salt '*' cpuinfo.lscpu logicals

We have extended Salt.

The final module

When we put all of the preceding code together, we end up with the following code:

'''
SaltStack module returning some information about the CPU
architecture.  This module parses the output of the command lscpu.
'''
# Import Python libs
import logging
# Import salt libs
from salt.exceptions import CommandExecutionError

__virtualname__ = 'cpuinfo'

def __virtual__():
    '''
    Only run on Linux systems
    '''
    if __grains__['kernel'] != 'Linux':
        return (False,
            'The {0} execution module cannot be loaded: '
            'only available on Linux systems.'.format(
            __virtualname__))
    return __virtualname__

log = logging.getLogger(__name__)

def _verify_run(out):
    '''
    Crash to the log if command execution was not
    successful.
    '''
    if out.get('retcode', 0) and out['stderr']:
        log.debug('Return code: {0}'.format(
            out.get('retcode')))
        log.debug('Error output\n{0}'.format(
            out.get('stderr', 'N/A')))
        raise CommandExecutionError(out['stderr'])

def _lscpu():
    '''
    Get available CPU information.
    '''
    try:
        out = __salt__['cmd.run_all']("lscpu")
    except:
        return None
    _verify_run(out)

    data = dict()
    for descr, value in [elm.split(":", 1) \
        for elm in out['stdout'].split(os.linesep)]:
        data[descr.strip()] = value.strip()

    cpus = data.get('CPU(s)')
    sockets = data.get('Socket(s)')
    cores = data.get('Core(s) per socket')

    return (cpus, sockets, cores)

def lscpu(*args):
    '''
    Return the number of core, logical, and CPU sockets,
    by parsing the lscpu command and following back to
    /proc/cpuinfo when this tool is not available.

    CLI Example:

        .. code-block:: bash

            salt '*' cpuinfo.lscpu
            salt '*' cpuinfo.lscpu logicals
    '''
    (cpus, sockets, cores) = _lscpu()
    infos = {
        'cores': cores,
        'logicals': cpus,
        'sockets': sockets
    }
    if not args:
        return infos
    try:
        ret = dict((arg, infos[arg]) for arg in args)
    except:
        raise CommandExecutionError(
            'Invalid flag passed to {0}.proc'.format(
            __virtualname__))
    return ret

You can find other examples in this GitHub page.

A Python module for writing Xymon external scripts

monitoring-icon

I’m very happy to announce the immediate availability of PyXymon, release 3.

PyXymon is a simple helper Python module that can help you write Xymon external scripts in Python. PyXymon provides some methods for rendering the messages you want to display in the Xymon web page and for sending them to the Xymon server.

PyXymon reads the required information from the Xymon environment variables, so you do not need to add any extra configuration file.

This project is hosted at GitHub.

artificial-intelligence-brain-color

Another pretty good step forward in Deep Neural Networks from DeepMind.

They took inspiration from neurosciences-based theories about the consolidation of previously acquired skills and memories in mammalian and human brains: connections between neurons are less likely to be overwritten if they have been important in previously learnt tasks. This mechanism is known as “synaptic consolidation“.

The result is a neural network model that can learn several tasks without overwriting what was previously learnt (a known limitation of the current neural network approach,  known as “catastrophic forgetting”).

The new algorithm has been called “Elastic Weight Consolidation” (EWC).

All the details can be read in their last  PNAS paper.

cryptography-color

Bruce Schneier on New Security Threats from the Internet of Things

An article that’s worth reading.
The main point, that seems to me the original part of the speech, is the following equality:

IoT (Internet of Things)
==
world-size distributed robot across the Internet.

«Through the sensors, we’re giving the Internet eyes and ears. Through the actuators, we’re giving the Internet hands and feet. Through the processing — mostly in the cloud — we’re giving the Internet a brain. Together, we’re creating an Internet that senses, thinks, and acts. This is the classic definition of a robot, and I contend that we’re building a world-sized robot without even realizing it».

The Internet will be a computerized, networked, and interconnected world that we’ll live in, and made with devices often sold at a low margin by companies that simply don’t have the expertise to make them secure: computer security is becoming everything security.

Some emergent properties can eventually arise. These are unexpected behaviors that stem from interaction between the components of a complex system and their environment. In some contexts, emergent properties can be beneficial; users adapt products to support tasks that designers never intended. They can also be harmful if they undermine important safety requirements.

This is a definitely interesting topic.

I’m pleased to announce the immediate, free availability of the Nagios Plugins for Linux version 20.
Full details about what’s included can be found in the release notes.
As usual, you can download the sources from GitHub.
Bug reports, feature requests, and ideas for improvements are welcome!

nagios-plugins-linux-logo-color

Security fixes

Some insecure data handling issues discovered by Coverity in the new test framework have been fixed.

Enhancements

The Clang Static Analyser can now be executed by running the command

make -C tests check-clang-checker

in the project root directory. All the warnings spotted by this code analyser have been fixed.

A new Docker-based framework for packaging the Nagios Plugins for Linux (rpm and deb packages) is now available. The supported Linux distributions follow:

CentOS/RHEL 5, 6, 7
Debian 6, 7, 8
Fedora 24/25/rawhide.

The messages displayed in case of a too large “count” or “delay” error have been improved.

The release 19 of the Nagios Plugins for Linux is now available for download!

You can download the tarball from GitHub.

As usual, bug reports, feature requests, and ideas for improvements are welcome!

nagios-plugins-linux-logo-color

Fixes

check_multipath

Recent versions of multipath no longer open a multipathd socket file in the file system, but instead use an abstract namespace socket. Thanks to Chris Procter “chr15p” for reporting the issue and creating a pull request.

check_load

Fixed the performance data output.

Enhancements

check_multipath

Fixed the long-standing gcc compiler warning “dereferencing type-punned pointer might break strict-aliasing rules”.  This was a false problem, but the code has been modified to quiet the warning.

A larger buffer for queries is now set, to make this plugin working with systems that have lots of mapped disks.

Test Suite

A framework for testing the code (make check) has been added and some tests are now available.

Compatibility issues

check_multipath

By default the abstract namespace socket “/org/kernel/linux/storage/multipathd” is now selected at build time.
If you need to monitor old distributions (RHEL5 and RHEL6 for instance) you need to configure this package as followed:

./configure --with-socketfile=/var/run/multipathd.sock

 

What is one piece of advice you would like to give to students?

When deciding how to spend your time, I recommend you take into account two criteria:
  • Whether what you’re doing can change the world;
  • How much you’ll learn.

Even today, this is how I decide how to spend my time.

Our society today is incredibly good at giving individuals the opportunities to change the world. With digital technology and modern communications, ideas and products can spread faster than ever before. With the right ideas and strong execution, any person can quickly help a lot of others on our planet.

So, ask yourself: If what you’re working on succeeds beyond your wildest dreams, would you have significantly helped other people? If not, then keep searching for something else to work on. Otherwise you’re not living up to your full potential.

Second, especially when you’re young, don’t estimate the value of investing in your own future education.

My definition of “young” is anyone less than 100 years old.

Anything you learn will pay off for decades to come. But it won’t be easy. Once you’re out of school, investing time in learning has relatively few short-term rewards. There’s no teacher standing over your shoulder to give you a grade or motivate you to keep studying. But if you can inspire yourself or make it fun to keep reading, keep playing with ideas, keep talking to people that you can learn from, then over a span of years you can become incredibly talented in your areas of study.

For myself, I love reading. I have >1000 books on my kindle, and spend a lot of time in the evenings and weekends reading. My reading diet includes academic research papers, books on business strategy, the innovation process, products, biographies of people I admire, and more. I sometimes take MOOCs. I also love talking to people who can teach me new things, whether an old friend or a new acquaintance.

The process of learning will also help you decide what to work on. When you’ve seen enough examples of what others are doing to change the world, you’ll also get more and more ideas for how you can change the world yourself.

To summarize: Keep investing in your own learning, even when it’s hard. And keep searching for a way to contribute to something that helps humanity!

Andrew Ng, Chief Scientist at Baidu; Chairman/Co-Founder of Coursera;  Stanford faculty.

Here is it, version 18 of the Nagios Plugins for Linux.

nagios-plugins-linux-logo-color

It’s manly a bugfix release with a fix for an issue recently pointed out by Paul Dunkler: some of the plugins did not terminate with the correct return code when reaching a warning or critical threshold.

The check_memory plugin no more reports as cached memory the unreclaimable slab values, which cannot be reclaimed even under memory pressure.

The check_cpu plugin executed with the ‘-i | —cpuinfo‘ switch, now correctly detect on 64-bit architectures the CPU 64-bit op-mode.

A minor memory resource leak reported by the Coverity Scan tool has also been fixed.

You can download the source code (.xz compressed tarball) here and visit the GitHub project web page for more information.

As usual, bug reports, feature requests, and ideas for improvements are welcome!

Gradient boosting ensemble technique for regression

Gradient boosting is a machine learning technique for regression and classification  problems, which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees. It builds the model in a stage-wise fashion like other boosting methods do, and it generalizes them by allowing optimization of an arbitrary differentiable loss function. (source: Wikipedia)

This is a great video tutorial from Alexander Ihler, Associate Professor at Information & Computer Science, UC Irvine.

You can found other interesting data science tutorials made by Alexander Ihler in this YouTube channel:

https://www.youtube.com/user/atihler