CK knows Wayne

Supercalifragilisticexpialidocious

Published at by Christian Kruse, updated at
Filed under: death metal, mary poppins

This is genius!

If Mary would sing this way I would like it, too ;)

0 comments

Migrating a file server with zfs

Published at by Christian Kruse, updated at
Filed under: migration, server, zfs

At the company I’m working for we have a central NAS. This NAS contains all user homes, backups and such. It is a small box with four HDDs running a FreeBSD on a ZFS RAID-Z2. Recently we switched to newer hardware. Transferring the data traditionally takes 13 hours - a big time frame. I can’t shut down the server for 13 hours, working hours are from 07:00 o’clock to 21:00 o’clock. Gladly we use ZFS! I created a snapshot and sent it to the new server:

zfs snapshot zroot/var/storage@base
zfs send zroot/var/storage@base | ssh 10.0.0.24 zfs receive zroot/var/storage

This took round about 13 hours. But during that time the users could work normally, due to the snapshot the changes did not interfere with the zfs send. When it was finished I waited until 21:00, created another snapshot and sent only the difference to the new server:

zfs snapshot zroot/var/storage@latest
zfs send -i base zroot/var/storage@latest | ssh 10.0.0.24 zfs receive -F zroot/var/storage

After that I shut down the old server, set the IP on the new server to the IP of the old one and voila - everything works as expected! The actual switch took me just 10 minutes.

ZFS really rocks :)

0 comments

Automatic configuration deployment with cdist

Published at by Christian Kruse, updated at
Filed under: cdist, server

In my job as „the IT guy“ at a company in Ettlingen I am also responsible for the company network and the configuration and maintenance of the server infrastructure. In the past I did all this by hand: I configured every service directly on the server. Every configuration change I did with a text editor directly on the server.

While with this approach one is able to react very fast on problems, on the other hand it is error-prone. You can’t test changes, you don’t have a documentation. And if you have a documentation, it regularly lacks of the latest changes.

If you ever have to migrate to another hardware, you have to copy the right configuration files to the right places and change the right things to adopt it to the new environment. You also have to install the right software dependencies of the software running on the server. In an emergency case a really bad starting situation.

When I recently replaced some old boxes by newer ones to avoid hardware failure, I took the chance and decided to invest some time to automate configuration of the new boxes. The world of automated configuration deployment is rather small. There are just a few solutions, the most often mentioned are Puppet, Chef as well as CFEngine. For my purposes (a rather small company network) the three are too big, too complex. I was looking for something less „magical,“ something following the KISS principle. After a lot of research some friends of mine suggested cdist. cdist has been written by Nico Schottelius, a FOSS hacker and system administrator living in Zurich. It is free software (licensed as GLPv3). cdist configuration deployment consists basically of executing shell scripts which copy or modify files and install packages. Pretty easy, for example this file adds the PostgreSQL repository for Debian to the sources.list:

__block sources-list \
        --state present \
        --file /etc/apt/sources.list \
        --text - <<EOF
deb http://apt.postgresql.org/pub/repos/apt/ wheezy-pgdg main
EOF

The __block call is a function defined by cdist, it calls this kind of construct types. Everything you do is calling these types to manipulate, copy, link and move files.

A cdist configurations entry point is the manifest/init file. In this entry point file you specify what to do, an easy example:

__package apache2 --state present

This would install the apache2 webserver packet on the target host.

Configuration deployment by service types

One of the requirements I had was the ability to quickly move a service to another machine, for example because of a hardware failure. So my first approach was to define service functions and define via environment variable the type of service I’d like to deploy. This way I could for example call

SERVICES="web mail" cdist config -c ./conf/ 10.0.0.2

and deploy the webserver configuration and its dependencies as well as the mailserver configuration and its dependencies to the node 10.0.0.2. But this way I’ve got the problem again that it is not documented when a service moves to another host, and it is error-prone as well: what if I type a service name wrong or add a service name which doesn’t belong to this host?

Configuration deployment by host definitions

So I went to a different type of configuration deployment: host definitions. I define in an external file hosts and specify (among some other things) the services to deploy:

SERVICES_10002="web mail"

When I move a service to another host I change this file and make a new git commit; this way every change is documented via git history. It also is less error-prone, since I don’t have to specify the services again and again on the command line - I’m sure one day I would forget a service or add a service to the wrong box. And due to this structure my manifest/init file is pretty easy:

for file in conf/manifest/*.sh; do
    . $file
done

This part reads every .sh file containing the modules for the services.

__file /etc/cdist-configured
__cdistmarker

HOSTCONFIG="$(getmap "SERVICES_" "$__target_host")"
HOSTNAME="$(getmap "HOSTNAME_" "$__target_host")"

Since Bash below v4 doesn’t support dictionaries, we have to do a little trick to get the configuration for the host. We call a self-defined function getmap, this function basically we constructs a string of the form <first arg><second arg>, gets the value of the variable with that name and returns it to the caller. Thus we read the modules to activate and the hostname we should set.

MY_OS="$(cat "$__global/explorer/os")"
if [ "$MY_OS" = "freebsd" ]; then
    base_freebsd
else
    base_linux
fi

This snippet calls a function which does the basic configuration of the system (hostname, time zone, etc, pp) depending on the target OS.

fun="base_$HOSTNAME"
fn_exists "$fun" && eval "$fun"

This snippet checks if a host-specific configuration function exists, for example to install the RAID monitoring utilities on the right host. If it exists, the function gets called.

for feature in $HOSTCONFIG; do
    if [ fn_exists "$feature" ];
        eval "$feature"
    else
        echo "$feature does not exist!"
        exit 1
    fi
done

This snippet loops to each service defined by the host configuration and calls the function configuring the service. That’s it, now we have a host configuration based configuration deployment. Of course, the functions which deploy the actual services still are missing ;-)

Deploying a service configuration

I moved the configuration of a specific service to a function. This way I get a basic modularization; for example the function webserver installs all software requirements and the configuration for the software on the target host. I will try to explain some things with the configuration function for the web server:

webserver() {
    __user ssh_data_in \
           --state present \
           --create-home
    __user git \
           --state present \
           --create-home

This snippet ensures that the users ssh_data_in and git exist, with created home directories.

    require="__user/ssh_data_in" __directory /home/ssh_data_in/.ssh/ \
                --owner ssh_data_in \
                --group ssh_data_in \
                --mode 0700
    require="__directory/home/ssh_data_in/.ssh/" __file /home/ssh_data_in/.ssh/authorized_keys \
           --state present \
           --owner ssh_data_in \
           --group ssh_data_in \
           --source - <<EOF
ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIB+bxdCSP8zo04Xmf+ItBlJtgXWhbBP69HetzSj/IH18ugnKrflylDL0SaCAizjKJKZnjnkW0RNqKEHQzMsMSOs8bSfu9L2W5Qg3peJ4T4BZy3Q7cfYNOYkMM6vYl3pv2coJvDlnUc2/BY95NlVHYhW4YtHYPjBTwB3a4eaotl4RQ==
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDosf/bRU/avCgY47OgIY3thRM5cLejAicnghRrjeqzfCR8WGvkaJeKxGc9B+4O4DX3/kp3USOZNzD45Uk6s7LYPRTMWoY+CPgiyuC2fNYmON+lpsnl3ebmY0wyigOIH6HiKf3GZlsnp6E1efxbLlgNBIbyhp3MSs6LeNok+/3090LNdEOJvwOpEHYveJ/qCeHT2ipbzToZS5LGMJ192xqYik/4Ov8drg3WHhusYw1HJl/CM/jE3XPZScCaCGmvGvetsa4nJiy2PBn7Qnp3AYckHw3RyyGShmJb/IfeXI8ypKTqtzmVOZ4RnZk+OS0q/5/o/9pJBg6TB9OuJgBhah+V ckruse@vali.local
EOF

The feature I use in this block is a dependency. cdist does not execute the types/functions in the order I call them, but it may generate some remote code and executes it later. With the special viarable require you can specify dependencies; in this case I want to ensure that the directory /home/ssh_data_in/.ssh/ exists and belongs to the user and group ssh_data_in - but after the user has been created. When the directory has been created we create a file authorized_keys containing two public keys.

    __package libapache2-mod-xsendfile \
              --state present
    __package rsync \
              --state present
    __package syslog-ng \
              --state present
    __package build-essential \
              --state present
    __package ruby1.9.3 \
              --state present
    __package ruby-dev \
              --state present
    __package curl \
              --state present
    __package libcurl4-gnutls-dev \
              --state present
    __package libxml2-dev \
              --state present
    require="__package_update_index" __package libpq-dev \
              --state present
    require="__package_update_index" __package nodejs \
              --state present
    __package php-mail \
              --state present
    __package php-mail-mime \
              --state present
    require="__package/nodejs __package/curl" __install_npm
    __package apache2 \
              --state present
    require="__package/apache2 __package/ruby1.9.3" __package libapache2-mod-passenger \
           --state present

    __package php5 \
              --state present
    require="__package/apache2 __package/php5" __package libapache2-mod-php5 \
           --state present

    require="__package/php5" __package phpldapadmin \
           --state present
    require="__package/php5" __package phppgadmin \
           --state present
    require="__package/php5" __package phpmyadmin \
           --state present

    __package pdftk --state present
    __package git --state present
    require="__package/ruby1.9.3" __package_rubygem bundler --state present

This snippet installs a lot of software packets we need for the software running on the web server. __install_npm is a self-written function installing NPM via node.js.

    require_for_restart="__package/apache2 __package/libapache2-mod-passenger __package/libapache2-mod-php5 __package/libapache2-mod-xsendfile"

    for site in ./files/webserver/apache/sites/*; do
        site_name="`basename $site`"
        doc_root="`grep DocumentRoot $site | sed 's/DocumentRoot//' | tr -d ' \t'`"
        require_for_restart="$require_for_restart __file/etc/apache2/sites-available/$site_name __directory/$doc_root"

        require="__package/apache2" __file /etc/apache2/sites-available/$site_name \
               --source $site \
               --state present

        # if it is a ruby thing we use my user as owner
        if [[ $doc_root == *public* ]]; then
            __directory $doc_root \
                        --state present \
                        --owner ckruse \
                        --group ckruse \
                        --parents
            __directory ${doc_root%/}/.. \
                        --state present \
                        --owner ckruse \
                        --group ckruse \
                        --parents
        else
            __directory $doc_root \
                        --state present \
                        --owner www-data \
                        --group www-data \
                        --parents
            __directory ${doc_root%/}/.. \
                        --state present \
                        --owner www-data \
                        --group www-data \
                        --parents
        fi

        if [ "$site_name" == "default" ]; then
            require_for_restart="$require_for_restart __link/etc/apache2/sites-enabled/000-$site_name"
            require="__file/etc/apache2/sites-available/$site_name" __link /etc/apache2/sites-enabled/000-$site_name \
                   --source ../sites-available/$site_name \
                   --type symbolic
        else
            require_for_restart="$require_for_restart __link/etc/apache2/sites-enabled/$site_name"
            require="__file/etc/apache2/sites-available/$site_name" __link /etc/apache2/sites-enabled/$site_name \
                   --source ../sites-available/$site_name \
                   --type symbolic \
                   --state present
        fi
    done

    require="__package/apache2" __file /etc/apache2/sites-available/default-ssl \
           --state absent
    require="__package/apache2" __link /etc/apache2/sites-enabled/default-ssl \
           --source ../sites-available/default-ssl \
           --type symbolic \
           --state absent
    require="__package/apache2" __link /etc/apache2/mods-enabled/ssl \
           --source ../mods-available/ssl \
           --type symbolic \
           --state absent
    require="__package/apache2" __link /etc/apache2/mods-enabled/expires.load \
           --source ../mods-available/expires.load \
           --type symbolic \
           --state present
    require="__package/apache2" __link /etc/apache2/mods-enabled/headers.load \
           --source ../mods-available/headers.load \
           --type symbolic \
           --state present
    require="__package/apache2" __link /etc/apache2/mods-enabled/xsendfile.load \
           --source ../mods-available/xsendfile.load \
           --type symbolic \
           --state present
    require="__package/apache2" __file /etc/apache2/mods-available/xsendfile.conf \
           --state present \
           --source <<EOF
XSendFile On
XSendFilePath /mnt/storage/noodles/
EOF
    require="__file/etc/apache2/mods-available/xsendfile.conf" __link /etc/apache2/mods-enabled/xsendfile.conf \
           --source ../mods-available/xsendfile.conf \
           --type symbolic \
           --state present

    require_for_restart="$require_for_restart __file/etc/apache2/sites-available/default-ssl __link/etc/apache2/sites-enabled/default-ssl __link/etc/apache2/mods-enabled/ssl __link/etc/apache2/mods-enabled/xsendfile.conf"

    for to_enable in passenger.load passenger.conf rewrite.load; do
        require_for_restart="$require_for_restart __link/etc/apache2/mods-enabled/$to_enable"
        require="__package/apache2" __link /etc/apache2/mods-enabled/$to_enable \
               --source ../mods-available/$to_enable \
               --type symbolic \
               --state present
    done

    require_for_restart="$require_for_restart __block/apache2-charset"
    require="__package/apache2" __block apache2-charset \
            --state present \
            --file /etc/apache2/conf.d/charset \
            --text - <<EOF
AddDefaultCharset UTF-8
EOF

    require_for_restart="$require_for_restart __remove/etc/apache2/conf.d/phpldapadmin"
    require="__package/phpldapadmin" __remove /etc/apache2/conf.d/phpldapadmin \
           --pattern 'Alias /phpldapadmin'

    require_for_restart="$require_for_restart __remove/etc/apache2/conf.d/phppgadmin"
    require="__package/phppgadmin" __remove  /etc/apache2/conf.d/phppgadmin \
           --pattern "Alias /phppgadmin"

    require="$require_for_restart" __service apache2 --action restart
}

This block configures the Apache web server, enables some modules and installs some virtual hosts. It also ensures that some files don’t exist. For example we use a NGINX for SSL termination so we don’t want the SSL module of Apache to be enabled. When all files and packages have been installed, we restart the service. And bam, we have a working web server setup! Of course there is a little bit more in the real deployment function, I shortened it a bit to the essential things.

Conclusion

cdist is a nice and simple tool. The complete company infrastructure now gets configured using this approach. Yay!

2 comments

Blogging with Emacs

Published at by Christian Kruse, updated at
Filed under: blog, emacs

I’m a really lazy guy. I hate everything I have to do more than once and I hate everything I can’t do in my normal work space.

Writing blog posts fulfills both criteria: I have to do several tasks manually for each blog post I compose (e.g. upload the images I reference in the post) and I have to switch to the browser, open the web page of the admin interface. And I have to deal with the crappy text area field for the content. This may sound ridiculous, but I really hate this. It feels like blogging in the nineties.

So I thought how I can make this process easier and more automated. The first thing that came to my mind: why not using Emacs? I already use it all the day, and blogging via a simple key stroke would really be great. I knew about org2blog, but since I use a home-grown blogging software this solution is too wordpress specific. I needed something for my own software and something that uses Markdown: I like Markdown much more for writing.

Picture of a blog post with Emacs

Well, nothing easier than that: I wrote a little Ruby script which takes the posting content, looks for images, uploads them and publishes the post afterwards. Now I can post to my blog via a simple shortcut. Heureka!

7 comments

A Rust adventure

Published at by Christian Kruse, updated at
Filed under: programming, rust

tl;dr: Rust rocks

During the 31C3 I started learning Rust. I wanted to for a long time, but finally I had the time to do so: a compiled, system programming language featuring static type safety, a type system inspired by Haskell’s type classes and easy parallelism with nearly no runtime overhead? Sounds like a dream comes true!

So I started the Rust nightly install (in archlinux this is just a yaourt -S rust-nightly-bin && yaourt -S cargo-nightly-bin), read through the guide and started hacking a small project to get into the language a little bit more: a parallel grep.

First steps were pretty easy, creating new projects and building them is, thanks to cargo, just a command away and the guide is pretty easy to understand. My basic concept was to create a std::sync::Future (Rust’s futures implementation) for each file we have to search through and return the results as an array to the main thread when finished.

The problems began when I tried to find out how to work with the standard library: the API documentation is pretty unusable. Try to find out how to search for a substring in a string or how to open a file. I wasn’t able to find them by myself, I always had to do a web search. This has to get better!

The first „woah!!“ occured when I tried to put a std::sync::Future into a vector. I definitely have to get used to type inference, I tried to use a type annotation for the vector since I didn’t think that the compiler is able to get the type right. Thus I’ve been reading futures.rs to get the type annotation right: Vec<Future<_>>. The underscore seems to tell the type checker that we don’t care about the exact future type, it is just some future. Man, that is really cool stuff! Then (after I spent an hour to get the type right) I tried to simply leave out the type - and it works! This is really amazing.

Another problem occured when I tried to iterate over the vector to gather the results. The default way to iterate over a vector in Rust is using iterators:

for result in results.iter() { let lines = result.get(); for line in lines.iter() { print!("{}", line); } }

results is the Vec<Future<_>>. The code above leads to this error message:

/home/ckruse/dev/ngrep/src/main.rs:48:21: 48:27 error: cannot borrow immutable dereference of `&`-pointer `*result` as mutable
/home/ckruse/dev/ngrep/src/main.rs:48         let lines = result.get();
                                                             ^~~~~~
error: aborting due to previous error
Could not compile `ngrep`.

To learn more, run the command again with --verbose.

This error message is not that helpful for beginners, but after reading the guide again I was able to track down the problem: the vector iterator returns an immutable value, but the .get() method needs the std::sync::Future to be mutable. So after searching the web again I was able to find a solution: I iterate over the array the old way with a counter and call .get_mut() which returns the element mutable:

for i in range(0, results.len()) { let mut result = match results.get_mut(i) { Some(v) => v, None => panic!("error! no value!") }; let lines = result.get(); for line in lines.iter() { print!("{}", line); } }

This works, but is ugly. Maybe a reader knows a better solution?

The rest of the code is pretty straight forward:

use std::os; use std::sync::Future; use std::io::BufferedReader; use std::io::File; fn main() { let args = os::args(); let mut results = vec![]; for file in args.slice(2, args.len()).iter() { let path = Path::new(file); let pattern = args[1].clone(); let delayed_value = Future::spawn(move || -> Vec<String> { let mut retval: Vec<String> = vec![]; let fd = match File::open(&path) { Ok(f) => f, Err(e) => panic!("Couldn't nopen file: {}", e) }; let mut file = BufferedReader::new(fd); let mut lineno : uint = 0; for maybe_line in file.lines() { let line = match maybe_line { Ok(s) => s, Err(e) => panic!("error reading line: {}", e) }; if line.contains(pattern.as_slice()) { let s = path.as_str().unwrap(); retval.push(format!("{}:{}: {}", s, lineno, line)); } lineno += 1; } retval }); results.push(delayed_value); } for i in range(0, results.len()) { let mut result = match results.get_mut(i) { Some(v) => v, None => panic!("error! no value!") }; let lines = result.get(); for line in lines.iter() { print!("{}", line); } } }

Another really cool feature are the valued enums. Let’s have a look at this:

let mut result = match results.get_mut(i) { Some(v) => v, None => panic!("error! no value!") };

The return value of get_mut() is just an enum, but in Rust enums may contain additionally a value. This leads to constructs like above. The match keyword introduces a pattern matching construct and we basically say „when the return value of get_mut() is Some give me the value into the variable v and return it; if it is None simply panic out.“ This rocks! Now we can forget about all this int somefunc(char *real_retval) bullshit from C. We can properly distinguish between error cases and real return values. Yay!

Another interesting point was that I tried to use the pattern variable directly in the closure. The compiler did forbid that with some remarks about the lifetime of the variable and I didn’t understand what exactly it tried to say. After a web search I found this: everything is fine during the first loop. The closure takes over the pattern variable and we can use it. But after the first loop (well, to be exact: after the first closure finished) it would clean up after itself and free the memory. Thus we would access invalid memory, potentially leading to a crash. This has been caught by the compiler! Wow, great! So the solution was to create a copy for each loop. Finished.

All in all I really like that language. The functional elements (immutable variables by default, pattern matching, tuples, etc, pp) are integrated well and fit into the flow. Paralellism is easy and remindes me of Erlang. It is really hard to introduce bugs or crashes, the compiler detects a lot of potential and real problems. Of course there is still work to do (some error messages are hard to understand, the API documentation is really bad), but I could definitely imagine that this will be my first-choice language in future.

11 comments

Facebook laws for idiots

Published at by Christian Kruse, updated at
Filed under: college humor, facebook

I am hereby immune to gonnorhea!

0 comments

Emacs: run rails test at point

Published at by Christian Kruse, updated at
Filed under: emacs, rails

Lately I’ve been writing a lot of tests. To shorten the round trip times and for some comfort I wanted to run tests in Emacs: I don’t need to switch to the console and I can simply hit enter on an error message to get to the right file at the right position.

While projectile-rails supports running rake tasks, rake can only run the complete test suite or single test cases, not just a single test. This increases the round trip time too much for me, so Emacs to the rescue! I wrote some elisp to run a single rails test:

(defun get-current-test-name () (save-excursion (let ((pos) (test-name)) (re-search-backward "test \"\\([^\"]+\\)\" do") (setq test-name (buffer-substring-no-properties (match-beginning 1) (match-end 1))) (concat "test_" (replace-regexp-in-string " " "_" test-name))))) (defun run-test-at-point () (interactive) (let ((root-dir (projectile-project-root))) (compile (format "ruby -Ilib:test -I%s/test %s -n %s" root-dir (expand-file-name (buffer-file-name)) (get-current-test-name)))))

This little snippet runs the test at point, so the test the cursor is currently located in. I bound it to C-cct.

2 comments

Things You Should Never Do

Published at by Christian Kruse, updated at
Filed under: development, mistakes, software

Last night I was reading an article about the worst mistake in software development: a complete rewrite of the companies flagship software.

I can definitely confirm that it is a really bad idea in general to do a complete rewrite: by personal experience because I for myself did that mistake and by second hand experience - I have several friends struggling with the same problem.

On the other hand it can be a very good idea to cut off old code and replace it by new code: have a look at the openssl/libressl. Another example may be the nginx project, which is able to outperform Apache just because the code base is not that crufted.

I can’t tell you what the right way for your project is, but I thought it might be a good idea to share that link from above. It is an interesting read either way.

0 comments

Risiko-Abwägung (German)

Published at by Christian Kruse, updated at
Filed under: media

This!

Risken

0 comments

Trouble at the Koolaid Point

Published at by Christian Kruse, updated at
Filed under: feminism, gamergate

I didn’t want to say something to “gamergate” since I’m not even a real gamer, just a casual. But I find it simply horrible what people are able to do, and death threats and harassment should be punished with jail or a very substantial money fine. This has nothing to do with “free speech,” your freedom ends where the freedom of the other begins.

That said I’d like to point you to a very interesting blog post by Seriouspony. It talks about a phenomena called “the kool-aid point”. It describes that in the beginning people who don’t like you or your ideas ignore you and as soon as you get attention they get nasty, because they think that you don’t deserve this and because they want to get some kind of revenge.

I can totally confirm that theory. I made similar experiences (although not that hard, I never got death treats but I’ve been doxed and harassed).

This is serious, people. It’s not less serious because you do it on the internet. It is even more serious, because people can still read it after years.

0 comments