Arxiu de la categoria: english

Notes from FOSDEM

Going to FOSDEM has always been a mix of feelings: is that time of year when you meet many friends from the Free Software community, you learn some interesting things that you didn’t know about, you share some knowledge, and you may have a fair amount of chocolate and beer in a usually cold weather.

Sometimes talks are not what they seem, and oftentimes you can’t get into a room because it’s full. But there’s always the chance to learn something new, so here’s my list of notes:

  • Play etcd if you want to try it and see what happens when you make changes.
  • Minikube: mini Kubernetes for developing on your laptop.
  • Software Heritage API is publicly available.
  • OpsTheater offers a stack for IaaS with Puppet, Foreman, GitLab, Icinga, ELK+Graphana, Mattermost (integrates easily with GitLab).
  • Recommendation: move things from Hiera to Foreman smart parameters. Debugging Hiera can be a nightmare if you have hundreds of YAML files.
  • octocatalog-diff compares two Puppet catalogs without deploying the changes. Facts are not live and changes in providers won’t show. A Foreman plugin is available too as a proof of concept.
  • Puppeteer helps find configuration smells that violate recommended best practices.
  • Legacy docs are big, comprehensive, and feature based. Modular docs are lean, concise, targeted, and user-story based. Content rot makes docs hard to find and navigate. Document only what users need, as user stories.
  • Perl6 grammars make it easy to implement informal DSLs. Reading recommendation: Domain Specific Languages, by Martin Fowler (2010).

Happy hacking!

Config Management Camp

This was my first time at the Config Management Camp in Gent and I had a great time and you’ll see from my notes below that it was definitely worth it.

Day 1

  • Recommendation: use find-nodes from PuppetDB with i parallel SSH (pssh).
  • Services resiliance depends on human resiliance (HumanOps).
  • tiny puppet installs applications on any OS (slides):
    • e.g. tp install puppetdb
    • tinydata is the default source for application data.
  • Vox Pupuli maintain abandoned puppet modules.
  • Reading recommendation: Thinking in Systems: A Primer, by Donella Meadows.
  • Puppet extensions:
    • Ruby functions can take lambda arguments.
    • dalen-puppetdbquery query_resources function to find other nodes resources.
    • Puppet faces allows new puppet subcommands (dalen-puppetls).
  • Foreman unknown gems (slides):
    • Foreman hooks plugin.
    • Trends show changes over time.
    • Bookmark searches. Puppet can ask the search API for information with puppet-foreman module.
    • Class import has rules to hide things in the UI.
    • hammer ssh -c ‘uptime’ -s ‘architecture=…’
    • foreman-rake hosts:scan_out_of_sync
    • There are several Foreman UI themes.
    • API docs are available in your Foreman instance as http://foreman/apidoc
  • Types and providers:
    • require “wirble” in ~/.irbrc
    • Pro tip: use Puppet types and providers for managing web APIs.

Day 2

  • Inspiring story by Annie Hedgpeth, My Journey Into Technology Through Inspec (video).
  • Getting data to the end user:
    • Memex maps the Dark Web.
    • NASA beards like GitHub, sysadmins don’t.
    • juju allows users choose their applications, configure and scale them.
  • Someone mentioned that libral (a native Resource Abstraction Layer) seemed interesting.
  • Quality automation with rudder-dev (slides)
  • undef: refactoring old puppet code (slides)
    • Puppet 3.x is EOL.
    • Hiera overload, bloated YAML. Clean it up!
    • Lack of validation/CI:
      • Syntax error should not be deployable.
      • Fix style with puppet-lint -f
      • rspec-puppet to test special cases.
      • Beaker or Test Kitchen for acceptance tests.
    • VCS top notch:
      • Make it easy as possible to avoid mistakes.
      • Put full context in the commit message.
      • Use the body to explain what and why, not how.
      • Commit often, perfect later, publish once.
      • The git pickaxe shows you how to find any text in the commits.
      • GitMagic helps setting contribution guidelines.
    • Make newbie experience better:
      • Start with control-repo.
      • Pick supported forge modules, then pick approved ones.
      • puppet module skeleton
      • Write as little as possible.

Hope you find the notes useful. Let me know if you have any questions.

 

Packaging SoftiWARP kernel module for Debian

One of the HPC clusters we have at work has a mixed set of nodes: a few of them have InfiniBand interfaces and the others don’t. A few weeks ago we were requested to install the SoftiWARP kernel module on those nodes that lack the InfiniBand interface. We had already tried to build the module from source and it worked well, but now the challenge was to install it as a Debian package with DKMS, so it would be built for all installed kernel versions on each node.

We use Puppet to manage cluster node configuration, so you may wonder why not using Puppet instead. Well, for one we’re talking about installing the source of a kernel module, with no configuration at all. But there’s also the fact that Puppet delegates to package providers the management of software, which know much better how to deal with software upgrades. Lastly, there’s the challenge of learning something new: though I had previous knowlegde of DKMS, I had no idea on how to make a Debian package out of it.

Fortunately, I found Evgeny Golov‘s DKMS playground on Debian wiki. With those tips and my recently updated experience on packaging Perl modules for Debian, I was confident enough to try my first DKMS Debian package. Actually, it came out quite easy: I just had to adapt debian/rules a bit to accommodate modern debhelper best practices:

#!/usr/bin/make -f
pdkms:=siw-dkms
sname:=siw
sversion:=$(shell dpkg-parsechangelog|grep "^Version:"|cut -d" " -f2|rev|cut -d- -f2-|rev|cut -d':' -f2)
 
%:
    dh $@
 
override_dh_auto_install:
    dh_installdirs -p$(pdkms)  usr/src/$(sname)-$(sversion)
    cp -a *.txt Makefile *.c *.h debian/$(pdkms)/usr/src/$(sname)-$(sversion)
    sed "s/__VERSION__/$(sversion)/g" debian/dkms.conf.in > debian/$(pdkms)/usr/src/$(sname)-$(sversion)/dkms.conf

Funny enough, I spend more time filling the details on debian/copyright and debian/control files than actually setting up DKMS, so big kudos to Evgeni!

Take a look at the full debian packaging for further details. You may notice that this package has a dependency on libsiw-dev (SoftiWARP userland library), which I had to package first and was a bit trickier. More on that next time.

Please mount Volume or label a new one?

A Monday morning we found several messages like this from Bacula:

09-Sep 18:27 Tape-A JobId 15196: Please mount Volume "A0000038" or label a new one for:
    Job:          sites.2012-09-08_01.50.03_40
    Storage:      "LTO-4" (/dev/nst0)
    Pool:         Default
    Media type:   LTO-4

That was Bacula complaining about the wrong volume on the drive. However, after examining the status of storage the weird thing was that the tape with barcode “A0000038” was actually mounted on the drive, it was purged and had 0 bytes used… WTF?

We started suspecting our tape rotation algorithm was not taking into account that maybe purged tapes were not used if data retention hadn’t expired, but Automatic Volume Recycling says otherwise and some tests on other purged tapes with expired data retention failed too.

Several days of searching similar issues later, it seemed clear that there was something wrong only with some purged tapes from the media catalog. So i came up with a quick test for each one of our purged tapes in order to make sure that they were labeled correctly. First, i loaded “A0000038” and read its label:

# mtx -f /dev/sg4 load 22
Loading media from Storage Element 22 into drive 0...done
# dd if=/dev/nst0 bs=65000 count=1 | od -a | head
0+0 records in
0+0 records out
0 bytes (0 B) copied, 3.81024 s, 0.0 kB/s
0000000

You can see that it failed to find any Bacula label, so next i tried the same test with another tape in full state that had been used the day before:

# mtx -f /dev/sg4 load 16
Loading media from Storage Element 16 into drive 0...done
# dd if=/dev/nst0 bs=65000 count=1 | od -a | head
0+1 records in
0+1 records out
64512 bytes (65 kB) copied, 3.73563 s, 17.3 kB/s
0000000   G   ;   =   < nul nul nul   D nul nul nul nul   B   B   0   2
0000020 nul nul stx  gs   L etx   O sub del del del   ~ nul nul nul nul
0000040 nul nul nul 240   B   a   c   u   l   a       1   .   0       i
0000060   m   m   o   r   t   a   l  nl nul nul nul nul  vt nul eot dc3
0000100   1   8   O  si  ff nul eot dc3   J   +   W stx   p nul nul nul
0000120 nul nul nul nul nul nul nul nul nul nul nul nul nul   A   0   0
0000140   0   0   0   0   6 nul nul   D   e   f   a   u   l   t nul   B
0000160   a   c   k   u   p nul   L   T   O   -   4 nul   v   m   -   h
0000200   o   s   t   -   4 nul   T   a   p   e   -   A nul   V   e   r
0000220   .       5   .   0   .   1       2   4       F   e   b   r   u

(Note that device /dev/nst0 has a minimum block size of 64512 bytes, otherwise dd would raise a Cannot allocate memory error.)

This time the Bacula label was found on the tape (the Bacula 1.0 immortal string is a clue), so we were now on the right track. Then i reviewed our wiki for the tape change procedure and there was an optional step in the tape purging section explaining how to blank a tape from another Bacula director before labeling it for this one. Therefore, it seemed that someone had been blanking the tapes after purging them and Bacula didn’t know what to make with them even though they had the right barcodes.

The fix was quick: i tested every purged tape to make a list of the blanked ones, then deleted them from the catalog and labeled them back from the bconsole:

* delete volume="A0000038"
* label barcodes slots=22 pool=Default

Bacula is happy again, and we are too! 🙂