Systemd journal: What does “systemd-journal: Suppressed N messages from /system.slice/…” mean?

Under pressure, my system was generating lots of messages like this:

# systemctl -l status systemd-journald.service
Jun 24 13:47:23 localhost systemd-journal[155]: Suppressed 15460 messages from /system.slice/...

This means the system is generating lots of messages and journal is configured to drop some of them. This is called rate limit, and is useful to not overload the logging system.

Sometimes, however, you just want to get all messages, or perhaps increase these limits. This can be achieved by setting the variables RateLimitInterval and RateLimitBurst inside the config file /etc/systemd/journald.conf.

Quoting the man page for journal: To turn off any kind of rate limiting, set either value to 0.

Linux: Getting rid of “net_ratelimit: N callbacks suppressed” messages

Linux has a mechanism to avoid a DoS attack – with regard to logging – called rate limit. Every message logged by the kernel (including its modules), with printk(), is checked if it’s allowed to be actually printed through this mechanism.

The limits can be configured by tuning the files /proc/sys/kernel/printk_ratelimit and /proc/sys/kernel/printk_ratelimit_burst. In my machine, the values for these files are 5 and 10, respectively, meaning: It’s allowed 10 messages every 5 seconds. Exceeding this will make the kernel discard the message and print something like “ratelimit N: callbacks suppressed”.

However, the networking code in the kernel has its own limit configuration. They obey the same logic above, they use a different path just to allow independence from the generic logging functions. The files are: /proc/sys/net/core/message_cost and /proc/sys/net/core/message_burst. They are similar to their generic “parents” mentioned above.

The message_cost file contains the interval and message_burst contains the maximum number of messages allowed in that interval.

To disable this mechanism and allow every message to be logged, simply set the interval to 0:

# sysctl -w net.core.message_cost=0

Write “net.core.message_cost=0” to /etc/sysctl.d/some-file to make this change persistent to reboots.

This will make the message “net_ratelimit: N callbacks suppressed” go away. It’s up to you do disable this mechanism. Sometimes it’s just necessary, right?

Recovering lost files after a git reset –hard

This morning I’ve messed things up in my env…

I did a commit yesterday and thought I had pushed it, but not.
Today I did a git pull and was asked to do a merge, because there were changes in the remote.

I totally ignored this (don’t ask me why, perhaps absence of coffee…) and did a git reset –hard HEAD^^

So, yep, I lost my commit that existed only in my machine (this commit in particular was a file addition).

After blaming myself a lot, and a bit of google search, I’ve found this amazing blog post: http://zsoltfabok.com/blog/2012/06/recover-deleted-files-with-git/

I’m summarizing here the steps to have your commit (in my case, the file) back:

git fsck --no-reflog

This command will return a hash for the lost commit, something like

dangling commit fafeade9f348b18f37835ab49dab40172efde693

You can use this hash to do a checkout, for instance, and recover your file. \o/

So, let’s have a bit of coffee now :)

Moving on

[ENGLISH]

Hi guys, I’d like to share with you the news that my family and some friends already know: Since last month I’m not working at Intel anymore.

For those whom don’t know my history, I’ll summarize it here:

  • About 3 years ago I left my home city to the biggest one in Brazil (São Paulo) to experience a full time Linux job
  • The instability of my department at the time led me to look for another job; Then I got that opportunity at Intel
  • Intel OTC office is located in Campinas (120km far away from São Paulo)
  • After more than one year having to travel about 240km every day I had to make a decision: Move to Campinas or leave Intel.

It was not an easy decision. I have a family and I didn’t want to affect them again with a city change. On the other side, Intel is a great company and people at OTC Campinas are amazing.

So, two weeks ago I started working for an IT Security company, working again with Embedded Linux and with Open Source technologies. Oh, 8km (15 min) far from my home :).

That’s it, I hope to blog a bit more now, as I used to do in the past. See you!

 [PORTUGUÊS]

Oi gente, gostaria de compartilhar uma novidade com vocês, que minha família e alguns amigos já sabem: Desde o mês passado que eu não trabalho mais na Intel.

Pra quem não conhece minha história, vou resumir aqui:

  • Cerca de 3 anos atrás eu mudei de Maceió, minha cidade natal pra São Paulo, pra trabalhar exclusivamente com Linux
  • Alguns problemas no meu departamento me levaram a buscar outro trabalho; Aí que entra a Intel
  • O escritório onde fui alocado (OTC, Open Source Technology Center) fica em Campinas, distante 120km de São Paulo
  • Depois de mais de 1 ano viajando cerca de 240km todos os dias eu tinha que tomar uma decisão: Mudar pra Campinas ou sair da Intel

Com certeza não foi uma decisão fácil. Tenho uma família e não queria afetá-los com mais uma mudança de cidade. Por outro lado, a Intel é uma ótima empresa e o pessoal do OTC Campinas é ótimo.

Então… duas semanas atrás comecei a trabalhar para uma empresa de segurança em TI, voltei a trabalhar com Linux embarcado e com Open Source de fato. Ah, e a 8km (15min) de casa :).

É isso, gente. Espero voltar a blogar com mais frequencia, como no passado. Até mais!

Parasite, wake up!

Hi there!

In the last weeks I’ve been working on GTK+ themes, customizing some apps, and I missed the tools we have on the web world that allow us to to some live editing on CSS.

I remembered we had Parasite, a tool similar to Firebug, but for GTK+ apps. Unfortunately it didn’t support CSS tweaking. But it does now! I’ve made some improvements on it mainly to allow live editing of CSS, but not limited to that.

Some changes so far:

  • Live editing of CSS, for the whole application, or only for the selected widget
  • Ability to add/remove a CSS class for a specific widget. (Editing of pseudo-classes like :hover is planned)
  • In the Property inspector, if a property itself is another Object, we can also inspect it. (For example, you can inspect a Buffer inside a TextView)
  • A bit of UI change

I have made a small video showing the improvements:


Link to the video

The code is on the usual place, github: https://github.com/chipx86/gtkparasite

Please, test it and file bugs/wishes/pull requests. /me is now wearing the maintainer hat :)

Note: it requires GTK+ 3.10 (and optionally python+gobject introspection for the python shell)

Joining Intel

Today is my last day at Oi WiFi.

It has been 1 year and a half since I moved from my small city (Maceió) to the biggest, craziest Brazilian city, São Paulo. I don’t regret!

I’m lucky to have joined a great company (Vex at the time. Oi WiFi nowadays), with great people where I learnt a lot. I’m glad for the things I helped to improve, I’m sure we have better products than before and I’m proud to be part of that progress. I leave as legacy the spirit of the Free Software, where we can (and should) contribute back to projects we use and improve internally. Every improvement we made here we submitted back to projects like Openwrt, busybox, glib, etc.
However things and priorities in the company have changed a bit in the last few months. Time to look for a new challenge in my career.

What a challenge!

At Intel I’ll join the OTC – Intel Open Source Technology Center, and will work on Open Source projects such as Tizen, EFL, Webkit and hopefully GTK+ :)
The team I’ll work with is formed by the former Profusion company, acquired by Intel in the beginning of the year. Profusion was a company that I admired even before it was acquired by Intel :)

I’m very excited to join Intel. It’s a great opportunity in a great company and I don’t want to disappoint them!

I hope to publish here very soon the things I’m working on under the Intel umbrella. See you!

Shell script that updates itself

Recently I needed to write a shell script that updates itself, and, surprising, I found it an easy job to do. I will share the recipe here.

In my use case, I’m developing a kind of software updater and, before updating the system packages, I need to check if there is a new version of this software updater. If there is, then I update myself and run my new copy on-the-fly.

Enough talk, show me the code. I’ll paste here a simple shell script that talks by itself:

#!/bin/sh

SCRIPT_NAME="$0"
ARGS="$@"
NEW_FILE="/tmp/blog.sh"
VERSION="1.0"

check_upgrade () {

  # check if there is a new version of this file
  # here, hypothetically we check if a file exists in the disk.
  # it could be an apt/yum check or whatever...
  [ -f "$NEW_FILE" ] && {

    # install a new version of this file or package
    # again, in this example, this is done by just copying the new file
    echo "Found a new version of me, updating myself..."
    cp "$NEW_FILE" "$SCRIPT_NAME"
    rm -f "$NEW_FILE"

    # note that at this point this file was overwritten in the disk
    # now run this very own file, in its new version!
    echo "Running the new version..."
    $SCRIPT_NAME $ARGS

    # now exit this old instance
    exit 0
  }

  echo "I'm VERSION $VERSION, already the latest version."
}

main () {
  # main script stuff
  echo "Hello World! I'm the version $VERSION of the script"
}

check_upgrade
main

To try this script:
1) save it somewhere
2) save a copy of it at /tmp/blog.sh (as pointed at line 5)
3) modify the variable “VERSION” (line 6) of that copy, to, say, “2.0”.
4) run the original script (not the one at /tmp)

You will see that the script updated itself and ran the “new” 2.0 version.

Try running again the original script (step 4 above). See the difference? It doesn’t update itself anymore, because it is the “latest” version.

A small thing you might notice: at line 19, I deleted the “new file”. That’s merely for this educational example, that we check if there’s a new version of the script by just checking if a file exists in the disk. On real life (with apt/yum or any smarter process) this is not needed as our check for a new version (line 13) will naturally fail.

This was tested with bash, dash and busybox’s ash. Worked fine.

I hope it’s useful to someone. Comments are welcome!

g_list_copy_deep and g_slist_copy_deep in GLib

If you have in your code things like that:

copy = g_slist_copy (list);
g_slist_foreach (copy, g_object_ref, NULL);

In GLib 2.34 (to be released soon) you can do:

copy = g_slist_copy_deep (list, g_object_ref, NULL);

Thus eliminating another loop in the list (one loop is done when copying the list).

In other words, there’s now a function to do a deep (full) copy of a GList and GSList. It accepts, besides the list to be cloned, a function as a second argument, and a user data as the third argument. That function will be called for each element in the list.

To free the new list, use g_list_free_full().

Programmatically managing iptables rules in C: IPTC

Sometimes we need to manage iptables rules from inside our own programs. The recommended solution given by iptables developers is to spawn the iptables command with execl() or system(). It’s explicitly stated that there’s no stable/public API to do that: http://www.netfilter.org/documentation/FAQ/netfilter-faq-4.html#ss4.5

However, if you have control of your box, including kernel, iptables, and all related packages, it’s safe to use IPTC to programmatically manage iptables rules.

IPTC is an internal iptables library that actually inserts/removes the rules. It’s used by the iptables (xtables-multi, iptables-save, iptables-restore) binaries. Depending on how iptables was compiled, there might be a shared libiptc.so object and/or a static libiptc.a object.

You should check that in order to correctly build your application. In Debian/Ubuntu, which this post is based, it’s enough to install the iptables-dev package. It will install the shared libraries, the header files and the pkg-config facility.

So, let’s take a look at a complete function that adds a rule to iptables:

#include <stdio.h>
#include <errno.h>
#include <string.h>
#include <libiptc/libiptc.h>

static int
insert_rule (const char *table,
             const char *chain, 
             unsigned int src,
             int inverted_src,
             unsigned int dest,
             int inverted_dst,
             const char *target)
{
  struct
    {
      struct ipt_entry entry;
      struct xt_standard_target target;
    } entry;
  struct xtc_handle *h;
  int ret = 1;

  h = iptc_init (table);
  if (!h)
    {
      fprintf (stderr, "Could not init IPTC library: %s\n", iptc_strerror (errno));
      goto out;
    }

  memset (&entry, 0, sizeof (entry));

  /* target */
  entry.target.target.u.user.target_size = XT_ALIGN (sizeof (struct xt_standard_target));
  strncpy (entry.target.target.u.user.name, target, sizeof (entry.target.target.u.user.name));

  /* entry */
  entry.entry.target_offset = sizeof (struct ipt_entry);
  entry.entry.next_offset = entry.entry.target_offset + entry.target.target.u.user.target_size;
  
  if (src)
    {
      entry.entry.ip.src.s_addr  = src;
      entry.entry.ip.smsk.s_addr = 0xFFFFFFFF;
      if (inverted_src)
        entry.entry.ip.invflags |= IPT_INV_SRCIP;
    }

  if (dest)
    {
      entry.entry.ip.dst.s_addr  = dest;
      entry.entry.ip.dmsk.s_addr = 0xFFFFFFFF;
      if (inverted_dst)
        entry.entry.ip.invflags |= IPT_INV_DSTIP;
    }

  if (!iptc_append_entry (chain, (struct ipt_entry *) &entry, h))
    {
      fprintf (stderr, "Could not insert a rule in iptables (table %s): %s\n", table, iptc_strerror (errno));
      goto out;
    }

  if (!iptc_commit (h))
    {
      fprintf (stderr, "Could not commit changes in iptables (table %s): %s\n", table, iptc_strerror (errno));
      goto out;
    }

  ret = 0;
out:
  if (h)
    iptc_free (h);

  return ret;
}

int main (int argc, char **argv)
{
  unsigned int a, b;

  inet_pton (AF_INET, "1.2.3.4", &a);
  inet_pton (AF_INET, "4.3.2.1", &b);

  insert_rule ("filter",
               "INPUT",
               a,
               0,
               b,
               1,
               "DROP");
  return 0;
}

Note: The code above was compiled and tested against the 1.4.13 version of iptables.
To compile:

cc `pkg-config –cflags –libs libiptc` source-file.c -o test-iptc

So, the code in main() above does the equivalent of:

iptables -t filter -A INPUT -s 1.2.3.4/32 ! -d 4.3.2.1/32 -j DROP

The main functions used were iptc_init, iptc_append_entry and iptc_commit. There are functions to remove a rule (iptc_delete_num_entry), to iterate over all the rules (iptc_first_rule, iptc_next_rule) and to clear all rules (iptc_flush_entries).

You can check the header file libiptc/libiptc.h to see all available functions and their descriptions. I can post some more examples here in the future if there’s demand to it.

Again, remember that there’s no API stability in IPTC, so, it can change in each new version of iptables.

Hope it’s useful to someone. Feel free to ask anything or to report any issues you have. See you!

Linux firewall: Mac filtering – iptables and arptables|Firewall no Linux: filtro por mac – iptables e arptables

[english]

It’s common: you don’t want just filter the access by IP address, but also by the MAC address. The solution can be quite simple: use the MAC module of iptables:

iptables -A INPUT -s 1.2.3.4 -m mac ! –mac-source aa:bb:cc:dd:ee:ff -j DROP


The command above says: Drop every packet with source ip address 1.2.3.4 and its MAC address is NOT aa:bb:cc:dd:ee:ff. Thus you’re forcing the origin of the packet to be the desired one.


However there is a minor issue with that approach: iptables manages packets at the network (IP) layer, which is above the link (MAC) layer. What does that mean? Simple: Once the packet arrives at the network interface, it’s handled by the MAC layer and then it’s forwarded to the IP layer. That’s where iptables filter it. Again, what’s the problem? At this pointer, the ARP table of the kernel was touched, meaning that (using the example above), if a packet with source IP 1.2.3.4 and source MAC 00:00:00:00:00:01 arrives, it will change the ARP entry in the kernel table to point the IP 1.2.3.4 to that wrong MAC address. After that, the packet is forwarded to the above layer, which is then dropped by iptables from go on over the network. Even if the packet is dropped, which is what you want, your ARP table is messed up right now.


That’s where arptables command comes in. It works just like iptables, but it filters packets at MAC layer, preventing the kernel ARP table being touched by wrong packets. Its syntax is very similar to iptables (in fact, it’s a copy of iptables adapted to work in the lower level). The same command (iptables) above could be better implemented with arptables:

arptables -A IN -s 1.2.3.4 ! –source-hw aa:bb:cc:dd:ee:ff -j DROP


You just need this rule, it’s not necessary to have the iptables rules anymore. arptables needs a kernel with support built in, which I guess most distros do, just like they do with iptables.


Take a look at the arptables man page, it has some examples. Its similarity with iptables facilitates our lives.

[português]

É uma situação comum: você não quer apenas filtrar o acesso por endereço IP, mas também pelo endereço MAC. A solução pode ser bastante simples: usar o módulo MAC do iptables:

iptables -A INPUT -s 1.2.3.4 -m mac ! –mac-source aa:bb:cc:dd:ee:ff -j DROP


O comando acima diz: Descarte todos os pacotes com o endereço IP de origem 1.2.3.4 e cujo endereço MAC não é aa:bb:cc:dd:ee:ff. Assim, você está forçando a origem do pacote a ser a desejada.


No entanto, há um pequeno problema com essa abordagem: o iptables gerencia pacotes na camada de rede (IP), que é acima da camada de enlace (MAC). O que significa isso? Simples: Quando o pacote chega na interface da rede, ele é tratado pela camada MAC e depois é encaminhado para a camada IP. É onde iptables o filtra. Novamente, qual é o problema? Neste ponto, a tabela ARP do kernel foi alterada, o que significa que (usando o exemplo acima), se um pacote com o IP origem 1.2.3.4 e MAC 00:00:00:00:00:01 chega, ele vai mudar a entrada na tabela ARP do kernel para apontar o IP 1.2.3.4 para o endereço MAC errado. Depois disso, o pacote é encaminhado para a camada superior, que é então descartado pelo iptables de continuar na rede. Mesmo o pacote sendo descartado, que é o que você deseja, sua tabela ARP está uma bagunça agora.


É aí que o comando arptables entra. Ele funciona exatamente como o iptables, mas filtra os pacotes na camada MAC, evitando que a tabela ARP do kernel seja alterada por pacotes errados. Sua sintaxe é muito semelhante ao iptables (na verdade, é uma cópia do iptables adaptado para trabalhar num nível mais baixo). O mesmo comando (iptables) acima poderia ser melhor implementado com arptables:

arptables -A IN -s 1.2.3.4 ! –source-hw aa:bb:cc:dd:ee:ff -j DROP


Você só precisa dessa regra, não é necessário ter as regras de iptables mais. o arptables precisa que o kernel tenha suporte, o que eu acho que a maioria das distribuições já fazem, assim como eles fazem com o iptables.


Dê uma olhada no man do arptables, lá tem alguns exemplos. Sua semelhança com o iptables facilita nossas vidas.