Archive for the ‘ksplice blog’ tag

Strace: The Lost Chapter


I wrote another post last week for the Ksplice blog: Strace — The Sysadmin’s Microscope. If you’re running a Linux system, or just writing or maintaining a complex program, sometimes strace is indispensable — it’s the tool that tells you what a program is really doing. My post explains why strace is so good at showing the interesting events in a program (hint: it gets to sit between the program and everything else in the universe), describes some of its key options, and shows a few ways you can use it to solve problems.

Unfortunately there’s only so much you can say in a blog post of reasonable length, so I had to cut some of my favorite uses down to bullet points. Here’s one such use, which I can’t bear to keep off of the Web, just because I thought I was so clever when I came up with it in real life a couple of months ago.

(If you haven’t already, I encourage you to go read the main post first. I’ll be here when you come back.)

Strace As A Progress Bar

Sometimes you start a command, and it turns out to take forever. It’s been three hours, and you don’t know if it’s going to be another three hours, or ten minutes, or a day.

This is what progress bars were invented for. But you didn’t know this command was going to need a progress bar when you started it.

Strace to the rescue. What work is your program doing? If it’s touching anything in the filesystem while it works, or anything on the network, then strace will tell you exactly what it’s up to. And in a lot of cases, you can deduce how far into its job it’s gotten.

For example, suppose our program is walking a big directory tree and doing something slow. Let’s simulate that with a synthetic directory tree and a find that just sleeps for each directory:

  $ mkdir tree && cd tree
  $ for i in $(seq 1000); do mktemp -d -p .; done >/dev/null
  $ find . -exec sleep 1 \;

Well, this is taking a while. Let’s open up another terminal and ask strace what’s going on:

  $ pgrep find
  $ strace -p 2714
  fstat(5, {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
  fchdir(5)                               = 0
  getdents(5, /* 2 entries */, 4096)      = 48
  getdents(5, /* 0 entries */, 4096)      = 0
  close(5)                                = 0
  fstat(5, {st_mode=S_IFDIR|0755, st_size=36864, ...}) = 0
  fchdir(5)                               = 0
  close(5)                                = 0
  newfstatat(AT_FDCWD, "tmp.MiHDWiBURu", {st_mode=02, st_size=17592186044416, ...}, AT_SYMLINK_NOFOLLOW) = 0
  clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb19c92a770) = 13044
  wait4(13044, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 13044
  --- SIGCHLD (Child exited) @ 0 (0) ---
  fstat(5, {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
  fchdir(5)                               = 0

The find just looked at tmp.HvbzfbbWSa, and now it’s going into tmp.MiHDWiBURu. How far is that into the total? ls will tell us the list of directories that the find is working from; we just have to tell it to give them to us in the raw, unsorted order that the parent directory lists them in, with the -U flag. And then grep -n will tell us where in that list the entry tmp.HvbzfbbWSa appears:

  $ ls -U | grep -n tmp.HvbzfbbWSa
  $ ls -U | grep -n tmp.MiHDWiBURu
  $ ls -U | wc -l

So tmp.HvbzfbbWSa is entry 258 out of 1000 entries in this directory — we’re 25.8% of the way there. If it’s been four minutes so far, then we should expect about twelve more minutes to go.

(But With The Benefit Of Foresight…)

I’d be remiss if I taught you this hackish approach without mentioning that if you realize you want a progress bar before you start the command, you can do it much better — after all, the ‘progress bar’ above doesn’t even have a bar, except in your head.

Check out pv, the pipe viewer. In my little example, you’d have the command itself print out where it is, like so:

  $ find . -exec sh -c 'echo $1 && sleep 1' -- \{\} \;

and then you could get a real, live, automatically-updated progress bar, like so:

  $ find . -exec sh -c 'echo $1 && sleep 1' -- \{\} \; \
     | pv --line-mode --size=$(find . | wc -l) >/dev/null
   175 0:02:57 [0.987/s ] [=====>                               ] 17% ETA 0:13:55

Here we’ve passed --line-mode to make pv count lines instead of its default of bytes, and --size with an argument to tell it how many lines to expect in total. Even if you can’t estimate the size, pv will cheerfully tell you how far you’ve gone, how long it’s been, and how fast it’s moving right now, which can still be handy. pv is a pretty versatile tool in its own right — explaining all the ways to use it could be another whole blog post. But the pv man page is a good start.

That’s Just One

There’s lots of other ways to use strace — starting with the two I described in my main post, and the three more, besides this one, that I only mentioned there. I don’t really know anymore how I used to manage without it.

Liked this post? Subscribe and keep ‘em coming.

Written by Greg Price

August 16th, 2010 at 12:50 am

Ksplice and the intern army meet the Internet

leave a comment

This week I wrote a post on the Ksplice blog, our first substantive post, following an intro post by Waseem. As I mentioned last month, we swelled from 8 to 20 people this January with interns, and were triumphant in making the whole scheme work productively. If you want to know how we did it, read the post. In fact, just go read it. I’ll wait.

The crackerjack Ksplice PR team (*) got my post to show prominently all day Wednesday on Reddit and Hacker News, and then it went up on Slashdot all Wednesday evening and Thursday during the day. Traffic numbers were much, much more than anything else I’ve ever written, except YouTomb.

Naturally, we learned some things about interacting with your average comment-leaving reader on the Internet. The first wave of comments, a few both on link aggregators and on the post itself, were vicious denunciations of us for the (apparently) illegal practice of employing unpaid interns to do real work. These commenters were of course wrong—you can’t get any intern in software for free, let alone the kind of people we wanted, and we paid as much or more than they could make with their skills in research jobs on campus. I clarified that, I and others replied, and the comments shifted to mostly positive. Then when we landed on Slashdot, the text was a classic opposite-of-the-article Slashdot item: we had claimed to “bust” Fred Brooks’ pioneering observations on software project management. Dozens of commenters poured in to grouch that we hadn’t disproved his law, only sidestepped it—which was of course our point.

Fortunately, not all commenters are just being wrong. We had several good comments, but this afternoon came one last comment from a source far beyond any response I imagined. I feel a twinge of regret now for comparing the OS/360 project to Windows Vista, apt though it was. Prof. Brooks, of couse, did far better than the Vista managers in the end, in that he learned lessons from the experience and put them in a book that the whole profession learned from.

How we’re going to top that comment in our next post, I don’t know—it might be tough, for example, to get a comment from a man who hasn’t used email since before blogging was invented.

(*) Namely, us and our friends on zephyr/twitter lending a few upvotes to our posts. Several others at Ksplice made substantial comments and edits before the post was published, too, which greatly improved it.

[Update, 2010-03-18: there is now a straight-up newspaper-style article about... the comment threads on my post. The Internet never ceases to amaze me.]

Written by Greg Price

March 15th, 2010 at 3:23 am