Shared memory vs. files

Today is my sister's birthday. Happy birthday, Lora!

It is also the day Ewelina's sister and sister's boyfriend arrive to visit us for the next 10 days. Welcome, Pedro and Asia!

But that's not what I want to write about. Before getting back to my PageKite hacking, I want to take a moment to document the extreme geekery my dad and I got up to on Christmas Eve...

Using Linux Files As Shared Memory

My dad writes C code for Linux machines, as part of his work as a scientist for the Icelandic met office. We were discussing one of his coding projects on Christmas Eve, and he was telling me how he considered shared memory to be generally a bad idea and he would rather just use files in /dev/shm for simple inter-process (producer and consumer) communication.

I was sceptical about whether some of his assumptions about the kernel's behavior would hold true - he was pretty sure that for any block in any file, the kernel would at any given time either consider a single region in RAM or the physical disk sectors to be authoritative, but never both.

If this assumption holds true, then processes can easily communicate with each other by simply opening a file and reading/writing non-overlapping regions. This doesn't solve synchronization issues, so code must be written in such a way that reading a partial update isn't a problem, but that isn't necessarily too hard.

After discussing back and forth some ways this could break horribly and mangle his projects, I decided to write a simple test program:

/* writetest.c */
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

void main(int argc, char **argv) {
  char *filename = argv[1];
  unsigned char c;
  int i, fd, offset;

  /* Start 16 forks */
  for (offset = 0; offset < 15; offset++) {
    if (!fork()) break;
  offset *= 0x100;

  /* Write sequences of 0..255 to a 256 byte block */
  fd = open(filename, O_CREAT | O_RDWR, S_IRWXU);
  for (c = 0x00; c < 0xef; c++) {
    for (i = 0; i < 0x100; i++) {
      lseek(fd, i+offset, SEEK_SET);
      write(fd, &c, 1);

The program creates 16 processes, which all open the same file and in a loop read/write non-overlapping regions in the file with predictable data. If my dad's assumptions hold true, the file should end up containing nothing but the character 0xee, otherwise the file contents should be random garbage due to partially written regions overwriting each other.

We compiled, ran and evaluated the results with:

$ gcc -o writetest writetest.c
$ ./writetest foo.bin
$ hexdump foo.bin

At first, it seemed I was right and the file was full of crap. However, it turns out my initial test program had off by one errors in every single loop, and ultimately suffered from an infinite loop due to the fact that an unsigned char can never exceed the value 0xff. Since the loop never ended, of course the file's contents never became stable... programming is hard. :-P

However, once I fixed all my bugs, the test program performed as predicted by my dad: foo.bin was full of 0xees! This worked even for non-block-aligned region sizes. At least on my Linux laptop, it seems normal files are perfectly useable for simple IPC between processes.

Pretty cool!

Lots of questions remain unanswered though.

Is this behavior guaranteed? I certainly wouldn't expect it to work well over NFS, although even that might work. And what is SysV shared memory for? Is it just an outdated relic of a time when kernels didn't have efficient caching? And what about /dev/shm? Is it also pointless? Thanks to the kernel's block cache, frequently accessed normal files are just as fast as ones written to /dev/shm, so why bother wasting RAM which the kernel could otherwise manage more efficiently?

Exploring these and other questions was prevented by Christmas dinner. :-)

Another experiment: comments for this post have been out-sourced to Hacker News. Will anyone comment?

Tags: life, tech

Recent posts