HDFS namenode won't start after disk full error

If you have trouble restarting the NameNode in a Hadoop cluster after a disk full error, if you don't mind losing some data, you can do the following to get it back up.

Find the 'edits' file in the hadoop dfs.name.dir/current and write this sequence to it :

printf "\xff\xff\xff\xee\xff" > edits

After that, you should be able to start hadoop. Credit here.

Bash - prevent multiple copies of script from running

Since bash commands each spawn its own process, we can't lock files to achieve single copy running semantics. Why? Because file locks are per process and they are automatically cleared when the process dies. Thus it is nonsensical to expect a linux command to lock a file, why, when that command returns, the lock file will be automatically unlocked defeating the purpose of the lock completely!

One easy way to prevent multiple copies running is to find an atomic Linux command that can both do a certain operation and return whether that operation succeeded atomically. This command must fail on the second time. The command to make a directory - mkdir - is one such command.

So the script could try to mkdir a particular directory - let's call this the lock directory. If it fails, we don't start. Now if it works, we must remove the lock directory when the script ends so that the script can run again. We do this using the trap command - trap will make sure a given command will execute when the script exits at any point.

Here is the code:

#!/bin/bash                                                                                                                                                                                        
mkdir /tmp/locka 2>/dev/null || {
    exit
}
trap "rmdir /tmp/locka" EXIT
#script work, the sleep 10 below is to test this
#without having a real script.
sleep 10

Java : write binary data to a mysql out file

I had the need to generate - within Java code - a mysql out file with both text and binary data. The binary data is for some content that has been gzipped and stored as a blob in a mysql table. While it is trivial to write binary data to a blob field directly using JDO, for performance reasons, we had to use the "load infile" approach. Thus the first step was to create an outfile.

Here is the function that would convert binary data to a form that can be written to an out file. It follows the algorithm implemented by mysql for its "SELECT INTO outfile" functionality as described here.

    public static byte[] getEscapedBlob(byte[] blob) {
        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        for (int i=0; i<blob.length; i++) {
            if (blob[i]=='\t' || blob[i]=='\n' || blob[i]=='\\') {
                bos.write('\\');
                bos.write(blob[i]);
            } else if (blob[i] == 0) {
                bos.write('\\');
                bos.write('0');
            } else {
                bos.write(blob[i]);
            }
        }
        return bos.toByteArray();
    }

This is how you would use this function to generate a mysql outfile.

                //gen infile for mysql
                byte[] out = getEscapedBlob(data);
                BufferedOutputStream f = new BufferedOutputStream(new FileOutputStream("/path/to/data.csv")) ;
                String nonBlobFields = "\\N\t10\t20100301\t18\t1102010\t2010-03-01 00:00:00\t";
                byte[] nonBlobData = nonBlobFields.getBytes("UTF-8");
                f.write(nonBlobData, 0, nonBlobData.length);
                f.write(out, 0, out.length);
                f.write('\n');
                f.close();

This writes some integer data followed by the blob data to the outfile, which can then be loaded back using "LOAD INFILE".

Ubuntu : Install packages on a cluster of machines

Sometimes, you have a cluster of machines where some packages need to be installed. It would be nice to be able to automate this so that you could do everything from a single terminal. We have seen how a command can be run on multiple machines from a single terminal before. This only works if you have password-less ssh set up between the machine that you are running the command from and the cluster on which you want the command to actually run. The only aspect that makes this a little harder for installing software is that you need to be root to install packages and ssh keys are not generally set-up for root.

However, there is an option -S that you can provide sudo that will make sudo read the password from stdin. We can use this combined with the bash loop to come up with a one liner that would install a package across a cluster of machines.

for m in m1 m2 m3 m4 ; do echo $m; ssh $m "echo password | sudo -S apt-get -y install curl" ; done

The -S option makes sure that the command will not prompt you for a password or complain about a missing tty. The -y option for apt-get prevents it from prompting you prior to the install.

Mac / Microsoft Excel / newlines (\r \n)

It is a frequently the case that the business department hands over Excel files to the engineering department for some type of data processing. The first step here is to convert this to a proper comma separated text file (csv).

If you are doing this conversion using Microsoft Excel on a Mac, you'll note that the resulting file does not have Unix-style newlines. A Unix new line is the 0x0a character, also written as \n. What Excel produces is the 0x0d character, also written as \r.

Most Linux commands do not recognize \r as a line ending. There are several ways to convert the \r characters to proper Linux style line endings. Using the vi editor is a common method. However, there is also the issue that sometimes if the Excel spreadsheet has blank columns, Excel insists on writing a possibly large number of \r characters at the end of the  csv file. The vi method would write a newline per each of these \r characters and that is not ideal.

Instead, you could use this perl one-liner to accomplish both : turn all \r into \n except for the trailing \r characters :

perl -ne 's/([^\r])\r/$1\n/g; s/\r//g; print;'  imported.csv

The regular expression replaces any non \r character followed by \r with the non \r character followed by a \n. Since the trailing \r characters do not match this pattern, they are thus ignored. The second regexp removes these \r characters.

Linux Shell, HUP and process status on logout

It used to be the case that all processes a user starts are killed by the shell upon logout. Not any more, as recent experiments with Ubuntu 10.04 shows.

The shell can be configured to send a HUP signal to its children when the shell exits. This is controlled by the huponexit shell option as explained in the bash man page:

If the huponexit shell option has been set with shopt, bash sends a SIGHUP to all jobs when an interactive login shell exits.

Determine the setting of huponexit with:

shopt huponexit

If it is "off", then processes started by the user will remain running after logout. This setting makes it easier to start a long running process simply from within the shell, without invoking a screen and without having to wrap the process in nohup.

Here is a discussion on the issue.

However, this setting seems to cause problems for interactive sessions when a new user could start referring to an old user's now invalid processes.

Asynchronous UDP server using Java NIO

UDP is a light-weight protocol as compared to TCP. When the data transmitted is small (in hundreds of bytes), and an occasional loss of data is not critical, UDP can be used to improve throughput of the program.

The native sockets library (C) provides the epoll function - available on Linux 2.6.x kernels - that can be used for both TCP and UDP sockets. In an earlier post, I described a framework that can be used to implement an asynchronous client that connects to multiple servers using TCP. I found several code examples that described how Java NIO can be used for this purpose. It turns out that it is even simpler to write a NIO server for UDP.

I would not recommend writing a UDP server if the request/response cannot be transmitted in a single UDP packet or if a packet has a dependency on an earlier packet. UDP packets can arrive out of order and the headers have no sequence numbers to enable re-ordering. If you want to handle reordering, you will be implementing what TCP provides for this purpose and it is probably a better idea to stick with TCP.

The following program does well when the request/response sticks in a single UDP packet. 512 bytes is generally considered the safe maximum size and the DNS protocol mandates a maximum packet size of 512 when it uses UDP.

public class ASyncUDPSvr {
    static int BUF_SZ = 1024;

    class Con {
        ByteBuffer req;
        ByteBuffer resp;
        SocketAddress sa;

        public Con() {
            req = ByteBuffer.allocate(BUF_SZ);
        }
    }

    static int port = 8340;
    private void process() {
        try {
            Selector selector = Selector.open();
            DatagramChannel channel = DatagramChannel.open();
            InetSocketAddress isa = new InetSocketAddress(port);
            channel.socket().bind(isa);
            channel.configureBlocking(false);
            SelectionKey clientKey = channel.register(selector, SelectionKey.OP_READ);
            clientKey.attach(new Con());
            while (true) {
                try {
                    selector.select();
                    Iterator selectedKeys = selector.selectedKeys().iterator();
                    while (selectedKeys.hasNext()) {
                        try {
                            SelectionKey key = (SelectionKey) selectedKeys.next();
                            selectedKeys.remove();

                            if (!key.isValid()) {
                              continue;
                            }

                            if (key.isReadable()) {
                                read(key);
                                key.interestOps(SelectionKey.OP_WRITE);
                            } else if (key.isWritable()) {
                                write(key);
                                key.interestOps(SelectionKey.OP_READ);
                            }
                        } catch (IOException e) {
                            System.err.println("glitch, continuing... " +(e.getMessage()!=null?e.getMessage():""));
                        }
                    }
                } catch (IOException e) {
                    System.err.println("glitch, continuing... " +(e.getMessage()!=null?e.getMessage():""));
                }
            }
        } catch (IOException e) {
            System.err.println("network error: " + (e.getMessage()!=null?e.getMessage():""));
        }
    }

    private void read(SelectionKey key) throws IOException {
        DatagramChannel chan = (DatagramChannel)key.channel();
        Con con = (Con)key.attachment();
        con.sa = chan.receive(con.req);
        System.out.println(new String(con.req.array(), "UTF-8"));
        con.resp = Charset.forName( "UTF-8" ).newEncoder().encode(CharBuffer.wrap("send the same string"));
    }

    private void write(SelectionKey key) throws IOException {
        DatagramChannel chan = (DatagramChannel)key.channel();
        Con con = (Con)key.attachment();
        chan.send(con.resp, con.sa);
    }

    static public void main(String[] args) {
        ASyncUDPSvr svr = new ASyncUDPSvr();
        svr.process();
    }
}

When dealing with small data sizes that fit in one packet, clearly if the NIO interface signals us that data is available to be read, then all the data must be available. Thus the protocol does not need to worry about accumulating network data in buffers. We still do need an object that is tied to each client connection as the reading and writing happen in two distinct parts of the code.

First, after establishing our UDP socket locally on the server, we signal NIO that the socket is ready for reads. When NIO wakes us up - via the select() call - we can immediately read the full request made by the client. At this point, we form our response but do not want to write it back to the network right away, as the kernel buffers may be full and the write may block. So, we store the response on the object attached to the client connection (via the SelectionKey object), signal NIO that we are now ready to write and go back to our select() loop.

Next when NIO wakes us up from the select() call, we can proceed to write. Again since the data fits in one packet, we know that the send() call need not be retried, and all data will be sent.

However, the nature of UDP does not provide the advantages TCP provides in epoll() mode. A UDP server does not provide a separate socket for each new client. Thus the epoll selector always has just the single socket. Each new client sends its datagrams to the single UDP receive buffer of the server.

A threaded server without the use of epoll() might be more advantageous. Each thread could wait on the single server socket, using a receive() call. The kernel will ensure that only one thread wakes up from the receive() call. I hope to use such an implementation and measure both designs.