deleting an iterator in accumulo

I learnt the hard way that setting an iterator in the accumulo shell sets it for a table permanently. To make matters worse, I set this iterator in the metadata table and made everything fail.

Removing the iterator was tricky. First I had to find what accumulo decided to call the iterator as I did not specify a name but just the java class:

Here was the command I used:

setiter -class org.apache.accumulo.core.iterators.FirstEntryInRowIterator -p 99 -scan

Here is how I found what the iterators for accumulo.metadata table were called:

config -t accumulo.metadata -f iterator


SCOPE      | NAME                                                  | VALUE


table      | table.iterator.majc.bulkLoadFilter .................. | 20,org.apache.accumulo.server.iterators.MetadataBulkLoadFilter

table      | table.iterator.majc.vers ............................ | 10,org.apache.accumulo.core.iterators.user.VersioningIterator

table      | table.iterator.majc.vers.opt.maxVersions ............ | 1

table      | table.iterator.minc.vers ............................ | 10,org.apache.accumulo.core.iterators.user.VersioningIterator

table      | table.iterator.minc.vers.opt.maxVersions ............ | 1

table      | table.iterator.scan.firstEntry ...................... | 99,org.apache.accumulo.core.iterators.FirstEntryInRowIterator

table      | table.iterator.scan.firstEntry.opt.scansBeforeSeek .. | 10

table      | table.iterator.scan.vers ............................ | 10,org.apache.accumulo.core.iterators.user.VersioningIterator

table      | table.iterator.scan.vers.opt.maxVersions ............ | 1

The iterator I added seemed to be named "table.iterator.scan.firstEntry", so I tried to delete that:
root@work accumulo.metadata> deleteiter -n table.iterator.scan.firstEntry -t accumulo.metadata

2014-12-20 15:13:42,854 [shell.Shell] WARN : no iterators found that match your criteria

You have to specify just the last part of the iterator name:
root@work accumulo.metadata> deleteiter -scan -n firstEntry -t accumulo.metadata

grep many files while printing file name

A useful trick I found:

find . -exec grep -n hello /dev/null {} \;

Including more than one file makes grep print the file name as well as the line number. So we use the handy /dev/null as one extra file to do the job.

Attribution: http://www.linuxquestions.org/questions/programming-9/find-grep-command-to-find-matching-files-print-filename-then-print-matching-content-328036/

Decoding HTML pages with Content-Encoding : deflate

All web servers do not implement zlib protocol the same way when they return data with Content-Encoding set to deflate. Some servers return a zlib header as specified in RFC 1950, but some return the compressed data alone.

Java Inflator class can be used to deal with both cases, but first we must check for the header. The first two bytes denote the header and it is a simple check :

    static boolean isZlibHeader(byte[] bytes) {
        //deal with java stupidity : convert to signed int before comparison
        char byte1 = (char)(bytes[0] & 0xFF);
        char byte2 = (char)(bytes[1] & 0xFF);
        
        return byte1 == 0x78 && (byte2 == 0x01 || byte2 == 0x9c || byte2 == 0xDA);
    }

    private void inflateToFile(byte[] encBytes, int offset, int size, BufferedOutputStream f) throws IOException {
        Inflater inflator = new Inflater(true);
        inflator.setInput(encBytes, isZlibHeader(encBytes) ? offset+2 : offset, isZlibHeader(encBytes) ? size-2 : size);
        byte[] buf = new byte[4096];
        int nbytes = 0;
        do {
            try {
                nbytes = inflator.inflate(buf);
                if (nbytes > 0) {
                    f.write(buf,0,nbytes);
                }
            } catch (DataFormatException e) {
                //handle error
            }
        } while (nbytes > 0);
        inflator.end();
    }

An example URL that had to be processed this way : http://game4fun.square7.ch Here is the Wireshark capture, showing the Content-Encoding set to deflate as well as the de-chunked header (the first 2 bytes "78 9c") at the lower bottom pane of the display:

Mapping sockets of a process to the remote end point

Recently, one of our long running processes started exhibiting a high number of open file handles. We were leaking handles somewhere. The first thing is to figure out what handles are open, which is easy in Linux with /proc. Just plug in the PID of your process:

ls -ltr /proc/21657/fd

This spits out all the open file handles for the process with PID 21657. Here is an example of an open socket:

lrwx------ 1 user user 64 May 20 13:20 649 -> socket:[2336308491]

This alone doesn't tell us much. Our application use sockets for many reasons. There are connections to mysql, memcache and mongodb. There are sockets listening and responding to requests. There are connections made to web servers.

To get an idea of the two end points of the socket, we need to look at /proc/net/tcp (as well as tcp6, udp, udp6) :

user@host ~$ cat /proc/net/tcp6 | grep 2336308491
 129: 0000000000000000FFFF00004D29650A:9030 0000000000000000FFFF00005881754A:01BB 08 00000000:00000001 00:00000000 00000000   237        0 2336308491 1 ffff81061cf60740 371 40 0 4 -1

This is a connection to a SSL port on the remote end; 0x01BB = 443

telnet relocation error: symbol krb5int_labeled_fopen

Ever had telnet not work on a machine? It happened to me recently, on a Centos 5.8, with this error message:

telnet: error: relocation error: symbol krb5int_labeled_fopen, version krb5support_0_MIT not defined in file libkrb5support.so.0 with link time reference (fatal)

Googling hinted at a conflict in the shared library providing kerberos authentication. So I ran telnet under LD_DEBUG=all :

LD_DEBUG=all telnet host port

which, showed me the problem:

29655:    symbol=krb5int_labeled_fopen;  lookup in file=/usr/local/greenplum-db/lib/libkrb5support.so.0 [0]
29655:    telnet: error: relocation error: symbol krb5int_labeled_fopen, version krb5support_0_MIT not defined in file libkrb5support.so.0 with link time reference (fatal)

So, an installation of greenplum had inserted its version of the kerberos library ahead of the search path for libraries the linux loader uses. The kerberos version of the library did not export the said function.

This all can be verified quickly :

login@host ~$ nm -D /usr/local/greenplum-db/lib/libkrb5support.so.0 | grep krb5int_labeled_fopen

login@host ~$ nm -D /usr/lib64/libkrb5support.so.0 | grep krb5int_labeled_fopen
00000033aea040b0 T krb5int_labeled_fopen

The greeenplum installation was using the LD_LIBRARY_PATH to allow it preferential status, so inserting the /usr/lib64 before it, was sufficient to help telnet find the right library.

login@host ~$ export LD_LIBRARY_PATH=/usr/lib64:$LD_LIBRARY_PATH

login@host ~$ telnet host port
Trying host...
Connected to host (ip).
Escape character is '^]'.