Some web servers are in love with 30X redirects

Here is a URL that redirects via 30X response headers no less than 10 times:

http://join.scoreondemand.com/strack/MTAwNC45LjQ0LjQ0LjI5LjAuMC4wLjA/scoreondemand/64/0/Default.aspx

The evidence:

mpire@seaxoaff01:~$ curl -I http://join.scoreondemand.com/strack/MTAwNC45LjQ0LjQ0LjI5LjAuMC4wLjA/scoreondemand/64/0/Default.aspx
HTTP/1.1 302 Found
Date: Fri, 20 May 2011 19:56:12 GMT
Cneonction: close
Location: http://join.scoreondemand.com/track/MTAwNC42NC40Ny40Ny4yOS4wLjAuMC4w/Default.aspx?switched=1&strack=0
ScoreTracker: scash04
Content-Type: text/html
Set-Cookie: NSC_tdpsfdbti-obut-80=ffffffff090a1f1e45525d5f4f58455e445a4a423660;Version=1;Max-Age=3600;path=/;httponly

mpire@seaxoaff01:~$ curl -I "http://join.scoreondemand.com/track/MTAwNC42NC40Ny40Ny4yOS4wLjAuMC4w/Default.aspx?switched=1&strack=0"
HTTP/1.1 302 Found
Date: Fri, 20 May 2011 19:56:31 GMT
Set-Cookie: PHPSESSID=rbra31g59vc4me30bbofgei7d1; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
nnCoection: close
Set-Cookie: nats=MTAwNC42NC40Ny40Ny4yOS4wLjAuMC4w; expires=Mon, 30-May-2011 19:56:31 GMT; path=/; domain=scoreondemand.com
Set-Cookie: nats_cookie=No%2BReferring%2BURL; expires=Mon, 30-May-2011 19:56:31 GMT; path=/; domain=scoreondemand.com
Set-Cookie: nats_unique=MTAwNC42NC40Ny40Ny4yOS4wLjAuMC4w; expires=Sat, 21-May-2011 19:56:31 GMT; path=/; domain=scoreondemand.com
Set-Cookie: nats_sess=726064a93aca6b9d49d72dd57f477c57; expires=Sun, 28-Aug-2011 19:56:31 GMT; path=/; domain=scoreondemand.com
Location: http://www.scoreondemand.com/Default.aspx?nats=MTAwNC42NC40Ny40Ny4yOS4wLjAuMC4w&switched=1&strack=0
ScoreTracker: scash01
Content-Type: text/html
Set-Cookie: NSC_tdpsfdbti-obut-80=ffffffff090a1f1d45525d5f4f58455e445a4a423660;Version=1;Max-Age=3600;path=/;httponly

mpire@seaxoaff01:~$ curl -I "http://www.scoreondemand.com/Default.aspx?nats=MTAwNC42NC40Ny40Ny4yOS4wLjAuMC4w&switched=1&strack=0"
HTTP/1.1 302 Found
Date: Fri, 20 May 2011 19:56:47 GMT
X-AspNet-Version: 2.0.50727
Location: http://join.eboobstore.com/strack/MTAwNC42NC40Ny40Ny4yOS4wLjAuMC4w/eboobstore/64/0/apple/
Set-Cookie: ASP.NET_SessionId=xhivgbmef0bqm1uhihz1r455; path=/; HttpOnly
Set-Cookie: SVOD1=UserID=11133628&SessionID=1bM0125818hnmy5CAG9P; expires=Thu, 18-Aug-2011 19:56:47 GMT; path=/
Set-Cookie: NATS=MTAwNC42NC40Ny40Ny4yOS4wLjAuMC4w; expires=Thu, 18-Aug-2011 19:56:47 GMT; path=/
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 206

mpire@seaxoaff01:~$ curl -I "http://join.eboobstore.com/strack/MTAwNC42NC40Ny40Ny4yOS4wLjAuMC4w/eboobstore/64/0/apple/"
HTTP/1.1 302 Found
Date: Fri, 20 May 2011 19:57:13 GMT
Cneonction: close
Location: http://join.eboobstore.com/track/MTAwNC42NC41MC41MC4yOS4wLjAuMC4w/apple/?switched=1&strack=0
ScoreTracker: scash04
Content-Type: text/html; charset=UTF-8
Set-Cookie: NSC_tdpsfdbti-obut-80=ffffffff090a1f1e45525d5f4f58455e445a4a423660;Version=1;Max-Age=3600;path=/;httponly

mpire@seaxoaff01:~$ curl -I "http://join.eboobstore.com/track/MTAwNC42NC41MC41MC4yOS4wLjAuMC4w/apple/?switched=1&strack=0"
HTTP/1.1 302 Found
Date: Fri, 20 May 2011 19:57:39 GMT
Set-Cookie: PHPSESSID=utrmur64derb0n2onaotf08640; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
nnCoection: close
Set-Cookie: nats=MTAwNC42NC41MC41MC4yOS4wLjAuMC4w; expires=Mon, 30-May-2011 19:57:39 GMT; path=/; domain=eboobstore.com
Set-Cookie: nats_cookie=No%2BReferring%2BURL; expires=Mon, 30-May-2011 19:57:39 GMT; path=/; domain=eboobstore.com
Set-Cookie: nats_unique=MTAwNC42NC41MC41MC4yOS4wLjAuMC4w; expires=Sat, 21-May-2011 19:57:39 GMT; path=/; domain=eboobstore.com
Set-Cookie: nats_sess=a7568ec181d54a6316d6452656565dba; expires=Sun, 28-Aug-2011 19:57:39 GMT; path=/; domain=eboobstore.com
Location: http://www.eboobstore.com/apple/?nats=MTAwNC42NC41MC41MC4yOS4wLjAuMC4w&switched=1&strack=0
ScoreTracker: scash01
Content-Type: text/html; charset=UTF-8
Set-Cookie: NSC_tdpsfdbti-obut-80=ffffffff090a1f1d45525d5f4f58455e445a4a423660;Version=1;Max-Age=3600;path=/;httponly

mpire@seaxoaff01:~$ curl -I "http://www.eboobstore.com/apple/?nats=MTAwNC42NC41MC41MC4yOS4wLjAuMC4w&switched=1&strack=0"
HTTP/1.1 302 Found
Date: Fri, 20 May 2011 19:58:46 GMT
Location: http://www.eboobstore.com/urlmunge/munger/nats=MTAwNC42NC41MC41MC4yOS4wLjAuMC4w&switched=1&strack=0_URL_apple/
Content-Type: text/html; charset=UTF-8

mpire@seaxoaff01:~$ curl -I "http://www.eboobstore.com/urlmunge/munger/nats=MTAwNC42NC41MC41MC4yOS4wLjAuMC4w&switched=1&strack=0_URL_apple/"
HTTP/1.1 302 Found
Date: Fri, 20 May 2011 19:58:58 GMT
Set-Cookie: PHPSESSID=33hcivt9bgl7o6qv5bpfv7aun4; path=/; domain=eboobstore.com
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Location: http://www.eboobstore.com/apple
ScoreTracker: web04
Content-Type: text/html; charset=UTF-8

mpire@seaxoaff01:~$ curl -I "http://www.eboobstore.com/apple"
HTTP/1.1 301 Moved Permanently
Date: Fri, 20 May 2011 19:59:10 GMT
Location: http://eboobstore.com/apple/
Content-Type: text/html; charset=UTF-8

mpire@seaxoaff01:~$ curl -I "http://eboobstore.com/apple/"
HTTP/1.1 301 Moved Permanently
Date: Fri, 20 May 2011 19:59:48 GMT
Location: http://www.eboobstore.com/apple/
Content-Type: text/html; charset=UTF-8

mpire@seaxoaff01:~$ curl -I "http://www.eboobstore.com/apple/"
HTTP/1.1 200 OK
Date: Fri, 20 May 2011 20:00:30 GMT
ScoreTracker: web06
Content-Type: text/html; charset=UTF-8

Insanely compressed html files

Today, I discovered a URL that sent some insanely compressed content. The compressed content was sent by the server using Content-Encoding: gzip and Transfer-encoding: chunked. The compressed size of the content was 2,921,925 bytes and it decompressed to 1,004,263,982 bytes. The decompressed content was roughly 344 times the size of the compressed content.

This caused certain things to go wrong in the production process. I had set a limit of a few Megs on all fetches and had assumed that a single fetch could not be more than a few Megs. This was the first time I have seen such a huge decompression rate. This caused a subsequent file mapping to fail due to inadequate memory.

The downloaded content suggested why this would compress so well. The URL was http://www.jeltel.com.au/news.php There seems to be a dynamically generated part on this URL. If you examine its source, you will see a marker like this:

<!-- JELTEL_CONTENT_BEGIN -->

Content after that seems dynamically generated. You will find markup like this:

<h2></h2> - <br/><h4>... <a href="">read more</a></h4>

On this particular instance, there was an unusually large amount of fake content generated. The downloaded file had just 33 lines, but the last long line was a huge repeating pattern of :

<a href="">read more</a></h4><br/><br/><h2></h2> - <br/><h4>... 

This would of course compress well.

Linux sort : bug with , separator and confusing period?

user@host:~/$ echo -e "alan,20,3,0\ngeorge,5,0,0\nalice,3,5,0\ndora,4,0.9,5" | sort -n -k 2 -t ,
dora,4,0.9,5
alice,3,5,0
george,5,0,0
alan,20,3,0user@host:~/$ 

The line with "dora" as the first term should be printed after "alice" and before "george", as we are asking sort to sort on the second column. The 3rd column value of "0.9" seems to confuse sort on this.

This is not a bug in sort but due to the locale setting on different operating systems.

On the above link, look for "Sort does not sort in normal order".

Setting LC_PATH=C sorts as expected:

user@host:~/$ echo -e "alan,20,3,0\ngeorge,5,0,0\nalice,3,5,0\ndora,4,0.9,5" | LC_ALL=C sort -n -k 2 -t ,
alice,3,5,0
dora,4,0.9,5
george,5,0,0
alan,20,3,0
user@host:~/$ 

A useful, scriptable way to remove offending known_hosts keys

You can use ssh-keygen -R to remove invalid keys from the known_hosts file. This becomes useful if the host names are hashed in the file. The default in Ubuntu/Lucid is to hash the host names.

Java code to tail -N a text file

This code allows you to go over the last N lines of a specified file. It has a "head" method, which simply allows you to go over the file from the beginning.

import java.io.*;
import java.nio.channels.FileChannel;
import java.nio.CharBuffer;
import java.nio.ByteBuffer;
import java.util.Iterator;
import java.util.NoSuchElementException;public class MMapFile {

    public class MMapIterator implements Iterator<String> {
        private int offset;

        public MMapIterator(int offset) {
            this.offset = offset;
        }
        
        public boolean hasNext() {
            return offset < cb.limit();
        }

        public String next() {
            ByteArrayOutputStream sb = new ByteArrayOutputStream();
            if (offset >= cb.limit())
                throw new NoSuchElementException();
            for (; offset < cb.limit(); offset++) {
                byte c = (cb.get(offset));
                if (c == '\n') {
                    offset++;
                    break;
                }
                if (c != '\r') {
                    sb.write(c);
                }

            }
            try {
                return sb.toString("UTF-8");
            } catch (UnsupportedEncodingException e) {}
            return sb.toString();
        }

        public void remove() {

        }
    }


    private ByteBuffer cb;
    long size;
    private long numLines = -1;
    public MMapFile(String file) throws FileNotFoundException, IOException {
        FileChannel fc = new FileInputStream(new File(file)).getChannel();
        size = fc.size();
        cb = fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size());
    }

    public long getNumLines() {
        if (numLines != -1) return numLines;  //cache number of lines
        long cnt = 0;
        for (int i=0; i <size; i++) {
            if (cb.get(i) == '\n')
                cnt++;
        }
        numLines = cnt;
        return cnt;
    }

    public Iterator<String> tail(long lines) {
        long cnt=0;
        long i=0;
        for (i=size-1; i>=0; i--) {
            if (cb.get((int)i) == '\n') {
                cnt++;
                if (cnt == lines+1)
                    break;
            }
        }
        return new MMapIterator((int)i+1);
    }

    public Iterator<String> head() {
        return new MMapIterator(0);
    }

    static public void main(String[] args) {
        try {
            Iterator<String> it = new MMapFile("/test.txt").head();
            while (it.hasNext()) {
                System.out.println(it.next());
            }
        } catch (Exception e) {
            
        }

        System.out.println();

        try {
            Iterator<String> it = new MMapFile("/test.txt").tail(2);
            while (it.hasNext()) {
                System.out.println(it.next());
            }
        } catch (Exception e) {

        }

        System.out.println();

        try {
            System.out.println("lines: "+new MMapFile("/test.txt").getNumLines());
        } catch (Exception e) {

        }

    }

}

The technique is to simply map the file into memory using java.nio.channels.FileChannel.map in the Java NIO library and manipulate the file data using memory techniques.

For the "tail" function, we walk back the mapped bytes counting newlines. The MMapIterator class conveniently provides a way to iterate over lines once we find the starting line.

There is a point where care must be taken in the MMapIterator.next() implementation. That is making sure that bytes are converted to the appropriate string encoding. We use "UTF-8" but if you are dealing with a different encoding in the input file, this should be changed.

Java : all 8-bit data cannot be cast to char type

If you have a byte and want to make a String, a simple cast to a char will only work if you are dealing with 7-bit ASCII. If that byte could be extended ASCII, the cast will not encode to the correct character code.

This is the sure way to get all 8-bit characters represented in Strings:

new String(new byte[]{c}, "ISO-8859-1")

This is to do with the default encoding used by the JVM which is likely not "ISO-8859-1". On Linux, it is likely to be "UTF-8" and it is "MacRoman" on Snow Leopard (Mac).

Running HBase - some issues to be aware of

I want to take a moment to note down a few issues I had with setting up a distributed HBase environment in case it helps someone else.

First, I set up Hadoop from the 0.20 append branch as described here. I used two machines where the first machine was the master and both machines were used as slaves. This is a guide I used.

mkdir ~/hadoop-0.20-append
cd ~/hadoop-0.20-append
svn co https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append/ .
ant jar

At the end of this, you will have the hadoop jar file at ~/hadoop-0.20-append/build

The first mistake I made was to use the IP address of the name node for fs.default.name in the conf/core-site.xml file. There is a bug in Hadoop 0.20 release that prevents the use of IP address in this context.

Interestingly, the basic HDFS shell commands worked (ex: get, ls) with the IP address being used for fs.default.name. The problem only cropped up after I setup HBase and tried to use the HBase shell.

To setup HBase, I followed the steps outlined here.

Before I discovered the IP related issue, I encountered an error that showed I was not following the steps faithfully enough. While HBase ships with a version of Hadoop from presumably the 0.20 append branch, it was not identical to the version I built from the 0.20 append branch. As stated in the documentation, I then copied the Hadoop jar I built over the jar shipped with HBase.

Next, I ran into the IP issue. Generally, changing fs.default.name and restarting the Hadoop cluster is not enough in such cases as certain data has been written to HDFS name and data directories already and any mismatch emboldens further "namespace mismatch" errors. Thus, before changing the fs.default.name, I removed the directories specified by dfs.name.dir and dfs.data.dir. In case of dfs.data.dir, I had to remove it on both slaves. Then I changed the IP over to the machine name, formatted the name node and re-started the Hadoop cluster.

It still was not over. This time there was the issue of these machine names not being in the DNS. They happened to be simply names assigned to these machines which were not in the domain name system used by the machines to communicate to one another. Thus I went into the /etc/hosts file on both machines and added appropriate entries to allow each box to resolve the domain name to an IP.

After which, I could create a table and insert rows into it as explained here.

The next step was to programmatically create a table and add rows to it. I adapted the example from here. The programming API by default allows the code to find the hbase configuration files using the class path. Thus, I added the path to the hbase/conf directory to the classpath to get the program to work. Alternately, you could use

org.apache.hadoop.hbase.HbaseConfiguration.addHbaseResources(org.apache.hadoop.conf.Configuration) 

which would have you use

org.apache.hadoop.conf.Configuration.addResources(org.apache.hadoop.fs.Path)

to add the paths to individual configuration files.