block quotes in bash

:<<supercalifragilisticexpialidocious
echo oki doki goobledi loks
supercalifragilisticexpialidocious

The long string better not appear in the block. It could be any string of your choosing.

The annoying preciseness of Java : Charset.isSupported

If the charset is supported, return true, else return false - it does sound pretty simple, doesn't it.

It would be simple if the language designers focused on usability vs pristine accuracy. Java, obviously went for the latter.

Thus if you were to ask whether an illegal charset is supported, it won't return false, it will decide to throw the IllegalCharsetNameException exception. How precise.


How annoying. Now when all you wanted was to check for the availability of a charset, suddenly you end up checking for exceptions as well as the return value from Charset.isSupported.


Such is the art of programming in Java.

Jericho Parser new version fixes choking on unusual charset

Each day I find something completely wacko on the Net. Today it is an extremely interesting charset present in the headers of a certain site:

mpire@brwdbs01:~$ curl -I http://uk.real.com/realplayer/
HTTP/1.1 200 OK
Expires: 0
Date: Thu, 13 May 2010 22:44:16 GMT
Content-Length: 2690
Server: Caudium
Connection: close
Content-Type: text/html; charset='.().'
pragma: no-cache
X-Host-Name: hhnode21.euro.real.com
X-Got-Fish: Yes
Accept-Ranges: bytes
MIME-Version: 1.0
Cache-Control: no-cache, no-store, max-age=0, private, must-revalidate, proxy-revalidate

a charset of '.().' Strange indeed. This manages to choke the JerichoParser I'm using and I'm not too sure what to do about it. The parser does a pretty nice job parsing all kinds of encodings, and since it has no idea of this kind, it gives up. I could try and make it use the default (ISO-8859-1).

This has been fixed on a newer version of the Jericho Parser.

Normalizing a URL

Today, I had this interesting problem to do with fetching a web page.
I was processing HTTP meta-refresh headers and ran into this type of header:

<meta http-equiv="REFRESH" content="0; URL=../cgi-bin/main2.cgi">

If I try to turn this into an absolute url (the url this content was fetched being http://popyard.com) I would be trying to do this:

> curl -I http://popyard.com/../cgi-bin/main2.cgi
> HTTP/1.1 400 Bad Request

However, turns out that the browser deals with this just fine. Faced with a malformed URL, it guesses and keeps going, fetching http://popyard.com/cgi-bin/main2.cgi

This required me to do some cleanup of the url to handle this dot segments (..). There is a well-known protocol for doing this. I coded this up simply using a stack based approach.

Here is the code:

    //removes .. sequences from the url string handling extra .. sequences by stopping
    //at the domain, thus always returning a correct url.
    //ex: http://www.ex.com/a/xt/../myspace.html => http://www.ex.com/a/myspace.html
    //    http://www.ex.com/a/xt/../../../myspace.html => http://www.ex.com/myspace.html    
    public static String normalizePath(String url) {
        if (url.indexOf("..") == -1)
            return url; //no .. seqs, no need to normalize
        String[] toks = url.split("/");
        int i;
        for (i=0; i<toks.length && (toks[i].length() == 0 || toks[i].toLowerCase().indexOf(":") != -1); i++);
        if (i==toks.length)
            return url;     //no proper path found, simply return the url
        // toks[i] is the domain

        LinkedList<String> s = new LinkedList<String>();
        for (; i<toks.length; i++) {
            if (!toks[i].equals(".."))
                s.push(toks[i]);
            else if (s.size()>1)
                s.pop();
        }

        if (s.size()<1)
            return url;     //no proper domain found, simply return the url

        int idx = url.indexOf("://");
        StringBuilder sb = new StringBuilder();
        sb.append( (idx != -1 ? url.substring(0, idx+3) : "")).append(s.removeLast()); //get proto://domain

        while (s.size()>0) {
            sb.append("/").append(s.removeLast());
        }

        return sb.toString();
    }

Basically, what this code does is strip the url into the domain and the path components and use a stack to manipulate these terms. We push a term onto the stack, unless the term is a dot segment - ".." - in which case, if there is at least two terms on the stack, we pop the first one. This way, we never pop the last term on the stack, which is the domain.

Then we can find the normalized URL following the terms in the stack from bottom to top.

This is the reason we can't really use a stack, as we can't traverse from the bottom to the top. So we use a LinkedList instead.

Interesting performance characteristic in Lucene.Analysis.StopFilter

I was tracking down a slow indexing process and got this callstack with YourKit:




There is a Collection.addAll() being called for each invocation of tokenStream() which is called many times per document. The StopFilter is the culprit. When I checked the constructor, it looks like this:

  public StopFilter(boolean enablePositionIncrements, TokenStream input, Set stopWords, boolean ignoreCase)
  {
    super(input);
    if (stopWords instanceof CharArraySet) {
      this.stopWords = (CharArraySet)stopWords;
    } else {
      this.stopWords = new CharArraySet(stopWords.size(), ignoreCase);
      this.stopWords.addAll(stopWords);
    }
    this.enablePositionIncrements = enablePositionIncrements;
    init();
  }

As you can see, if the stopWords set is not of type org.apache.lucene.analysis.CharArraySet, a whole copy is made. I was using a regular Set for the stopWords and that was the problem.

This was very instructive, as the reason I didn't use lucene's StopFilter.makeStopSet() and used a regular Set was assuming that under the covers Lucene made a regular Set, and I wanted to save it the trouble as I could generate a Set directly from my input.

Another reason to profile, profile and profile....

After changing my code to use the Lucene Set (by calling StopFilter.makeStopSet) here is the result, faster for sure:



The performance improvements could be significant. In a real-world scenario where this code was used in creating a 10G size index, I saw a 4X speed-up in indexing.

An In-Memory Dictionary with Java

I needed to build an in-memory dictionary to do fast lookups on terms retreived from a web page. The dictionary is around 13K words from English, including names, abbreviations etc. It was compiled from the SCOWL project.

The first thing I realized was that Java uses around 90M of memory to store 65M of string data. At first glance thus seems to be due to the overhead in String class - here is the relevant class structure showing the overhead:

public final class String  implements java.io.Serializable, java.lang.Comparable<java.lang.String>, java.lang.CharSequence{
    private final char[] value;
    private final int offset;
    private final int count;
    private int hash;
}

As you can see, apart from the char array, there is an offset, count and the hash of the string.

The offset, count fields are used to limit memory usage. Say, if lots of strings could be represented as substrings of others, there need to be only one copy of the char[] value array, the individual strings will reference the same array with different offset, count values.

The hash is used to speed up certain data access operations when Strings are used as a key in a collection like a Hash. Rather than computing the hash each time a String is searched for in a hash, the hash of the string can be stored in the String object. Since a string is immutable, this hash value does not need to be updated once set, making it easy to maintain.

But of course, it adds an extra 4 bytes of overhead.

However, the reason for the biggest increase seems to be the size of the char data type. Since it should be able to store a Unicode character, it takes 2 bytes. So even without the other overhead, we should expect twice the number of bytes as would be needed to store this with a byte array.

And interestingly, the offset, count trick seems to find enough substrings that the amount of memory required is below double the amount one would expect with a byte array. So what we thought was overhead in fact helped us here.

Next I tried to figure out the collisions rate, when the Strings are stored in a HashSet. I was using a HashSet for my dictionary. I found found 938 collisions from 628033 terms [0.14935522%] which is quite ok. Just to make sure that there was no unusual clustering around certain keys, I checked how many slots had more than one collision, meaning three or more keys would be stored on these slots. There were only 7 such slots, and they all had just 3 keys hashed onto them.

So the maximum length of the list at each hash slot was 3. This was quite acceptable.

Here is the code used to find the collision info.

import java.io.*;
import java.util.Set;
import java.util.HashSet;
import java.util.Map;
import java.util.HashMap;

public class Collider {
    static public void main(String[] args) {
        String dictDir = "/Users/thushara/code/platform/AdXpose/src/com/adxpose/affinity/en_dict";
        File[] files = new File(dictDir).listFiles();
        Set<Integer> set = new HashSet<Integer>();
        Map<Integer, Integer> collisions = new HashMap<Integer, Integer>();
        int dups = 0;
        int tot = 0;
        for (File file : files) {
            try {
                FileInputStream fstream = new FileInputStream(file);
                DataInputStream in = new DataInputStream(fstream);
                BufferedReader br = new BufferedReader(new InputStreamReader(in));
                String word;
                while ((word = br.readLine()) != null)   {
                    int hash = word.hashCode();
                    if (set.contains(hash)) {
                        dups++;
                        collisions.put(hash, collisions.get(hash) == null ? 1 : collisions.get(hash)+1);
                    } else {
                        set.add(hash);
                    }
                    tot++;
                }
            } catch (Exception e) {}
        }
        System.out.println("found " + dups + " collisions from " + tot + " terms [" + ((float)dups)/tot*100 + "%]");
        for (Map.Entry<Integer, Integer> entry : collisions.entrySet()) {
            if (entry.getValue()>1) System.out.println(entry.getKey() + ":" + entry.getValue());
        }
    }
}

with this output:

found 938 collisions from 628033 terms [0.14935522%]
78204:2
83505:2
76282:2
71462:2
-1367596602:2
79072:2
94424379:2

Of course the problem of collisons is addressed in the Birthday Paradox. We could also use this theory to determine if String.hashCode() is optimal.

Next I will make a simple light string class based on a byte array using the last byte as a terminator (like in the traditional C string, storing 0 as the last byte).

Java: building an asynchronous web page fetcher

Today I leveraged the earlier epoll() based framework (FishHooks) I built to fetch web pages. FishHooks is protocol-agnostic and as such most of the code in the fetcher has to do with HTTP. Send the proper request, parse the response etc.

There are, as usual, two files. The driver program ASyncWgetClient.java will use FishHooks to start the epoll() loop. WgetParam.java will extend ConnectionParam and implement the state machine peculiar to the HTTP request/response cycle. Among other things, we will handle the 30X redirects here.

import java.util.List;
import java.util.ArrayList;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.InetAddress;
import java.net.UnknownHostException;
import java.nio.charset.CharacterCodingException;

/**
 * @author: thushara
 * @version: Nov 18, 2009
 */
public class ASyncWgetClient {

    public void process() {

        class DNSBlob {
            String url;
            String path;
            String host;
            int port;
            URLWithRetries retryURL;
            byte[] inetAddr;

            public DNSBlob(String url, String path, String host, int port, URLWithRetries retryURL, byte[] inetAddr) {
                this.url = url;
                this.path = path;
                this.host = host;
                this.port = port;
                this.retryURL = retryURL;
                this.inetAddr = inetAddr;
            }
        };

        //ConnectionParam.verbose = true;
        //ASyncSocketHandler.verbose = true;

        ASyncSocketHandler hndlr = new  ASyncSocketHandler();

        String[] urls = {"http://zerohedge.com", "http://moneycentral.msn.com", "http://www.nakedcapitalism.com",
                "http://youtube.com", "http://democracynow.org", "http://love.com", "http://anncoulter.com",
                "http://peace.org", "http://grunge.com", "http://rockandroll.com", "http://shopping.com",
                "http://luck.com", "http://schools.org", "http://whipitthemovie.com",
                "http://war.org", "http://benstiller.com", "http://www.oprahwinfrey.com", "http://porche.com",
                "http://spanish.com", "http://healingart.com", "http://flash.com"};


        List<DNSBlob> dnsList = new ArrayList<DNSBlob>();

        for (String url : urls) {
            URLWithRetries retryURL = new URLWithRetries(url);
            URL urlObj = null;
            try {
                urlObj = new URL(url);
            } catch (MalformedURLException e) {
                WgetParam.writeErrorFile(WgetParam.getErrorFileName(url), url+" is not correct, skipping...");
                continue;
            }
            if (urlObj.getProtocol().equals("https")) {
                WgetParam.writeErrorFile(WgetParam.getErrorFileName(url), url+" not handling https, skipping...");
                continue;
            }
            String host = urlObj.getHost();
            String path = urlObj.getPath();
            if (urlObj.getQuery() != null) path += ("?"+urlObj.getQuery());
            if (path.length() == 0) path ="/";
            int port = urlObj.getDefaultPort();

            DNSResolver dnsRes = new DNSResolver(host);
            Thread t = new Thread(dnsRes);
            t.start();
            try {
                t.join(1000);
            } catch (InterruptedException e) {}
            byte[] inetAddr = dnsRes.get();

            if (inetAddr == null) {
                WgetParam.writeErrorFile(WgetParam.getErrorFileName(url), host + " has no DNS entry");
                continue;
            }

            dnsList.add(new DNSBlob(url, path, host, port, retryURL, inetAddr));
        }

        ConnectionParam[] params = new WgetParam[urls.length];
        for (int i=0; i<dnsList.size(); i++) {
            DNSBlob dnsBlob = dnsList.get(i);
            String request = "GET "+dnsBlob.path+" HTTP/1.0\r\nHost: "+dnsBlob.host+"\r\nAccept: */*\r\n\r\n";
            try {
                params[i] = new WgetParam(dnsBlob.inetAddr, dnsBlob.port, request, dnsBlob.retryURL, hndlr);
            } catch (CharacterCodingException e) {
                WgetParam.writeErrorFile(WgetParam.getErrorFileName(dnsBlob.url), e.getMessage());
                //just leave null
            }
        }

        try {
            hndlr.connect_all(params);
        } catch (Exception e) {
            System.err.println("unexpected :" + e.getMessage());
            e.printStackTrace();
        }
    }

    public class DNSResolver implements Runnable {
        private String domain;
        private byte[] inetAddr;

        public DNSResolver(String domain) {
            this.domain = domain;
        }

        public void run() {
            try {
                byte[] addr = InetAddress.getByName(domain).getAddress();
                set(addr);
            } catch (UnknownHostException e) {

            }
        }

        public synchronized void set(byte[] inetAddr) {
            this.inetAddr = inetAddr;
        }
        public synchronized byte[] get() {
            return inetAddr;
        }
    };

    public static void main(String[] args) {
        ASyncSocketHandler.verbose = true;
        ASyncWgetClient aWget = new ASyncWgetClient();
        aWget.process();
    }
}

import java.io.*;
import java.nio.charset.CharacterCodingException;
import java.nio.channels.FileChannel;
import java.nio.MappedByteBuffer;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.InetAddress;
import java.net.UnknownHostException;

/**
 * @author: thushara
 * @version: Nov 18, 2009
 */
public class WgetParam extends ConnectionParam {

    public static final String OUTPUT_DIR = "/Users/thushara/affinity3/";

    private URLWithRetries retryURL;
    private ASyncSocketHandler hndlr;

    public WgetParam(byte[] ip, int port, String request, URLWithRetries retryURL, ASyncSocketHandler hndlr) throws CharacterCodingException {
        super(ip, port, request);
        this.retryURL = retryURL;
        this.hndlr = hndlr;
    }

    // returns the next op we're interested in: OP_READ / OP_WRITE / OP_CLOSE
    // after connecting, we can proceed to send the GET request
    public int connected_ok() {
        return ASyncSocketHandler.OP_WRITE;
    }

    public int connect_failed() {
        return ASyncSocketHandler.OP_CLOSE;
    }

    // returns the next op we're interested in: OP_READ / OP_WRITE / OP_CLOSE
    // param: eod = true iif if the complete request was written to the socket
    //              if request is not fully written, return signal to write again
    //              else, we can go on to read, return read singnal.
    public int write_ok(boolean eod) {
        return eod ? ASyncSocketHandler.OP_READ : ASyncSocketHandler.OP_WRITE;
    }

    // params: eod = true iif all data was read from the socket, socket has no more data
    //               if all data is read, this socket won't return any more data so
    //               we should close the socket, signal such
    //               else we should continue reading, signal such.
    public int read_ok(boolean eod) {
        if (eod) {
            byte[] resp = getResponse();

            String fName = getFileName(retryURL.getOrigURL());
            String path = OUTPUT_DIR+fName;
            int code = ((int)(resp[9]-48))*100 + (resp[10]-48)*10 + resp[11]-48;
            switch(code) {
                case 200:
                case 403:
                case 404:
                case 400:
                    try {
                        writeFileWithoutHeaders(resp, path);
                    } catch (IOException e) {
                        writeErrorFile(getErrorFileName(retryURL.getOrigURL()), e.getMessage());
                    }
                    break;
                case 301:
                case 302:
                case 303:
                case 305: // Location: contains proxy, check if it is full url or just the proxy server name
                case 307:
                    if (retryURL.hitLimit()) {
                        writeErrorFile(getErrorFileName(retryURL.getOrigURL()), "URL attempts too many redirections - giving up");
                        break;
                    }
                    String redirURL = null;
                    try {
                        redirURL = getLocation(resp,  retryURL.getDomain());
                    } catch (MalformedURLException e) {
                        writeErrorFile(getErrorFileName(retryURL.getOrigURL()), e.getMessage());
                    }
                    addURL(redirURL);
                    break;
                //case 400:
                //    writeErrorFile(getErrorFileName(retryURL.getOrigURL()), "bad request: 400");
                //    break;
                case 408: // request timed out, try again
                    addURL(retryURL.url);
                    break;
                default:
                    writeErrorFile(getErrorFileName(retryURL.getOrigURL()), "unexpected response code: " + code);
                    break;
            }

        }
        return eod ? ASyncSocketHandler.OP_CLOSE : ASyncSocketHandler.OP_READ;
    }

    private void addURL(String redirURL) {
        retryURL.nextURL(redirURL);
        URL url = null;
        try {
            url = new URL(retryURL.url);
        } catch (MalformedURLException e) {
            writeErrorFile(getErrorFileName(retryURL.getOrigURL()), e.getMessage());
            return;
        }
        if (url.getProtocol().equals("https")) {
            writeErrorFile(getErrorFileName(retryURL.getOrigURL()), url+" not handling https, skipping...");
            return;     
        }
        String host = url.getHost();
        String path = url.getPath();
        if (url.getQuery() != null) path += ("?"+url.getQuery());
        if (path.length() == 0) path = "/";
        int port = url.getDefaultPort();
        InetAddress inetAddr = null;
        try {
            inetAddr = InetAddress.getByName(host);
        } catch (UnknownHostException e) {
            writeErrorFile(getErrorFileName(retryURL.getOrigURL()), e.getMessage());
            return;
        }
        String request = "GET "+path+" HTTP/1.0\r\nHost: "+host+"\r\nAccept: */*\r\n\r\n";
        ConnectionParam connParm = null;
        try {
            connParm = new WgetParam(inetAddr.getAddress(), port, request, retryURL, hndlr);
        } catch (CharacterCodingException e) {
            writeErrorFile(getErrorFileName(retryURL.getOrigURL()), e.getMessage());
            return;
        }
        try {
            hndlr.connect(connParm);
        } catch (ASyncSocketHandler.SelectorNotOpenException e) {}
        catch (IOException e) {
            writeErrorFile(getErrorFileName(retryURL.getOrigURL()), e.getMessage());
        }
    }

    private static String getFileName(String url) {
        try {
            MessageDigest md = MessageDigest.getInstance("MD5");
            byte[] hs = new byte[16];
            md.update(url.getBytes("UTF-8"), 0, url.length());
            hs = md.digest();
            return String.format("%02x%02x%02x%02x%02x%02x%02x%02x%02x%02x%02x%02x%02x%02x%02x%02x.url", hs[0],hs[1],hs[2],hs[3],hs[4],hs[5],hs[6],hs[7],hs[8],hs[9],hs[10],hs[11],hs[12],hs[13],hs[14],hs[15]);
        } catch (NoSuchAlgorithmException e) {
            System.err.println(e);
            e.printStackTrace();
            System.exit(-1);
        } catch (UnsupportedEncodingException e) {
            System.err.println(e);
            e.printStackTrace();
            System.exit(-1);
        }
        return null;
    }

    private static String getLocation(byte[] httpHeader, String domain) {
        String location = null;
        for (int i=0; i<Integer.MAX_VALUE; i++) {
            StringBuffer line = new StringBuffer();
            for (char c = (char)httpHeader[i]; c!= '\n'; i++,c=(char)httpHeader[i]) {
                line.append(c);
            }
            if (line.length()==0 || line.length()==1) break; // no Location field
            if (line.length()>"Location".length() && line.substring(0,"Location".length()).toLowerCase().equals("location")) {
                location = line.substring("Location".length()+2, line.length()-1);
                break;
            }
        }
        return location.toLowerCase().startsWith("http://") || location.toLowerCase().startsWith("https://") ? location : domain + (location.substring(0,1).equals("/") ? location : "/"+location);
    }

    private void writeFileWithoutHeaders(byte[] bytes, String path) throws IOException {
        File temp = File.createTempFile("page",null);
        FileOutputStream f = new FileOutputStream(temp);
        f.write(bytes);
        f.close();

        FileChannel chan =  new FileInputStream(temp).getChannel();
        MappedByteBuffer buf = chan.map(FileChannel.MapMode.READ_ONLY, 0, chan.size());
        int htmlOffset = 0;
        for (int i=0; i<Integer.MAX_VALUE; i++) {
            int j=0;
            for (char c = (char)buf.get(i); c!= '\n'; i++,j++,c=(char)buf.get(i));
            if (j==0 || (j==1 && buf.get(i-1) == '\r')) {
                htmlOffset = i+1;
                break;
            }
        }
        FileChannel chan2 = new FileOutputStream(path).getChannel();
        chan.transferTo(htmlOffset, chan.size()-htmlOffset, chan2);
        chan.close();
        chan2.close();
    }

    //two convenience methods to note errors, used from both WgetParam class and
    //ASyncWgetClient class.
    public static String getErrorFileName(String url) {
        return getFileName(url).substring(0,getFileName(url).length()-3)+"bad";
    }

    public static void writeErrorFile(String fName, String error) {
        try {
            FileWriter fstream = new FileWriter(OUTPUT_DIR+fName);
            BufferedWriter out = new BufferedWriter(fstream);
            out.write(error);
            out.close();
        } catch (Exception e) {//Catch exception if any
            System.err.println("Error writing error file: " + e.getMessage());
        }
    }


}

You can see there is a lot more code than the simpler client talking to time of day servers. Most of the code is to do with the networking protocol. You still don't deal with sockets, channels or byte buffers.