Reading UTF-8 data from asynchronous sockets to the file system

Using asynchronous sockets, data is generally read into ByteBuffer objects. The general pattern is to read multiple times until there is no more data, and each time when the ByteBuffer is full, transfer to a larger buffer, like a ByteArrayOutputStream.

Now if you want to manipulate data collected (which is now in the ByteArrayOutputStream) as a String, it has to be decoded. This can be done using the CharsetDecoder object like this:

ByteArrayOutputStream outStrm;

// read data to outStrm using nio

CharBuffer charBuffer = CharBuffer.allocate(outStrm.size());
byte[] ba = outStrm.toByteArray();
ByteBuffer byteBuffer = ByteBuffer.wrap(ba);
Charset charset = Charset.forName( "UTF-8" );
decoder = charset.newDecoder();
CoderResult res = decoder.decode(byteBuffer, charBuffer, true);
res = decoder.flush(charBuffer);
String out = charBuffer.flip().toString();

However, all this decoding does is translating UTF-8 characters to their respective code points. As a result, we can't save this data to a file (OutputStream) correctly.

If you were to print the out string to the display, it is not guaranteed to print valid UTF-8 characters. Of course it will work for the single byte characters, but not necessarily for the multi-byte characters. Ex: 0xca a0 represents a non-breaking space with a code point of 0xA0. The above decoding will decode this to the code point 0xA0, but if you now write this to an output stream, it will not be stored as UTF-8, as the decoding stripped the UTF-8 and replaced it with code points.

So the correct approach is to simply write the byte buffer to an output stream like this:

outStrm.writeTo(System.out);

This will present UTF-8 characters to the output stream and thus the file will be saved as correct UTF-8 data.

A lenient URL decoder for Java

The URLDecoder class in the JDK insists on doing a strict parsing of escape characters in an encoded URL string. Sometimes, the application might want to decode correctly escaped string sequences and leave incorrect sequences intact. In fact the Sun documentation states that this aspect of decode handling is implementation dependent. However, Sun's implementation is strict - it throws an exception when it encounters improper escape sequences rather than treating them like regular text.

I couldn't find a lenient implementation so hand-crafted this from the original source for URLDecode class found here.

Following is the lenient decode.

    public static String decodeLenient(String s, String enc)
           throws UnsupportedEncodingException {

       boolean needToChange = false;
       StringBuffer sb = new StringBuffer();
       int numChars = s.length();
       int i = 0;

       if (enc.length() == 0) {
           throw new UnsupportedEncodingException("URLDecoder: empty string enc parameter");
       }

       while (i < numChars) {
           char c = s.charAt(i);
           switch (c) {
               case '+':
                   sb.append(' ');
                   i++;
                   needToChange = true;
                   break;
               case '%':
                   /*
                  # * Starting with this instance of %, process all
                  # * consecutive substrings of the form %xy. Each
                  # * substring %xy will yield a byte. Convert all
                  # * consecutive bytes obtained this way to whatever
                  # * character(s) they represent in the provided
                  # * encoding.
                  # */

                   // (numChars-i)/3 is an upper bound for the number
                   // of remaining bytes
                   byte[] bytes = new byte[(numChars - i) / 3];
                   int pos = 0;

                   while (((i + 2) < numChars) &&
                           (c == '%')) {
                       String hex = s.substring(i + 1, i + 3);
                       try {
                           bytes[pos] =
                                   (byte) Integer.parseInt(hex, 16);
                           pos++;
                       } catch (NumberFormatException e) {
                           sb.append(new String(bytes, 0, pos, enc));
                           sb.append("%");
                           sb.append(hex);
                           pos = 0;
                       }
                       i += 3;
                       if (i < numChars)
                           c = s.charAt(i);
                   }

                   sb.append(new String(bytes, 0, pos, enc));

                   // A trailing, incomplete byte encoding such as
                   // "%x" will be treated as unencoded text
                   if ((i < numChars) && (c == '%')) {
                       for (; i<numChars; i++) {
                           sb.append(s.charAt(i));
                       }
                   }

                   needToChange = true;
                   break;
               default:
                   sb.append(c);
                   i++;
                   break;
           }
       }

       return (needToChange ? sb.toString() : s);
   }
}

Remove intrusive bing popups from content sites

Today as I was pursuing the moneycentral.com site, in my quest to acquire the financial knowledge of the Wall Street robber barons, I stumbled on the latest feature from Bing : annoying hover over popups that show a search listing.

They are rather annoying for two reasons:
1) They appear over the text you are reading
2) It is very difficult to navigate around them as all you need to do is to have the mouse cursor on the link area for the helpful popup to appear

So I modified my previous GreaseMonkey script to handle this as well. Fortunately it is easy as this popup is fired off an anchor tag with an attribute of "itxtdid". All we need to do is remove the attribute, and the popup gets disabled.

Below is the new script. Enjoy.

// ==UserScript==
// @name           test
// @namespace      http://userscripts.org/thushara/
// @include        *
// ==/UserScript==
var allLinks, thisLink;
allLinks = document.evaluate(
 '//a[@href]',
 document,
 null,
 XPathResult.UNORDERED_NODE_SNAPSHOT_TYPE,
 null);
for (var i = 0; i < allLinks.snapshotLength; i++) {
 thisLink = allLinks.snapshotItem(i);
 // do something with thisLink
 if (thisLink.href.substring(0,19)=="http://www.bing.com") {
   thisLink.removeAttribute("href");
 }
 if (thisLink.hasAttribute("itxtdid")) {
   thisLink.removeAttribute("itxtdid");
 }
}

Ubuntu 8.04 - sound on flash player with firefox

If sound works generally, but not inside Flash Player (in FireFox), try this:

thushara@agni:~$ sudo apt-get install libflashsupport
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
libflashsupport
0 upgraded, 1 newly installed, 0 to remove and 131 not upgraded.
Need to get 8326B of archives.
After this operation, 65.5kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com hardy/universe libflashsupport 1.9-0ubuntu1 [8326B]
Fetched 8326B in 0s (14.9kB/s)
Selecting previously deselected package libflashsupport.
(Reading database ... 99009 files and directories currently installed.)
Unpacking libflashsupport (from .../libflashsupport_1.9-0ubuntu1_i386.deb) ...
Setting up libflashsupport (1.9-0ubuntu1) ...

Processing triggers for libc6 ...
ldconfig deferred processing now taking place
thushara@agni:~$


restart FireFox and sound should be available...

remove bing links from content sites

Lately, I have seen numerous "bing" links appearing in certain content sites I frequent. A case in point is http://articles.moneycentral.msn.com

It is somewhat insidious as unless I happen to note it is a search link, I pursue it imagining it will take me to some good content.

So I wrote a GreaseMonkey script to disable those links. Here goes:

// ==UserScript==
// @name           test
// @namespace      http://userscripts.org/thushara/
// @include        http://articles.moneycentral.mn.com/*
// ==/UserScript==
var allLinks, thisLink;
allLinks = document.evaluate(
    '//a[@href]',
    document,
    null,
    XPathResult.UNORDERED_NODE_SNAPSHOT_TYPE,
    null);
for (var i = 0; i < allLinks.snapshotLength; i++) {
    thisLink = allLinks.snapshotItem(i);
    // do something with thisLink
    if (thisLink.href.substring(0,19)=="http://www.bing.com") {
      thisLink.removeAttribute("href");
    }
}

I'm not too familiar with XPath, but I believe it is possible to specify "http://www.bing.com" inside the XPath query itself, so that there is no need to iterate through all the links finding the search links. I couldn't get the syntax right for this. Please post if you find a way around this.

groovy, mysql and case sensitivity

Groovy seems to have a somewhat hard to grok policy on case sensitivity as it pertains to mysql columns. To illustrate, for a table apiaccess with a column Domain this fails as of groovy version 1.6.3:

  query = "select domain from apiaccess where apiaccessid=3052";
  row = sql.firstRow(query);
  dom = row.Domain;

The reason is that the Domain in row.Domain does not match regards case with domain in the filter string.

This works:

  query = "select domain from apiaccess where apiaccessid=3052";
  row = sql.firstRow(query);
  dom = row.domain;

However, on an earleir version (perhaps the RC1 candidate of 1.6), the first block of code worked. There groovy expected a match with the actual mysql column name vs the filter string.

On both versions, the case used in the filter string do not need to match the mysql column names with regards to case.

mv is not atomic in Mac OS

you shouldn't rely on `mv` being atomic on the regular file system under MacOS. i had a script that had to regularly update a file that is read by a different script. under this scenario i resorted to writing a temporary file and then `mv`ing the file to the permanent location. while this works for linux, it doesn't work for MacOS.

to demonstrate, open two command windows in you Mac and in one type this:

while true; do echo this better be a whole sentence > x1.txt; mv x1.txt x.txt; done

on the other, run this script:

while true
do
    F=`cat x.txt`
    echo $F
    if [ "$F" = "this better be a whole sentence" ]
    then
    echo ok
    else
    echo bad
    exit -1
    fi
done

notice the output:

mpire@brwdbs02:~$ ./x.sh
this better be a whole sentence
ok
this better be a whole sentence
ok
ok
this better be a whole sentence
ok
this better be a whole sentence
ok
cat: x.txt: No such file or directory

bad
[~]


bad
mpire@brwdbs02:~$