Java HTMLParser and meta http-equiv redirects

Java HTMLParser can be used to parse a web page into more meaningful chunks. For example, we can parse for links, H1 tags or images.

Some web pages (urls) in fact return a HTTP redirect response to the browser (instead of the 200 code for good content). The browser then makes a request to the redirected page (found in the location information supplied with the redirect response)

The HTMLParser handles this type of redirection automatically. However, there is another form of a redirect that is accomplished by an HTML meta tag like so:

<meta http-equiv="refresh" content="2;url=http://webdesign.about.com">

If a web page contains this tag, that instructs the browser to fetch the url mentioned under the content parameter after a 2 second delay. The content parameter thus provides two data inputs to the browser - the delay for the redirect and the new url.

Also note that if the url portion was omitted from the content parameter, this serves as an automatic refresh of the same url.

HTMLParser will not automatically redirect from a meta http-equiv tag.
Even HttpURLConnection has no facility for redirecting automatically from this type of directive. (It is possible to construct a HttpURLConnection and pass it to the HTMLParser, but like I said, this won't cover this special form of redirect either.)

So this is a case where we need to follow the redirect ourself. It is possible to extend HTMLParser to have this ability. Here is the bare-bones code one would use for following the redirect:


HttpURLConnection.setFollowRedirects(true);
HttpURLConnection urlCon = (HttpURLConnection)new
URL(url).openConnection();

Parser parser = new Parser(urlCon);

NodeList nodes;

//handle any meta-equiv refresh redirects
nodes = parser.extractAllNodesThatMatch(new NodeClassFilter(MetaTag.class));
for (int i=0; i<nodes.size(); i++) {


Node node = nodes.elementAt(i);
MetaTag meta = (MetaTag)node;
String httpEquiv = meta.getHttpEquiv();
if (httpEquiv != null &&
httpEquiv.toLowerCase().equals("refresh")) {

String content = meta.getMetaContent();
Pattern pattern = Pattern.compile("\\d+\\;url\\=(.*)");
Matcher matcher = pattern.matcher(content);
if (matcher.find()) {

url = matcher.group(1);
urlCon = (HttpURLConnection)new URL(url).openConnection();
parser = new Parser(urlCon);

}

}

}

building jaunty using git

Day 1:

Yesterday, I re-visited the task of building the kernel for ubuntu-jaunty. Obtaining the source using git is covered here. Compiling the source is covered here.

Snag1:
I was on Hardy (not Intrepid) and Hardy repositories do not have a needed utility makedumpfile. After a few hours of compiling, the build failed as it couldn't find makedumpfile. You can save yourself the trouble and manually install the package from here. Once you download the makedumpfile package, you can install it like so:

dpkg -i /path/to/file/makedumpfile_1.2.9-0ubuntu3_i386.deb

So I'm now starting a build again, but this time, to save some time, I will build just the generic flavor, like so:

AUTOBUILD=1 NOEXTRAS=1 fakeroot debian/rules binary-generic

I'm also removing the source directory and cloning the repository fresh. This is to
make sure that there are no conflicts with any files left over from the previous
build.

Day 2:
So today I got the build to go all the way. It generated two debs - the linux image
and headers respectively. I installed the image like so:

dpkg -i linux-image-2.6.28-4-generic_2.6.28-4.5_i386.deb

Since I had modified the /boot/grub/menu.1st file, this gave me the option of overwriting
it or keeping the old. I kept the old file as there were entries to all my
kernel versions I wanted to keep. Later I manually added the stanza
for the new kernel to /boot/grub/menu.1st like so:

title Ubuntu 8.04.1, git kernel 2.6.28-4-generic
root (hd0,0)
kernel /vmlinuz-2.6.28-4-generic root=UUID=b79346d4-7799-43ae-a033-99d70b6784be ro quiet splash
initrd /initrd.img-2.6.28-4-generic
quiet

Snag 2:
Then I tried installing the headers like so:

dpkg -i linux-headers-2.6.28-4-generic_2.6.28-4.5_i386.deb

But this failed as there was a dependency on linux-headers-2.6.28-4 which was not
present on the system. I found it here. I installed it like so:

dpkg -i linux-headers-2.6.28-4_2.6.28-4.5_all.deb

After that, I could successfully install the headers.

After which, I could reboot into the new kernel.

Wireless:

However my wireless connection did not work. Looking at the dmesg output I found these:

thushara@gini-sisila:~$ dmesg | grep acx
[ 12.782440] acx: Loaded combined PCI/USB driver, firmware_ver=default
[ 12.782450] acx: compiled to use 32bit I/O access. I/O timing issues might occur, such as non-working firmware upload. Report them
[ 12.782532] acx_pci 0000:00:0b.0: PCI INT A -> Link[LNKD] -> GSI 10 (level, low) -> IRQ 10
[ 12.782758] acx: found ACX111-based wireless network card at 0000:00:0b.0, irq:10, phymem1:0xE2020000, phymem2:0xE2000000, mem1:0xe0edc000, mem1_size:8192, mem2:0xe08c0000, mem2_size:131072
[ 12.783790] acx: loading firmware for acx1111 chipset with radio ID 16
[ 12.783801] acx_pci 0000:00:0b.0: firmware: requesting acx/1.2.1.34/tiacx111c16
[ 20.290162] acx: firmware image 'acx/1.2.1.34/tiacx111c16' was not provided. Check your hotplug scripts
[ 20.290177] acx_pci 0000:00:0b.0: firmware: requesting acx/1.2.1.34/tiacx111
[ 20.772774] acx: firmware image 'acx/1.2.1.34/tiacx111' was not provided. Check your hotplug scripts
[ 20.772786] acx: reset_dev() FAILED
[ 20.772840] acx_pci 0000:00:0b.0: PCI INT A disabled
[ 20.788119] acx_pci: probe of 0000:00:0b.0 failed with error -5
[ 20.788361] usbcore: registered new interface driver acx_usb

So it looked like the firmware files were missing. Doing a quick `locate tiacx` listed
that the firmware files were available on my previous kernel lib path but not on
the current kernel firmware path.

So I created the firmware directory for the acx driver under the directory for
the new kernel and copied just the single file mentioned by the dmesg to that
location, like so:

sudo cp /lib/firmware/2.6.24-19-generic/acx/1.2.1.34/tiacx111c16 /lib/firmware/`uname -r`/acx/1.2.1.34

After this, I could reboot into the new kernel and the wireless worked.

Java has no unsigned types

I came across this today as I need to create a hash that maps a tuple to an object. The tuple comprises of two keys. Each key is in reality an unsigned integer (32 bit). I was hoping to construct a key for the hash by shifting one key 32 bits and ORing them together to make a 64bit key.

A good article on this can be found here.

The problem with not being able to use an unsigned type supported by the language is the introduction of subtle bugs.

Here's an example:

int x = 0x865a672b;
long y = (0x12345678<<32) | x

1) you can shift an integer (32 bit value) a maximum of 31 bits. in fact if you were to do:

val << shift, where val is an integer, java internally does val << (shift%32)
Thus in this case, 0x12345678 is not shifted at all.

2) so the 0x12345678 never gets promoted to a 64-bit long. Thus we expected 0x1234567800000000, but we have 0x12345678.

3) 0x865a672b is a negative number (as it is an integer). So the 64-bit representation would be 0xffffffff865a672b

4) so we're in fact performing 0x12345678 | 0xffffffff865a672b = 0xffffffff967e777b

So how about this? (We're using longs now)

long x = 0x865a672b;
long y = (0x12345678L<<32) | x;

This gives us 0xffffffff865a672b which is still not what we want, but close. So what is happening here is that even though we declared x to be a long, 0x865a672b is treated as an integer, and java converts that negative integer to a long and in the process keeps the sign (-ve). So x is really 0xffffffff865a672b. So when we OR this to 0x1234567800000000, the high order 32 bits are lost.

Ok, last try:

long x = 0x865a672bL;
long y = (0x12345678L<<32) | x;

Notice we're coercing 0x865a672bL to be a long (the L at the end). This works, giving us 0x12345678865a672b. Enough already!

So if you want to work with unsigned 32-bit values in java, always work in long data type and convert any literal to long with an L at the end.

apt-get structure of repo listing

http://www.ibiblio.org/gferg/ldp/giles/repository/repository-2.html

Java string replace is slow

Java String.replace uses regular expressions and can be slow, especially for large strings requiring many replacements.

If you don't need regular expression replacement, you can use this function I wrote that performs better.

static public String replaceString(String orig, String search, String replace) {
  int cap = replace.length() > search.length() ? orig.length() + (replace.length() - search.length())*20 : orig.length();
  StringBuilder out = new StringBuilder(cap);

  int prev = 0;
  CharSequence okPart;
  for (int i=orig.indexOf(search); i!=-1; i=orig.indexOf(search, prev)) {
    okPart = orig.subSequence(prev, i);
    out.append(okPart).append(replace);
    prev = i + search.length();
  }
  if (prev < orig.length()) {
    okPart = orig.subSequence(prev, orig.length());
    out.append(okPart);
  }
  return out.toString();
}

Here is the performance data with Java String.replace:



Here is the performance data with the new function:



Interpreting the data:

With String.replace, time inside doFlowAction() was broken down 71/29% for enQueueWBImpression/getCPCWidgetCache. The hottest function inside getCPCWidgetCache was String.replace that accounted for 93% of the time.

With the new replacement function,
With String.replace, time inside doFlowAction() was broken down 89/11% for enQueueWBImpression/getCPCWidgetCache. This reduction of time spent inside getCPCWidgetCache (a 62% gain) is due to the replacement of the String.replace with the new string function.

Find the libraries your executable needs

LD_DEBUG is an environment variable that can be used to look under the hood of the Linux loader.
For ex: setting LD_DEBUG to libs will show you what the loader is doing to find the shared libraries needed to run an executable.

output:

mpire@sandbox mpire $ LD_DEBUG=libs ls
23994: find library=libacl.so.1; searching
23994: search path=/site/mpire/sys/mysql/lib/mysql/tls/i686/m ....

mysql - find missing rows in a join

The left join can be used to get all rows from a secondary table matching the rows of the first table, along with those rows from the first table including any rows that doesn't find a match on the secondary table. for rows with no match, NULL would be returned on the secondary table column. this can be useful when you want to find the rows that have no match:

select a.id, b.id from a left join b on (a.secid=b.id) where b.id is null;

And most times, you want to correct this by providing a good value for the NULL column in the secondary table. If you want to set all those rows with a NULL column to the same value, you can do:

update a left join b on (a.secid=b.id) set a.secid=12629002 where b.id is null;