Tuesday, January 14, 2014

telnet relocation error: symbol krb5int_labeled_fopen

Ever had telnet not work on a machine? It happened to me recently, on a Centos 5.8, with this error message:

telnet: error: relocation error: symbol krb5int_labeled_fopen, version krb5support_0_MIT not defined in file libkrb5support.so.0 with link time reference (fatal)

Googling hinted at a conflict in the shared library providing kerberos authentication. So I ran telnet under LD_DEBUG=all :

LD_DEBUG=all telnet host port

which, showed me the problem:

29655:    symbol=krb5int_labeled_fopen;  lookup in file=/usr/local/greenplum-db/lib/libkrb5support.so.0 [0]
29655:    telnet: error: relocation error: symbol krb5int_labeled_fopen, version krb5support_0_MIT not defined in file libkrb5support.so.0 with link time reference (fatal)

So, an installation of greenplum had inserted its version of the kerberos library ahead of the search path for libraries the linux loader uses. The kerberos version of the library did not export the said function.

This all can be verified quickly :

login@host ~$ nm -D /usr/local/greenplum-db/lib/libkrb5support.so.0 | grep krb5int_labeled_fopen

login@host ~$ nm -D /usr/lib64/libkrb5support.so.0 | grep krb5int_labeled_fopen
00000033aea040b0 T krb5int_labeled_fopen

The greeenplum installation was using the LD_LIBRARY_PATH to allow it preferential status, so inserting the /usr/lib64 before it, was sufficient to help telnet find the right library.

login@host ~$ export LD_LIBRARY_PATH=/usr/lib64:$LD_LIBRARY_PATH

login@host ~$ telnet host port
Trying host...
Connected to host (ip).
Escape character is '^]'.

Friday, November 22, 2013

A case of Occam's razor

I wanted to write about a seemingly bizarre issue to do with a web page fetch that ultimately proved to be none other than another validation of the Occam's razor, which is simply that the simplest explanation to a problem is generally the right one.

So, to give some background, I'm involved in doing some statistical calculations over a large number of web pages and this has the side effect of highlighting web pages that deviate from the norm. So I end up going through many web pages that stand out from the pack at first glance.

The fetcher I use talks HTTP directly, and deals leniently with the web servers out there that don't always implement HTTP according to spec. On this particular occasion, one web site : http://hairtype.naturallycurly.com responded to the fetcher with content that was nowhere close to what the browser retrieved.

Let me post here what the HTML looked like:

<html lang="en">
    <title>PHP Application - AWS Elastic Beanstalk</title>
    <link href="http://fonts.googleapis.com/css?family=Lobster+Two" rel="stylesheet" type="text/css"></link>
    <link href="https://awsmedia.s3.amazonaws.com/favicon.ico" rel="icon" type="image/ico"></link>
    <link href="https://awsmedia.s3.amazonaws.com/favicon.ico" rel="shortcut icon" type="image/ico"></link>
    <!--[if IE]><script src="http://html5shiv.googlecode.com/svn/trunk/html5.js"></script><![endif]-->
    <link href="/styles.css" rel="stylesheet" type="text/css"></link>
    <section class="congratulations">
Your AWS Elastic Beanstalk <em>PHP</em> application is now running on your own dedicated environment in the AWS&nbsp;Cloud<br />

        You are running PHP version 5.4.20<br />


    <section class="instructions">
What's Next?</h2>
<li><a href="http://docs.amazonwebservices.com/elasticbeanstalk/latest/dg/">AWS Elastic Beanstalk overview</a></li>
<li><a href="http://docs.amazonwebservices.com/elasticbeanstalk/latest/dg/create_deploy_PHP_eb.html">Deploying AWS Elastic Beanstalk Applications in PHP Using Eb and Git</a></li>
<li><a href="http://docs.amazonwebservices.com/elasticbeanstalk/latest/dg/create_deploy_PHP.rds.html">Using Amazon RDS with PHP</a>
<li><a href="http://docs.amazonwebservices.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html">Customizing the Software on EC2 Instances</a></li>
<li><a href="http://docs.amazonwebservices.com/elasticbeanstalk/latest/dg/customize-containers-resources.html">Customizing Environment Resources</a></li>
AWS SDK for PHP</h2>
<li><a href="http://aws.amazon.com/sdkforphp">AWS SDK for PHP home</a></li>
<li><a href="http://aws.amazon.com/php">PHP developer center</a></li>
<li><a href="https://github.com/aws/aws-sdk-php">AWS SDK for PHP on GitHub</a></li>

    <!--[if lt IE 9]><script src="http://css3-mediaqueries-js.googlecode.com/svn/trunk/css3-mediaqueries.js"></script><![endif]-->

This is nowhere close to the HTML retrieved by the browser. You can try it. The web page is about hair products.

My experience is that sometimes, based on the HTTP headers and originating IP, some web servers can return different content. Sometimes, the server has identified an IP as a bot and decided to return an error response or an outright wrong page.

So I tested the theory of the IP by running the fetcher from a different network, with a different outgoing IP. This time, the correct page was retrieved. Then I used curl to retrieve the page from the same network that had given me the incorrect page. To my surprise, curl retrieved the correct page. curl got the correct page from both networks.

This was quite puzzling. I thought that perhaps the web server might have done some sophisticated finger printing and thus having identified the User Agent and maybe other headers the fetcher was using had decided to send it a wrong page.

So using wireshark, I captured all the HTTP headers sent by the fetcher. Another team member then used curl, specifying these same headers.

curl -H 'User-Agent: rtw' -H 'Host: hairtype.naturallycurly.com' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-us,en;q=0.5'  -H 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7' -H 'Keep-Alive: 115' -H 'Connection: keep-alive' -H 'Accept-Encoding: gzip,deflate' http://hairtype.naturallycurly.com

I was positive that curl would then fail. But of course it still returned the correct page. So my theory of the sophisticated finger printing was wrong - or maybe it was even more sophisticated that I thought. I was stumped.

And then I realized, that I had missed looking at a very crucial piece of data in this whole operation. The IP the fetcher used to get the page. The first thing the fetcher does is to resolve the IP and since the DNS query can be expensive and we do lots of those, the IP is retrieved from a memcached instance if it is available. An IP may be cached for a number of hours. From the fetcher logs, I could see the IP that it was using:

DNS resolved from cache hairtype.naturallycurly.com -> /

But as dig showed, that was the incorrect IP :

>>$ dig hairtype.naturallycurly.com
; <<>> DiG 9.3.6-P1-RedHat-9.3.6-20.P1.el5 <<>> hairtype.naturallycurly.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28108
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 4, ADDITIONAL: 4

;hairtype.naturallycurly.com.    IN    A

hairtype.naturallycurly.com. 300 IN    CNAME    secure-nc-04-2015-1845606936.us-east-1.elb.amazonaws.com.
secure-nc-04-2015-1845606936.us-east-1.elb.amazonaws.com. 60 IN    A
secure-nc-04-2015-1845606936.us-east-1.elb.amazonaws.com. 60 IN    A

us-east-1.elb.amazonaws.com. 1703 IN    NS    ns-1119.awsdns-11.org.
us-east-1.elb.amazonaws.com. 1703 IN    NS    ns-1793.awsdns-32.co.uk.
us-east-1.elb.amazonaws.com. 1703 IN    NS    ns-235.awsdns-29.com.
us-east-1.elb.amazonaws.com. 1703 IN    NS    ns-934.awsdns-52.net.

ns-235.awsdns-29.com.    92612    IN    A
ns-934.awsdns-52.net.    92612    IN    A
ns-1119.awsdns-11.org.    92612    IN    A
ns-1793.awsdns-32.co.uk. 92510    IN    A

;; Query time: 11 msec
;; WHEN: Fri Nov 22 12:40:20 2013
;; MSG SIZE  rcvd: 345

All that remained now was to validate this - far simpler - hypothesis. It was trivial to do so, all I had to do was remove the domain->IP maping from memcached.

>>$ telnet localhost 11211
Connected to localhost.localdomain (
Escape character is '^]'.
get hairtype.naturallycurly.com
VALUE hairtype.naturallycurly.com 4096 4
delete hairtype.naturallycurly.com
get hairtype.naturallycurly.com
Connection closed by foreign host.

This time, the fetcher logs showed that indeed, it was picking the correct IP. And of course it fetched the correct page with all the hair product details.

DNS resolved hairtype.naturallycurly.com -> /

So once again, I was reminded of the Occam's Razor and how important it is to

1. Remember all the assumptions we make about how a certain software system works.
2. Validate all the assumptions, starting with the simplest first.

 Happy debugging the Net!

Wednesday, November 06, 2013

pretty print compressed JSON file

The highly useful json.tool stops short of parsing compressed JSON files. In particular, if you use the json-smart*.jar to produce your json files, you are out of luck with json.tool. But you can use jsonlint like this to get a readable view of your json file :

[~] echo '{json:"obj"}' | python -mjson.tool
Expecting property name: line 1 column 2 (char 1)
[~] echo '{json:"obj"}' | jsonlint -p 2> /dev/null
  json: "obj"

Can't find SymPy with iPython / iPython Notebook on Mac 10.8.2 (Mountain Lion)

SymPy allows you to see mathematical formulas in their hand written representation. It is just not easy to install on the Mac, at least on the Mountain Lion. It has to do with the package location for SymPy. Python finds the package, but not iPython and of course the notebook. So after installing SymPy, trying to import the module will fail like this:

ImportError: No module named sympy

However, inside the python prompt, I could import SymPy. And checking on the package path, showed where python was looking for, to find SymPy:
>>> import sympy
>>> print sympy.__file__

When I checked the module search path inside iPython, the list was shorter and the /opt/local path was not there :
In [7]: IPython.utils.module_paths.find_module("sympy")

In [8]:

In [10]: sys.path

In [11]:

So I copied the package over to /Library/Python/2.7/site-packages/sympy so that iPython could find it:

sudo cp -R /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sympy /Library/Python/2.7/site-packages/

In [13]: IPython.utils.module_paths.find_module("sympy")
Out[13]: '/Library/Python/2.7/site-packages/sympy'

In [14]: from IPython.display import display

In [15]: from sympy.interactive import printing

In [16]: printing.init_printing()

In [18]: from __future__ import division

In [19]: import sympy as sym

In [20]: from sympy import *

In [22]: x, y, z = symbols("x y z")

In [23]: Rational(3,2)*pi + exp(I*x) / (x**2 + y)
3⋅π    ℯ    
─── + ──────
 2     2    
      x  + y

Thursday, October 17, 2013

HTTP Request : Host header must be lower cased

In theory, the case of values specified in most HTTP Request headers is insignificant. But there are servers that do look for a specifically lower case Host header. These servers will return incorrect results if the case was different.

If you're building your own HTTP requests, and want to get back data that a typical browser would get, it would be a good idea to lower case the Host header before sending it to the server.

A case in point is http://www.BestBuys.com

Try to fetch the page with curl, like this : curl http://www.BestBuys.com

You do get a page, but look at it closely, it is an error page.

    <h1>PPI Exception (PDOException)</h1>
    <div><strong>File:</strong> /data/www_bestbuys_com/releases/20131017195710/PPI/Vendor/Doctrine/Doctrine/DBAL/Driver/PDOConnection.php</div>
    <div><strong>Line:</strong> 36</div>
    <div><strong>Message:</strong> SQLSTATE[HY000] [2002] No such file or directory</div>
Do this with wireshark running and observe the Host header :

Host: www.BestBuys.com\r\n

Now use curl, but specify a lowercased Host header :

curl -H "Host: www.bestbuys.com" http://www.BestBuys.com

Then you get the correct page. Browsers, understanding the imperfect implementations of web servers out there always lower case the Host. You can check this with wireshark. Try to get the page using any modern browser and look at the Host header in wireshark.

Here is another example of where the practical approach is not used in a very common library used to fetch web pages.

Wednesday, June 19, 2013

Java : remove specified characters from a string (and quickly!)

Recently, I had to remove extraneous carriage returns and line feeds from URLs on a path that had real-time performance concerns. There was already some code that did this job, albeit inefficiently :

    public static String remove(final String string,
                                final char remove)
        if (string == null) throw new IllegalArgumentException("string is null");

        int index = 0;
        final StringBuilder stringBuilder = new StringBuilder(string);
        while ((index = indexOf(stringBuilder, remove, index)) != -1) {
        return stringBuilder.toString();

This code, is inefficient as it forces the StringBuilder.deleteCharAt to shift all characters in the buffer left, each time a character that need to be removed is found. Here is how StringBuilder.deleteCharAt is implemented :
764 public AbstractStringBuilder deleteCharAt(int index) {
765         if ((index &lt; 0) || (index &gt;= count))
766             throw new StringIndexOutOfBoundsException(index);
767         System.arraycopy(value, index+1, value, index, count-index-1);
768         count--;
769         return this;
770     } 

So I searched around for a hopefully more efficient implementation of this basic function. There was a StringUtils (apache commons lang) function that looked hopeful. However, it was not really meant to remove characters from a string, but to replace them: StringUtils.replaceChars(String str, String searchChars, String replaceChars) Here is how that looks like:
public static String replaceChars(String str, String searchChars, String replaceChars) {
4168        if (isEmpty(str) || isEmpty(searchChars)) {
4169            return str;
4170        }
4171        if (replaceChars == null) {
4172            replaceChars = EMPTY;
4173        }
4174        boolean modified = false;
4175        int replaceCharsLength = replaceChars.length();
4176        int strLength = str.length();
4177        StrBuilder buf = new StrBuilder(strLength);
4178        for (int i = 0; i < strLength; i++) {
4179            char ch = str.charAt(i);
4180            int index = searchChars.indexOf(ch);
4181            if (index >= 0) {
4182                modified = true;
4183                if (index < replaceCharsLength) {
4184                    buf.append(replaceChars.charAt(index));
4185                }
4186            } else {
4187                buf.append(ch);
4188            }
4189        }
4190        if (modified) {
4191            return buf.toString();
4192        }
4193        return str;
4194    }

The code does not optimize for the case where the resulting string would be shorter than the original, so this code is likely to be slow. Next, I found a Guava function in com.google.common.base.CharMatcher : String removeFrom(CharSequence sequence). That used quite an interesting algorithm as follows:
  public String removeFrom(CharSequence sequence) {
    String string = sequence.toString();
    int pos = indexIn(string);
    if (pos == -1) {
      return string;

    char[] chars = string.toCharArray();
    int spread = 1;

    // This unusual loop comes from extensive benchmarking                                                                                
    OUT: while (true) {
      while (true) {
        if (pos == chars.length) {
          break OUT;
        if (matches(chars[pos])) {
        chars[pos - spread] = chars[pos];
    return new String(chars, 0, pos - spread);

It keeps track of the distance between the source and destination in the "spread" variable. However, this means that for each left shift, it needs to do a subtraction. Since these all seemed less than optimal, I decided to code my own. This was my first function, where I try to keep track of the first index for the destination.
    public static String remove2(final String string, final char... chars) {
        char[] arr = string.toCharArray();
        int dst = -1;
        for (int src=0; src<arr.length; src++) {
            boolean rm = false;
            for (char c : chars) {
                if (c == arr[src]) {
                    rm = true;
                    if (dst == -1)
                        dst = src; //set first dst pos 
            if (!rm && dst != -1) {
                arr[dst++] = arr[src];
        return dst == -1 ? string : new String(arr, 0, dst);

Upon a hunch that the JVM should be smart enough to not copy a value from an address to itsef, I then decided to remove the check for the first destination index. Here is that version:
   public static String remove3(final String string, final char... chars) {
        char[] arr = string.toCharArray();
        int dst=0;
        for (int src=0; src<arr.length; src++) {
            boolean rm = false;
            for (char c : chars) {
                if (c == arr[src]) {
                    rm = true;
            if (!rm) {
                arr[dst++] = arr[src];
        return dst == string.length() ? string : new String(arr, 0, dst);

Next I benchmarked the different functions with a simple test, using just one input. Depending on input, your results may vary, so it best to work with inputs you are likely to see in your scenario.

remove : 20.092s
Guava : 25.164s
apache commons langs : 1m33.573s
remove2 : 19.272s
remove3 : 12.109s

So for this simple test, Guava was worse than our hand-optimized remove3(). The apache common langs function took a big hit as it was not optimizing for removal. I'm curious why the Googlers went with their approach on Guava. It may be that all JVMs are not as smart as to prevent a dumb copy from the same address to itself, and Guava makes sure such code does not have a chance to get to the JVM, since the variable "spread" always being greater than 0, this assignment can never be from one address to itself :

chars[pos - spread] = chars[pos];

Wednesday, June 05, 2013

Java: splitting on a single character

If you want to split a Java String on a single character, on a compute intensive path in your code, you might want to stay clear of String.split. The JDK function uses a regular expression for splitting and before JDK 1.7, the String.split had no optimization for single characters.

An optimization was introduced in JDK 1.7, but if your split character happens to have special meaning in a regular expression (ex: ^ |), then the optimization will not apply.

I used org.apache.commons.lang.StringUtils.split to gain a roughly 3X advantage over the split call used in our servers.

Here is the performance test:

import org.apache.commons.lang.StringUtils;

public class TSplit {

    public static void main(String[] args) {
        if (args.length==0) {
            System.err.println("TSplit jdk|nojdk");
        String var = "here|is|a|string|that|must|be|split";
        if (args[0].compareTo("jdk")==0) {
            for (int i=0;i<10000000;i++) {
                String[] splits = var.split("\\|");
        } else {
            for (int i=0;i<10000000;i++) {
                String[] splits = StringUtils.split(var, '|');
The results from the test :
[~/] time java -cp `echo /path/to/jars/*.jar|tr ' ' :` TSplit jdk

real 0m16.027s
user 0m16.245s
sys 0m0.412s
[~/] time java -cp `echo /path/to/jars/*.jar|tr ' ' :` TSplit nojdk

real 0m5.354s
user 0m5.395s
sys 0m0.304s
As this post shows, Users who encountered these problems pre-1.7 have sometimes hacked their code to even pre-compile the single split character to a regular expression. This unfortunately means, that if and when they upgrade to 1.7, the optimization that Sun added will have no effect.