bash: single line for loop

run commands on multiple files at once:

[~/hadoop-src] for f in *; do echo $f; done
common
hdfs
mapreduce
[~/hadoop-src]for f in *; do svn up $f/trunk; done
At revision 787534.

Fetching external item into 'hdfs/trunk/src/test/bin'
External at revision 787534.

At revision 787534.

Fetching external item into 'mapreduce/trunk/src/test/bin'
External at revision 787534.

At revision 787534.
[~/hadoop-src]

date -d is different from Linux to Mac

familiar with:

date -d '1 hour ago'

well, it will work on the Linux command line, but no such luck on the Mac

here is the code you need if your script is to work on both OSes:

OS=`uname -a`
if [[ $OS == Darwin* ]]
then
    TODAY=`date -v-1H +"%Y-%m-%d.%H"`
else
    TODAY=`date -d '1 hour ago' +"%Y-%m-%d.%H"`
fi

Mac Office Excel - all text exports appear as one line

If you use Mac Office to convert a spread-sheet to text, you will see one long line with any Unix utility. The reason is that Office inserts the windows carriage return (^M) with no line feed.

To fix this, bring up the text file inside vi editor type:

:s/^V^M/^V^M/g

this will appear on the editor like

:s/^M/^M/g

as ^V is really an escape for control characters

Downgrading Subversion Working Copy

Sometimes, an IDE coupled to SVN (ex: Intellij) might upgrade your subversion working copy to the latest SVN server format that is incompatible with the command line svn client you have. In this case, if your distro has a command line svn of the latest version, you are probably fine. But if you don't have such luck (for ex: being a MacOS user who has v 1.4 of svn command line), you might want to downgrade the working copy using instructions here.

Hadoop - reading large lines (several MB) is slow

I ran into a performance issue running a Hadoop map/reduce job on an input that at times contained lines as long as 200MB. The issue was in org.apache.hadoop.util.LineReader.

LineReader uses org.apache.hadoop.io.Text to store potentially large lines of text. Unfortunately Text class does not behave well for large text.

Here is the yourKit profile of a simple block of code using Text class:



Here is a profile when Text is replaced with ByteArrayOutputStream:



Notice the Text.append version took 10 times longer to run.

I could get my map/reduce task that initially took over 20 minutes and crashed (as hadoop TaskTracker was timing out on child tasks that took too long) to work under 30s, with this simple change to LineReader:

  public int readLine(Text str, int maxLineLength,
                      int maxBytesToConsume) throws IOException {
    str.clear();
    boolean hadFinalNewline = false;
    boolean hadFinalReturn = false;
    boolean hitEndOfFile = false;
    int startPosn = bufferPosn;
    long bytesConsumed = 0;
    ByteArrayOutputStream os = new ByteArrayOutputStream();
    outerLoop: while (true) {
      if (bufferPosn >= bufferLength) {
        if (!backfill()) {
          hitEndOfFile = true;
          break;
        }
      }
      startPosn = bufferPosn;
      for(; bufferPosn < bufferLength; ++bufferPosn) {
        switch (buffer[bufferPosn]) {
        case '\n':
          hadFinalNewline = true;
          bufferPosn += 1;
          break outerLoop;
        case '\r':
          if (hadFinalReturn) {
            // leave this \r in the stream, so we'll get it next time
            break outerLoop;
          }
          hadFinalReturn = true;
          break;
        default:
          if (hadFinalReturn) {
            break outerLoop;
          }
        }        
      }
      bytesConsumed += bufferPosn - startPosn;
      int length = bufferPosn - startPosn - (hadFinalReturn ? 1 : 0);
      length = (int)Math.min(length, maxLineLength - os.size());
      if (length >= 0) {
        os.write(buffer, startPosn, length);
        LOG.info("os.size= " + os.size() + " just wrote from " + startPosn + " to " + length + " bytes");
      }
      if (bytesConsumed >= maxBytesToConsume) {
        str.set(os.toByteArray());
        return (int)Math.min(bytesConsumed, (long)Integer.MAX_VALUE);
      }
    }
    LOG.info("finished reading line");
    int newlineLength = (hadFinalNewline ? 1 : 0) + (hadFinalReturn ? 1 : 0);
    if (!hitEndOfFile) {
      bytesConsumed += bufferPosn - startPosn;
      int length = bufferPosn - startPosn - newlineLength;
      length = (int)Math.min(length, maxLineLength - os.size());
      if (length > 0) {
        os.write(buffer, startPosn, length);
      }
    }

    str.set(os.toByteArray());
    return (int)Math.min(bytesConsumed, (long)Integer.MAX_VALUE);
  }

Stop Emacs spash screen

Put this in your ~/.emacs :

;; no splash screen
(setq inhibit-startup-message t)


You can also start emacs with -Q option to avoid the splash screen.

seems this is a debated point!

Flash plugin for FireFox on Ubuntu 8.04.1

The installation doesn't work without a minor tweak after the fact:

thushara@agni:~$ sudo ln -s /usr/lib/libssl3.so.1d /usr/lib/libssl3.so
thushara@agni:~$ sudo ln -s /usr/lib/libplds4.so.0d /usr/lib/libplds4.so
thushara@agni:~$ sudo ln -s /usr/lib/libplc4.so.0d /usr/lib/libplc4.so
thushara@agni:~$ sudo ln -s /usr/lib/libnspr4.so.0d /usr/lib/libnspr4.so
thushara@agni:~$ sudo ln -s /usr/lib/libnss3.so.1d /usr/lib/libnss3.so


how did i figure this out? - just start firefox on the command-line and watch it complain:

thushara@agni:~$ firefox
LoadPlugin: failed to initialize shared library libXt.so [libXt.so: cannot open shared object file: No such file or directory]
LoadPlugin: failed to initialize shared library libXext.so [libXext.so: cannot open shared object file: No such file or directory]
LoadPlugin: failed to initialize shared library /usr/lib/adobe-flashplugin/libflashplayer.so [libnss3.so: cannot open shared object file: No such file or directory]
LoadPlugin: failed to initialize shared library /usr/lib/adobe-flashplugin/libflashplayer.so [libnss3.so: cannot open shared object file: No such file or directory]

out of these, our problem is with the libflashplayer.so, seems like a dependent module (libnss3.so) is not being found.

so now, use ld to see the dependencies for the libflashplayer.so :

thushara@agni:~$ ld /usr/lib/adobe-flashplugin/libflashplayer.so
ld: warning: libnss3.so, needed by /usr/lib/adobe-flashplugin/libflashplayer.so, not found (try using -rpath or -rpath-link)
ld: warning: libsmime3.so, needed by /usr/lib/adobe-flashplugin/libflashplayer.so, not found (try using -rpath or -rpath-link)
ld: warning: libssl3.so, needed by /usr/lib/adobe-flashplugin/libflashplayer.so, not found (try using -rpath or -rpath-link)
ld: warning: libplds4.so, needed by /usr/lib/adobe-flashplugin/libflashplayer.so, not found (try using -rpath or -rpath-link)
ld: warning: libplc4.so, needed by /usr/lib/adobe-flashplugin/libflashplayer.so, not found (try using -rpath or -rpath-link)
ld: warning: libnspr4.so, needed by /usr/lib/adobe-flashplugin/libflashplayer.so, not found (try using -rpath or -rpath-link)
ld: warning: cannot find entry symbol _start; not setting start address
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `NSS_CMSMessage_GetContentInfo@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `CERT_GetDefaultCertDB@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `NSS_CMSContentInfo_GetContentTypeTag@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `SSL_ForceHandshake@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `NSS_CMSSignedData_VerifySignerInfo@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `PR_Initialized'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `NSS_IsInitialized@NSS_3.9.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `CERT_DecodeCertFromPackage@NSS_3.4'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `NSS_CMSMessage_Destroy@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `SSL_OptionSet@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `NSS_CMSContentInfo_GetContent@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `SSL_ResetHandshake@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `PR_Read'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `SSL_AuthCertificate@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `SECMOD_CloseUserDB@NSS_3.11'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `CERT_ChangeCertTrust@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `PR_Write'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `NSS_Shutdown@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `PR_Init'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `CERT_DestroyCertificate@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `SSL_ImportFD@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `PR_Now'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `SECMOD_OpenUserDB@NSS_3.11'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `CERT_VerifyCACertForUsage@NSS_3.6'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `PR_ImportTCPSocket'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `NSS_Init@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `SSL_AuthCertificateHook@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `NSS_SetDomesticPolicy@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `NSS_CMSMessage_CreateFromDER@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `NSS_CMSMessage_GetContent@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `NSS_CMSSignedData_SignerInfoCount@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `PK11_FreeSlot@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `NSS_CMSSignedData_ImportCerts@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `PR_GetError'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `NSS_CMSSignedData_VerifyCertsOnly@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `PR_Close'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `SSL_BadCertHook@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `SSL_SetURL@NSS_3.2'
/usr/lib/adobe-flashplugin/libflashplayer.so: undefined reference to `PR_SetSocketOption'


from this, we find that certain libs are missing, or are they? let's just check if they are somewhere sneaky:

thushara@agni:~$ locate libnss3.so
/usr/lib/libnss3.so.1d
thushara@agni:~$ locate libsmime3.so
/usr/lib/libsmime3.so.1d
thushara@agni:~$ locate libssl3.so
/usr/lib/libssl3.so.1d
thushara@agni:~$ locate libplds4.so
/usr/lib/libplds4.so.0d
thushara@agni:~$ locate libplc4.so
/usr/lib/libplc4.so.0d
thushara@agni:~$ locate libnspr4.so
/usr/lib/libnspr4.so.0d

thus we can just symlink our way out of this:

thushara@agni:~$ sudo ln -s /usr/lib/libssl3.so.1d /usr/lib/libssl3.so
thushara@agni:~$ sudo ln -s /usr/lib/libplds4.so.0d /usr/lib/libplds4.so
thushara@agni:~$ sudo ln -s /usr/lib/libplc4.so.0d /usr/lib/libplc4.so
thushara@agni:~$ sudo ln -s /usr/lib/libnspr4.so.0d /usr/lib/libnspr4.so
thushara@agni:~$ sudo ln -s /usr/lib/libnss3.so.1d /usr/lib/libnss3.so

hit youtube and enJoy ~