remove carriage returns from Windows files on Linux

Sometimes you have to work with a file created in Windows from Linux. The file has the "\r" (0x0d) that can cause trouble for some Linux apps.

You can easily remove them using "tr" cmd.

cat windows_file | tr -d "\r"

The above command should write the new file to stdout. Save it to a new file, and this will not have "\r" characters.

What problems may happen
The "\r" character is generally interpreted by terminals to move the cursor to the beggining of the current line. The "\n" character moves the cursor to the next line. Generally, a text file created in Windows will have a "\r\n" sequence at the end of each line.

What would happen if you were to remove the last character from each line and try to display it to the terminal? Now after each line, the cursor would advance to the beginning of the line ("\r") but not advance to the next line. Thus each line will overwrite the other.

This could happen if you have a perl script where 'chomp' is used to remove the last character of each line.

cpan2rpm - perl modules and rpms

Today I discovered this amazingly useful utility. This helps in building the perl module rpms that are necessary for other perl module rpms (dependencies) to install.

If you have an rpm install, the dependent rpms need to be listed in the rpm database on your machine. The number 1 source for perl modules is CPAN but they don't necessarily release rpms for the modules. With the cpan2rpm, you can get the required perl module from CPAN, make the rpm and then install the module with rpm.

The really nice thing about cpan2rpm is that it can locate the module from CPAN automatically, for ex: if you invoke it like this: cpan2rpm XML::Simple, it will locate the XML::Simple module from CPAN, copy it over and build the rpm at /usr/src/redhat/RPMS/noarch/perl-XML-Simple-2.14-1.noarch.rpm.

Sometimes, cpan2rpm won't be able to find the module on CPAN. Then you can manually locate it from CPAN archives, download it and then point cpan2rm at the local gzip file and it will go ahead and build the rpm.

ex:
wget http://search.cpan.org/CPAN/authors/id/S/SB/SBURKE/HTML-Format-2.04.tar.gz
cpan2rpm /home/downloads/HTML-Format-1.23.tar.gz

Rather long winded - instead of simply installing the module, but if a latter part of the install uses rpm, you need to make sure the rpm database knows about the dependencies, so this is a necessary step.

Once again, it is a good thing that there's more than one way to do it!

Backing up DVDs - Elizabeth Town

I used DVDShrink to create a backup of this movie. The movie played fine on the hard disk but after copying it over to a DVD, I found that the DVD did not play past chapter 10.

I then used pgcedit to open the movie files on the hard disk. It immediately found the problem - apparently there is a bug in DVDShrink that creates a slightly corrupted index. Pgcedit offerred to correct the problem and change the DVD files accordingly.

After doing this step, I could burn a successful DVD with Roxio.

I'm not sure if DVDShrink encountered some form of copy protection or if it was actually a bug on DVDShrink. Most recently released movies seem to come up with different and newer DRM techniques.

Hotel Rwanda

This is a well made movie about a man who saved 1200 victims from one of the bloodiest massacres in recent history. The movie is based on the true story of Paul Rusesabagina.

Watching this movie, I couldn't quite come to grips with the limits of human atrocities against humans and the inability of the better-off people in the 1st world to act quickly to avoid such horrors.

Human Resources Watch reports the massacre. Reading it leads me to believe that the reason for the lack of intervention was the economics. The cost it took to save a million lives from terrible violence was too much to bear - for the wealthiest of nations.

There was also the fear of loosing personnel. I copy below:

"But instead of using the peacekeeping troops to stop the genocide, the U.N. sought primarily to protect its soldiers from harm. Dallaire was ordered to make avoiding risk to soldiers the priority, not saving the lives of Rwandans. To do so, he regrouped his troops, leaving exposed the Rwandans who had sought shelter in certain outposts under U.N. protection. In the most dramatic case—for which responsibility may belong to commanding officers in Belgium as much as to Dallaire—nearly one hundred Belgian peacekeepers abandoned some two thousand unarmed civilians, leaving them defenseless against attacks by militia and military. As the Belgians went out one gate, the assailants came in the other. More than a thousand Rwandans died there or in flight, trying to reach another U.N. post."

After exploiting Rwanda from 1916 through 1961, Belgium could not deal with the economics of saving a million Tutsi lives. These were the same people Belgium favored during their regime, the people, who under the Belgian command resorted to extreme violence against the Hutus. Some claim that the later uprising of the Hutus against Tutsis were largely due to the ethnic division enforced by Belgium in their divide-and-rule policies.

What is still not clear to me is why the 1st world did not threaten the genocidal government with economic sanctions. This would not have required much capital. Perhaps they looked at the situation from an economic view-point and just said - "why bother, what is in it for me?" I copy below:

"Discussion about the size, mandate, and strategy for a new peacekeeping force continued until May 17, in part because of U.S. rigidity in applying its new standards for approval of peacekeeping operations, in part because of hesitations sparked by RPF opposition to any intervention. Manoeuvering by nations supplying troops and those supplying equipment consumed another two months, so that the second peacekeeping force landed only after the RPF had defeated the genocidal government. The slowness and ineptness of national and international bureaucracies in mounting the operation was not unusual, nor was the attempt by participating nations to get the most or give the least possible. What was extraordinary was that such behavior continued to be acceptable in the context of genocide, by then openly acknowledged by national and international leaders."

Search your intranet with Nutch!

Our company maintains several wikis for each department. This is where the knowledge of how to set servers up, the milestones and deliverables of projects, new project ideas etc live. There is (actually was) one problem - the wiki was not searchable. The search engine that came with the wiki was pretty much broken. A search would most times hang the browser.

So I was looking around, and found this amazing open source project - Nutch - with which I built the infra-structure for a crawl of a few intranet sites. Within a couple of days, I had the system running and the results were really good.

Then another developer hooked up the search engine to the wiki's search button.

Later on, I increased the crawl to other web based information we have like - the bug database, the system that logs perforce changelists. The results were quite good, now from a single place, we can find a lot of information about an item scattered across many links.

I remember back at Microsoft, someone was always suggesting how all the systems that maintained different datasets should somehow be unified so that from one point, we can gather all the data for one item. This of course is a huge dev/test effort requiring major re-haul of a bunch of working systems. Enter search - a good one - and all this costly development work disappears.

I want to commend the people working in the Nutch project for giving us such a cool set of tools. Keep up the excellent work!

Philosophical differences btn Linux and Windows

A year ago when I was interviewing at Google, and the interviewer (hard core systems type) had finished grilling me about some of the guts of a kernel sub-system, I casually mentioned, that at that level Windows is probably not much different from Linux.

He said that he indeed sees philosophical differences btn the two operating systems even at that level.

A year later, a little wiser, I now ponder about that (I'm now working in a start-up in a Linux environment) and I see these similarities.

The most recent difference I encountered is the nature of file locking. In Windows, locking is mandatory - if a process takes a file lock, no other process will be able to access that file, until the first process gives up the lock. In Linux, the locking by default is advisory - another process can go ahead and access the file still! Only a "well behaved process" that tries to take the same lock before accessing the file, will be blocked until the first process gives up the lock.

At first, I was shocked - in fact somewhat disappointed in Linux which I'd grown to love each day. This actually meant that as a developer of a large distributed system, I had no guarantee that the file on which I have taken a lock is not being mucked around by say the Unix cp program.

So I started coding the locking, reading file stuff up and as I was debugging it discovered something quite pleasant. My program being debugged was modifying the file (after taking a lock) but it was still possible to use 'cat' from another shell to examine the contents of the file.

I realized that in my program, if things went wrong and a file got locked and the lock never got released, an admin can still go ahead and remove the file from the system - no reboots necessary. On the other hand, on Windows this is somewhat more complicated as the operating system can disallow the operation on a locked file.

This shows the protective nature of Windows that allows for some predictability by giving up on flexibility. Linux takes the opposite strategy.

From napalm to white phosphoros

The US military now admits it used white phosphoros against Iraqis.
Since chemical weapon usage is banned in warfare, they have come up
with the notion that "white phosphorous is not a chemical weapon".

http://www.timesonline.co.uk/article/0,,7374-1875728,00.html

Note how CIA mentions white phosphorous being a chemical weapon when
Saddam Hussein used it against Kurds in 1991.
http://thinkprogress.org/2005/11/21/phosphorus-chemical/

So, the definition can be changed depending on who uses it, it seems.

White phosphorous (like napalm) sticks to the skin and can burn the
flesh to the bone. It is used as an extreme terror tactic in war. An
excerpt from http://blogs.washingtonpost.com/earlywarning/2005/11/trying_the_hole.html:


On November 8, Rainews 24, an Italian satellite news channel of state
controlled RAI, aired a 20 minute documentary Fallujah: The Hidden
Massacre, reporting on the assault of Fallujah in November 2004. A
former American soldier, Army Specialist Jeff Englehart, is quoted
saying that white phosphorus was used intentionally to attack
civilians. Englehart says he "saw the burned bodies of women and
children," some melted "right down to the bone." He is quoted as
saying that he "heard the order to be careful because white phosphorus
was being used."

The documentary shows close-ups of Fallujah civilians, badly burnt,
their skin dissolved or caramelized. An Iraqi biologist in Fallujah is
interviewed, saying "a rain of fire fell on the city," burning
people's flesh, but strangely leaving "their clothes intact."