Saturday, December 24, 2005

Philosophical differences btn Linux and Windows

A year ago when I was interviewing at Google, and the interviewer (hard core systems type) had finished grilling me about some of the guts of a kernel sub-system, I casually mentioned, that at that level Windows is probably not much different from Linux.

He said that he indeed sees philosophical differences btn the two operating systems even at that level.

A year later, a little wiser, I now ponder about that (I'm now working in a start-up in a Linux environment) and I see these similarities.

The most recent difference I encountered is the nature of file locking. In Windows, locking is mandatory - if a process takes a file lock, no other process will be able to access that file, until the first process gives up the lock. In Linux, the locking by default is advisory - another process can go ahead and access the file still! Only a "well behaved process" that tries to take the same lock before accessing the file, will be blocked until the first process gives up the lock.

At first, I was shocked - in fact somewhat disappointed in Linux which I'd grown to love each day. This actually meant that as a developer of a large distributed system, I had no guarantee that the file on which I have taken a lock is not being mucked around by say the Unix cp program.

So I started coding the locking, reading file stuff up and as I was debugging it discovered something quite pleasant. My program being debugged was modifying the file (after taking a lock) but it was still possible to use 'cat' from another shell to examine the contents of the file.

I realized that in my program, if things went wrong and a file got locked and the lock never got released, an admin can still go ahead and remove the file from the system - no reboots necessary. On the other hand, on Windows this is somewhat more complicated as the operating system can disallow the operation on a locked file.

This shows the protective nature of Windows that allows for some predictability by giving up on flexibility. Linux takes the opposite strategy.


Thursday, December 15, 2005

From napalm to white phosphoros

The US military now admits it used white phosphoros against Iraqis.
Since chemical weapon usage is banned in warfare, they have come up
with the notion that "white phosphorous is not a chemical weapon".

http://www.timesonline.co.uk/article/0,,7374-1875728,00.html

Note how CIA mentions white phosphorous being a chemical weapon when
Saddam Hussein used it against Kurds in 1991.
http://thinkprogress.org/2005/11/21/phosphorus-chemical/

So, the definition can be changed depending on who uses it, it seems.

White phosphorous (like napalm) sticks to the skin and can burn the
flesh to the bone. It is used as an extreme terror tactic in war. An
excerpt from http://blogs.washingtonpost.com/earlywarning/2005/11/trying_the_hole.html:


On November 8, Rainews 24, an Italian satellite news channel of state
controlled RAI, aired a 20 minute documentary Fallujah: The Hidden
Massacre, reporting on the assault of Fallujah in November 2004. A
former American soldier, Army Specialist Jeff Englehart, is quoted
saying that white phosphorus was used intentionally to attack
civilians. Englehart says he "saw the burned bodies of women and
children," some melted "right down to the bone." He is quoted as
saying that he "heard the order to be careful because white phosphorus
was being used."

The documentary shows close-ups of Fallujah civilians, badly burnt,
their skin dissolved or caramelized. An Iraqi biologist in Fallujah is
interviewed, saying "a rain of fire fell on the city," burning
people's flesh, but strangely leaving "their clothes intact."



Friday, November 04, 2005

Compiling the Linux kernel

I'm writing this post from FireFox running on Linux kernel 2.6.14 that I just compiled, installed and booted into. Feels real good!

The newbies article was a great one to help with this effort. The information on installing the kernel using grub was absent. I use RedHat with grub and I found how to get grub updated with the new kernel following this blog post.

However, there were a couple of things I did differently due to paronia. :) I basically didn't want to allow the possibility of the new install replacing my working Linux kernel so I manually changed grub.conf file. So the steps are as follows:

1. copy kernel src gz file to /usr/src/
2. uncompress the gz file to create /usr/src/linux
3. make menuconfig
4. make bzImage (I had to do this instead of - make zImage due to poor machine spec)
5. make modules
6. make modules_install (This is the step that installs the new modules to /lib/modules/ so it is better to install a version greater than what you have so that there will be no file clobbering)
7. copy the kernel image (/usr/src/linux/arch/i386/boot/bzImage) over to /boot/vmlinuz-2.6.14-0.0000
8. copy /usr/src/linux/System.map to /boot/System.map-2.6.14-0.0000
9. mkinitrd /boot/initrd-2.6.14-0.img 2.6.14 (this step creates the RAM disk image that the kernel uses at early boot time, the second parameter corresponds to the directory where libs were installed; ie lernel_version in /lib/modules/)
10. Now open /etc/grub.conf in your favorite editor and add a stanza similar to what you have for the primary kernel. Change the kernel and initrd parameters to point to the new kernel and the RAM disk image.

Here is my grub.conf relevant parts (before):

title Fedora Core (2.6.12-1.1447_FC4)
root (hd0,0)
kernel /vmlinuz-2.6.12-1.1447_FC4 ro root=/dev/VolGroup00/LogVol00 rhgb quiet
initrd /initrd-2.6.12-1.1447_FC4.img

grub.conf relevant parts (after):

title Fedora Core (2.6.12-1.1447_FC4)
root (hd0,0)
kernel /vmlinuz-2.6.12-1.1447_FC4 ro root=/dev/VolGroup00/LogVol00 rhgb quiet
initrd /initrd-2.6.12-1.1447_FC4.img

title Fedora Core (2.6.14)
root (hd0,0)
kernel /vmlinuz-2.6.14-0.0000 ro root=/dev/VolGroup00/LogVol00 rhgb quiet
initrd /initrd-2.6.14-0.img

11. Turn on loud music and reboot, pick the new kernel and step into a new world ;-)

Thursday, October 06, 2005

Use Linux to copy files across two Windows boxes

I wrote earlier about the difficulties I had trying to share files across 2 machines running XP. 10 minutes ago, I managed to do this using a - wonder of wonders - Linux partition as a go-between.

This is the setup:

I have box A with only XP. The other box B has XP, Linux both. I have a vfat partition on this box that lets me access files regardless of which O/S I boot into.

I just installed a samba server on Linux running on B and managed to access a Linux share from box A running XP.

The amazing thing is that this was something I couldn't do natively inside windows for a few weeks.

Now, the next amazing this is that I'm actually a heavy Windows user/developer. I have worked as a programmer in that O/S for well over 15 years. I actually worked at Microsoft developing both kernel mode and user mode components for XP for the last 6 years. (I left that company 2 months back and switched to a mostly-Linux work environment)

Now I'm a complete newbie to Linux. I mean, I crawl here. But it never ceases to amaze me how versatile and flexible it is. Coupled with Google, it is relatively easy to find information about how to get something working. In contrast, I haven't been able to find information on Windows that easily. 

Saturday, September 10, 2005

A case study in claims against corporations

I want to share with you all an interesting experience I had in a dispute with Amex Travel. It was instructive to see how the system operates to make it increasingly difficult for an ordinary person to file a case against a big company, even when the amount of money is trivial.

When I was vacationing in LA, I had a rental through Avis, paid with Amex credit card.

On my second day of stay, I wanted to make sure I had accident protection on the card. When I called Amex up to verify this, they said that this was not so, but they can enroll me right then. So I did.

Unfortunately on my way home, I crashed the car against a post. I was thankful for getting the protection on the car. I filed the claim at the airport and came back home.

A month or so later I got a letter from AVIS detailing the damage charges - a little over 1000 $. I called up Amex and explained their liability and they took down the information and promised to look into it. I thought that was the end of it.

A couple of weeks later I got a second notice from Avis, a little more sternly this time, mentioning my liability and hinting at collection agency involvement.

I called up Amex again and they mentioned that the date of crash is reported as 7/17 which was actually the date I rented the car. They mentioned me not having coverage then, as a result my claim would be denied.

I explained to them that I crashed on my last day of stay (7/25) while driving down to the airport. They asked me to get Avis to send them some documentation that will verify this.

I called Avis, told them I must have made a mistake on the accident report and asked them to mention that in writing to Amex, which they did.

Now I was following this closely, calling Amex regularly to see the progress made on this. They continued to say that the claim cannot be honored because of the discrepancy on the date.

This is the time I called up some lawyers. One lawyer was willing to help and took me through how I could fight this in the small claims court. Basically I had to go and see the clerk at the court and file a suit against Amex travel.

Upon going to the court, I learnt that to file against a corporation, I need to find their "Registered Agent". I learnt that corporations have registered agents in different states to handle disputes and perform other related duties. The court clerk handed me a document describing the process I needed to go through to find who the registered agent was for Amex Travel.

It boiled down to calling a number for a government office (secretary of state Office division of Corporate records in Olympia) asking for the information. The number was (360) 753-7115.

I called the number up and the person on the other end told me that Amex was a big corporation with many different branches and that I'd have to find their UBI number first. And I had to call 1-900-463-600 (master licensing) to find that.

At first I thought I had heard the number wrong, but no, it was apparently a 1-900 number.

Rather than burning 3 dollars a minute calling this number (these are of course banned in the office telephone network), I managed to find a web site - www.secstate.wa.gov that had the information I wanted.

Armed with this information, I went back to the court and filed a suit.

A week later I got back a letter through UPS - from the registered agent. They mentioned that they serve thousands of companies but "has no business connection" with any of them, and as a result I can't name them as defendant on the suit. The letter instructed me remove their name as defendant from the suit and to mention the corporation against which I'm making the claim. It also said that they'll be collecting attorney fees from me, if they had to make a court appearance.

At this point I was very confused. The instructions in the document from the court clerk was to note the registered agent as the defendant. I called my lawyer and he said I had done everything right, and asked me to just go to court on that date.

However, I knew something was amiss somewhere. I consulted with a friend (who was a para-legal) who advised me to mention both the registered agent and the corporation on the suit.

I went back to the court and did the changes and mailed the registered agent a copy.

A week later, 3 days before the trial I got a call from Amex asking to settle, which I did for the full amount excluding 20$ court fees.

Now, I could go through with all these steps, partly because I had a flexible schedule at my job. I could take calls, leave office pretty much any time I wanted to. If this happened to someone who doesn't have that flexibility, there's a good chance he would have given up and Amex would have won.

This is just one example on how the legal system that is supposed to ensure fairness to everyone is heavily biased towards protecting corporate assets.

Sunday, May 22, 2005

dual boot XP with Linux

There are many good articles on the Net explainig this process.
Basically, there are two ways to do this:
1. Use ntldtr - NT's boot loader.
2. Use GRUB or LILO - most popular Linux boot loaders.
I just finished an installation of XP with RedHat using method 1. I ran into an issue on which no ready help was available - or as far as google knew, here was none. ;-)

Here are the links that helped me:

Illustrates using ntldr as the boot loader
A more comprehensive description on using ntldr for a dual boot
Explaination for missing hal.dll error

Basically, I had two disk drives and when I installed XP, it called the windows partition E: and the other drive C:.
So I copied the linux boot image to e:\ and added an entry in e:\boot.ini to reflect that.
So e:\boot.ini had this as the last line:

e:\linux.bin="Linux"

When I re-booted, NT boot loader showed me the entry for Linux, but when I selected it, there was a complaint about hal.dll being missing / corrupt or some such.

Turns out, ntldr don't know what e: is!! When I copied linux.bin to c:\ and updated e:\boot.ini to reflect that, everything worked.

Saturday, April 30, 2005

image checksums

When modifying images, it is a good idea to change the checksum as well. There is an API in imagehlp.dll for calculating the checksum from the image bytes.

If you change the image bytes, you should re-calculate the checksum and store in in the image header.

Since I couldn't find a command line tool that could show the original checksum in the header as well as the computed checksum, I wrote this little tool.

Complile this with cl /Zi checksum.cpp /link imagehlp.lib
Usage checksum.exe image_name

ex:
D:\temp>checksum c:\WINDOWS\system32\kerberos.dll
checksum orig 0x00051f00 computed 0x00051f00

D:\temp>

#include <windows.h>
#include <imagehlp.h>
#include <stdio.h>


int __cdecl main(int argc, char** argv){
argc; argv;
if (argc < 2){
printf("Usage: checksum filename\n");
return 1;
}

const char* fn = argv[1];
HANDLE hFile = CreateFile(fn, GENERIC_READ, FILE_SHARE_READ, 0, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);
if (hFile == INVALID_HANDLE_VALUE)
{
printf("culdn't find %s\n", fn);
return 1;
}

DWORD fileSize = GetFileSize(hFile, 0);
if (INVALID_FILE_SIZE == fileSize)
{
printf("couldn't find size of %s\n", fn);
return 1;
}

HANDLE hFileMap = CreateFileMapping(hFile, 0, PAGE_READONLY, 0, 0, 0);
if (hFileMap == 0)
{
printf("culdn't create mapping for %s\n", fn);
return 1;
}
PVOID hBaseFile = MapViewOfFile(hFileMap, FILE_MAP_READ, 0, 0, 0);
if (hBaseFile == 0)
{
printf("culdn't map %s\n", fn);
return 1;
}

DWORD hdr_checksum;
DWORD checksum;
PIMAGE_NT_HEADERS ntHeaders = CheckSumMappedFile(hBaseFile, fileSize, &hdr_checksum, &checksum);
if (!ntHeaders)
{
printf("checksum api failed\n");
return 1;
}

printf("checksum orig 0x%08x computed 0x%08x\n", hdr_checksum, checksum);
return 0;
}

Sunday, April 17, 2005

Inside look into Microsoft TCP/IP connection limit workaround

I couldn't find out any information on how this was hacked. I assume that a binary differ was used against the orginal tcpip.sys vs the patched one. Halvar Flake's bindiff plugin with IDA-Pro, quite probably.

However, after analyzing the fix, I found it instructive to disassemble tcpip.sys to find out what the hack accomplished.

Basically, tcpip.sys (XP SP2) introduces a pointer value in the image that points to the maximum number of concurrent connections. In a couple of places in the code a check is made to see if the current number of connections exceed this value.

The maximum number of connections is set with this instruction:

mov dword ptr [_ActiveOpenProgressThreshold], 0Ah

The hack changes this to:

mov dword ptr [_ActiveOpenProgressThreshold], 0FFFFFEh

The disassembly makes more sense if you have symbols for tcpip.sys. You can obtain this by setting the symbol path to Microsoft public symbol servers:

set _NT_SYMBOL_PATH=SRV*c:\symbols*http://msdl.microsoft.com/download/symbols

Microsoft Windows TCP/IP security limits

Windows XP SP2 introduced a limit on concurrent half-open TCP/IP connections that can be a problem in using P2P apps like eMule, BitTorrent.

http://forum.emule-project.net/lofiversion/index.php/t56016.html describes the general method of dealing with this.

However, there is at least one other version of tcpip.sys that is different. I believe this is part of a more recent windows update. The same hack can be applied but in two different places.

First check if your tcpip.sys has a size of 359,808. If so apply this patch:

Modify tcpip.sys offset 0x130 from C8 27 06 00 to BC 28 06 00

Modify tcpip.sys offset 0x0004f5a2 from 0A 00 00 00 to FE FF FF 00

An automated fix for this is found in http://www.lvllord.de/

I will post a more general description of what we're modifying later in this blog.