Say, there is a file of numbers called "nums" , each number seperated by a newline, and we want to sum the numbers:
cat nums | perl -ne 'BEGIN{$s=0;} chomp; $s+=$_; END {print "$s\n"}'
Friday, December 10, 2010
use perl BEGIN / END blocks for summations
Say, there is a file of numbers called "nums" , each number seperated by a newline, and we want to sum the numbers:
Tuesday, November 23, 2010
bash one liners to setup password-less SSH
ssh user@host "cat >> .ssh/authorized_keys2" < .ssh/id_rsa.pub
This will append the ssh public key on the local machine to the authorized_keys file in the remote machine so that the local machine will in the future be able to ssh to the remote without a password.
If you are setting up multiple machines this way, this one liner is faster than having to ssh into each remote to update the authorized_keys file.
You could build on this to setup multiple machines with a single command:
for m in host1 host2 host3 host4; do ssh user@$m "cat >> .ssh/authorized_keys2" < .ssh/id_rsa.pub; doneMonday, November 15, 2010
C2 A0 characters confusing bash?
user@host:$ ps auxww | grep java
grep: command not found
Here is the hex output of a correct "grep" and an incorrect "grep" command line:
user@host$ hexdump -C /tmp/x
00000000 70 73 20 61 75 78 77 77 20 7c 20 67 72 65 70 20 |ps auxww | grep |
00000010 6a 61 76 61 0a |java.|
00000015
user@host$ hexdump -C /tmp/y
00000000 70 73 20 61 75 78 77 77 20 7c c2 a0 67 72 65 70 |ps auxww |..grep|
00000010 20 6a 61 76 61 0a | java.|
00000016
The second output is the faulty one, notice the characters "C2 A0" cause the problem. A0 is the non-breaking space, and somehow, my keyboard at times produces these instead of "20" for the space character, thus confusing the shell.
This is on an ssh session to Linux 2.6, from a Mac.
Saturday, November 13, 2010
mysql deadlocks with concurrent inserts
The deadlocks occur due to gap locking done by mysql. There are several reasons for gap locking, and in this particular case, it has to do with preserving a unique key constraint on an index. The situation presents itself to us this way: There is a unique key constraint on a column and we are doing an insert. Mysql has to make sure that the lock it takes is sufficient to prevent another concurrent insert from adding a record with the same key, thus breaking the unique key constraint.
Mysql innodb engine performs row locking on inserts. If column A has a unique key constraint, and we are adding the value "bbb" for column A in an insert statement, mysql needs to lock any gap in the index between the two current records where "bbb" will be inserted at.
To illustrate the deadlock, let us start with a table schema:
TABLE vegetable (
id bigint(10) NOT NULL auto_increment,
name varchar(255) NOT NULL,
PRIMARY KEY (id),
UNIQUE KEY uk_name (name)
) ENGINE=InnoDB
Let us assume the existence of these records in the table, and look at them in 'name' index order:
id name
10 ggg
05 jjj
Now, imagine two concurrent connections executing the following inserts in the following order:
Connection 1:
insert ignore into vegetable values(null, "ppp");
For this insert to proceed, connection 1 will lock the gap between "jjj" and "ppp" in the name index.
Connection 2:
insert ignore into vegetable values (null,"iii");
This will require locking the gap after "ggg", upto "iii". Since the lock from connection 1 does not span this, it will succeed.
insert ignore into vegetable values (null, "mmm");
This needs to lock the gap after "jjj" upto "mmm". Since connection 1 has a lock between "jjj" and "ppp", effectively spanning the lock connection 2 is attempting to take, this will block.
Connection 1:
insert ignore into vegetable values (null, "hhh");
This requires the gap lock between "ggg" and "hhh". This will block as it spans the the lock ["ggg" to "iii"] held by connection 2.
Thus we have both connections blocked on each other. This is the deadlock.
Here is a diagram. Transactions to the left are done by Connection 2. Transactions to the right are done by Connection 1. The sequence of transactions is donated by numbers 1) through 4).
Connection 2 Connection1
--------------------------- ggg
G G
A AP
P <------------------- 4) hhh
Lock blocks (deadlock)
2) iii -------------------->
--------------------------- jjj
G G
A
P A
Lock
3) mmm ---------------> P
blocks
L
o
c
k
<--------------------- 1) ppp
You can avoid this if you can order the inserts on each connection on the same direction. The deadlock happens as connection 2 inserts in ascending order of the index, while connection 1 inserts on descending order.
If you can't do this for practical reasons, you could retry the operation. Unless there is a high level of concurrency with a high load on the db where each transaction takes a heavy hit, a simple retry should work.
Monday, November 08, 2010
Bash math, command expansion and pipes
echo $((`wc -l /path/to/file1.txt|cut -f 1 -d ' '`+`wc -l /path/to/file2.txt|cut -f 1 -d ' '`)) Saturday, October 23, 2010
Handling gzipped HTTP response with Transfer-Encoding: chunked
The chunked transfer consists of many chunks. Before each chunk, we have the size of the chunk terminated by a CRLF pair. There is a final zero length chunk.
The first time I wrote code to handle the chunked zipped transfer, I had the code decompress each chunk. This worked. But then later, content from other URLs did not decompress properly. Probing with Wireshark, I came to realize that each individual chunk cannot be reliably decompressed. This is because the server does not compress each chunk before sending it. The server compresses the full content, then chunks it up and sends each chunk on the wire. Thus, I needed to first build the full message and then decompress the full message. The reasons that my first attempt worked for the URL I was testing had to do with the fact that in that particular case, all data came in a single chunk.
Here is the relevant code as I couldn't find this easily anywhere:
String enc = httpHeaders.get("Content-Encoding");
if (enc != null && enc.toLowerCase().equals("gzip")) {
String te = httpHeaders.get("Transfer-Encoding");
if (te != null && te.toLowerCase().equals("chunked")) {
int idx = httpHeaders.length;
ByteArrayOutputStream os = new ByteArrayOutputStream();
int numBytes = -1;
try {
do {
StringBuilder numBytesBuf = new StringBuilder();
for (;idx<bytes.length && bytes[idx]!='\r';idx++) {
if (Utils.isHex((char)bytes[idx]))
numBytesBuf.append((char)bytes[idx]);
}
if (idx >= bytes.length)
throw new IOException("incorrect chunked encoding for : " + retryURL.getCurrentURL() + " based on " + retryURL.getOrigURL());
idx+=2; //skip over '\r\n'
try {
numBytes = Integer.parseInt(numBytesBuf.toString(), 16);
} catch (NumberFormatException e) {
throw new IOException("incorrect chunked encoding for : " + retryURL.getCurrentURL() + " based on " + retryURL.getOrigURL(), e);
}
if (numBytes > 0) {
//idx points to start and numBytes is the length
os.write(bytes, idx, idx+numBytes <= bytes.length ? numBytes : bytes.length-idx);
if (idx+numBytes > bytes.length) {
System.err.println("incorrect chunked encoding, " + (idx+numBytes) + " is outside " + bytes.length + " for: " + retryURL.getCurrentURL() + " based on " + retryURL.getOrigURL());
break;
}
}
idx += (numBytes+2); //+2 for '\r\n'
} while (numBytes > 0);
GZIPInputStream zip = new GZIPInputStream(new ByteArrayInputStream(os.toByteArray(), 0, os.size()));
byte[] buf = new byte[1024];
int len;
try {
for (;(len = zip.read(buf, 0, buf.length)) > 0;) { //decompress from <zip> to <buf>
f.write(buf,0,len); //transfer from fixed size <buf> to var size <os>
}
} catch (IOException e) {
if (!e.getMessage().equals("Corrupt GZIP trailer") && !e.getMessage().equals("Unexpected end of ZLIB input stream"))
throw e;
else {
System.out.println("handled spurious " + e.getMessage() + " on: " + retryURL.getCurrentURL() + " based on " + retryURL.getOrigURL());
}
}
os.close();
zip.close();
} catch (Exception e) {
System.err.println("failed on: " + retryURL.getCurrentURL() + " based on " + retryURL.getOrigURL());
System.err.println(ExceptionsUtil.toString(e));
}
}
GZipInputStream throws spurious exceptions in how it handles the end of the file
But unfortunately, there are no fixes. Also these spurious exceptions are not thrown just for large files. I have seen these errors for files as small as 5K. Here is a set of decompressed files, sorted by size that I managed to generate by ignoring these spurious errors:
x02:~$ cat /tmp/z | perl -ne 'if (/based on (.*)$/) {system("ls -ltr `echo -n $1|md5`.url");}' | sort -n -t ' ' -k 5
-rw-r--r-- 1 mpire mpire 5501 2010-10-23 14:03 409ff1c2b7ce2887db8a5c98d395b543.url
-rw-r--r-- 1 mpire mpire 38681 2010-10-23 14:03 8bc8f64132dd2e4bdbccff555cfa6966.url
-rw-r--r-- 1 mpire mpire 44554 2010-10-23 14:01 ba1c53b23f747efb3aa3a7531da80fb1.url
-rw-r--r-- 1 mpire mpire 45415 2010-10-23 14:03 073cd89a26dd69bac8f8a734bcaec7f1.url
-rw-r--r-- 1 mpire mpire 46058 2010-10-23 14:00 f0ae11a51e1975838c26c428ce14308a.url
-rw-r--r-- 1 mpire mpire 46192 2010-10-23 14:03 73972b286ec326e91404008b4c125e5a.url
-rw-r--r-- 1 mpire mpire 46414 2010-10-23 14:00 c6389b488bf912ddece9075884ff7c80.url
-rw-r--r-- 1 mpire mpire 47030 2010-10-23 14:00 07d9fe64764458b55626d9eb047d5d4b.url
-rw-r--r-- 1 mpire mpire 47565 2010-10-23 14:03 67a9869337a777c7c6a8411fd55e1b39.url
-rw-r--r-- 1 mpire mpire 49034 2010-10-23 14:01 0792cd32f0ef59cbe3592a5c2a7b5744.url
-rw-r--r-- 1 mpire mpire 58397 2010-10-23 14:03 3c973b623e9d27cd36234222f6542788.url
-rw-r--r-- 1 mpire mpire 58981 2010-10-23 14:03 780f0be38cbe0e73de788597cb482af4.url
-rw-r--r-- 1 mpire mpire 59177 2010-10-23 14:01 15dd65994702db5e360704144874f3f8.url
-rw-r--r-- 1 mpire mpire 60043 2010-10-23 14:03 c355cd84d17be2cf405b04fb3663d181.url
-rw-r--r-- 1 mpire mpire 63189 2010-10-23 14:03 43ab8fabf72b5564bc0e8b1ff3fcebe7.url
-rw-r--r-- 1 mpire mpire 70235 2010-10-23 14:01 3a24135855df237d829960a15cd8b170.url
-rw-r--r-- 1 mpire mpire 71536 2010-10-23 14:01 48f80b3fb1c51ea52313fe76b55a3849.url
-rw-r--r-- 1 mpire mpire 76932 2010-10-23 14:03 8c789913e5666e793c38d02001486532.url
-rw-r--r-- 1 mpire mpire 78825 2010-10-23 14:01 49728436874d94d8b1ab0ef17d8b4736.url
-rw-r--r-- 1 mpire mpire 80459 2010-10-23 14:05 c52a68d48d2c23ef846950dec999f084.url
-rw-r--r-- 1 mpire mpire 80459 2010-10-23 14:05 c52a68d48d2c23ef846950dec999f084.url
-rw-r--r-- 1 mpire mpire 83001 2010-10-23 14:00 a963a7357bf480fe24040ecf05e8927d.url
-rw-r--r-- 1 mpire mpire 105473 2010-10-23 14:01 db76aed3163beb6ad49670866045a9d8.url
-rw-r--r-- 1 mpire mpire 109405 2010-10-23 14:01 70da57c0544c122766a9ad6772757f2b.url
-rw-r--r-- 1 mpire mpire 110921 2010-10-23 14:01 579e0f08a036befe374ff8c70126a2bb.url
-rw-r--r-- 1 mpire mpire 111880 2010-10-23 14:00 3091fb91ae4a92b06392acead7170a57.url
-rw-r--r-- 1 mpire mpire 116796 2010-10-23 14:05 a326a52188abc263e2f8804444b48c8c.url
-rw-r--r-- 1 mpire mpire 154209 2010-10-23 14:06 e90211e8f799a88dc70ff83d1aba0748.url
-rw-r--r-- 1 mpire mpire 159089 2010-10-23 14:01 8ef49b62542f8283acba4e573e28ea59.url
-rw-r--r-- 1 mpire mpire 168786 2010-10-23 14:01 487d9c54fbd3e4b760c91dbf5f754d32.url
-rw-r--r-- 1 mpire mpire 212561 2010-10-23 14:03 b3cb42c946716cfe7e31096bd458e9f2.url
-rw-r--r-- 1 mpire mpire 222257 2010-10-23 14:03 f803ac8246dd2ab7dd16f7d28d5b4594.url
The decompressed files are good. I ran into this issue reading gzipped files from web servers that zip content and send this zipped data in chunks using Transfer-Encoding: chunked.
The error seems to be not related to the Java libraries. I saved the gzip content and tried to unzip with gunzip. This failed as well:
x02:/tmp$ gunzip 0792cd32f0ef59cbe3592a5c2a7b5744.gz
gzip: 0792cd32f0ef59cbe3592a5c2a7b5744.gz: unexpected end of file
Subscribe to:
Posts (Atom)