Monday, September 08, 2008

use ncftp to upload whole directories using ftp

yesterday, i came across the problem of figuring out how to transfer a whole directory to a hosting site which allowed no ssh access. without ssh, it wasn't possible to copy a tar file and untar it. the only protocol available was ftp. ftp doesn't allow you to upload a whole directory. if you use `mput *`, it will upload all the files within the directory, but it will not recurse into sub-directories.

i found a script someone had written that actually allowed downloading a full directory recursively. and there were a few posts that mentioned a `curl -T` way of doing it, but i couldn't get it to work.

then i hit upon ncftp, that worked really well.

simply:

ncftp -u username -p password ftp.somedomain.com

and when inside the ftp shell,

put -R somedir


Thursday, August 28, 2008

web 2.0 image clusters

at http://livemocha.com we use a cluster of lighttpd servers to handle the requests for image, audio/video files. the reason to use lighttpd instead of apache is to do with the superior capability of lighttpd to handle large number of concurrent requests with low latency.

the apache server will use one process to serve a request. while the request is being served, that process will not accept any other requests. any other request needs to be served by a different apache process.

for a low volume server, this is not a problem. there are enough apache processes lieing around to serve the concurrent requests.

it does become a problem with high volume. a typical web page may have 20-30 images, a css file, some javascript files. the browser will first make a request to apache (assuming apache is the web server) for the main web page. then as the browser parses the page, it will issue additional requests to the image files etc found on the page.

if these requests go back to apache, they will each have to be served by one apache process one at a time. the KeepAlive option at apache **may** provide some relief by keeping the TCP connection open at apache so that the browser can keep using that for the images. but this doesn't change the fact that one apache process will always block other requests while it is serving the image.

the image request is mostly I/O bound - meaning most of the time to fulfill the request will be spent waiting for the image file to be fetched from a hard drive, perhaps in some network share. in the case of livemocha, most images were served off a NFS server. So it is quite wasteful to have a process sitting around doing nothing but waiting for I/O to complete.

Having a process waiting around is bad because it takes memory. There is only a finite amount of it in the server, and when there are more processes than that can fit memory the O/S will have to swap a processes into disk to make room for another process. When the O/S context switches to the swapped process, it has to be brought back into memory again, possibly resulting in another process being swapped, so on and so forth. this is a vicious cycle that can and will bring the server to its knees - this phenomena is known as the "slashdot effect".

lighttpd can serve multiple requests from one process. it uses asynchronous I/O available in linux 2.6 kernels (ex: epoll) to multiplex multiple requests in one process. this uses very low memory and can perform well under the "slashdot effect".

nginx is another server that performs very well using similar technologies. we decided on lighttpd mainly because some other key sites were known to use it. from benchmarks, nginx seemed to be superior.

Tuesday, August 19, 2008

Ubuntu 8.04 - setup wireless to start on boot

Make sure your /etc/network/interfaces file looks like this:

auto lo
iface lo inet loopback
auto wlan0
iface wlan0 inet dhcp
wireless-essid your-essid
wireless-ap 00:25:43:DE:AC:2B
wireless-mode Managed

How do you find your essid and access point (ap)? In my case, I cheated by using the "Network configuration" icon on the top left. If you click on it, it will show the available interfaces, just pick the correct one. Then issue the iwconfig command on the shell, and you should see the essid and ap.

ex:

thushara@gini-sisila:~$ iwconfig
lo no wireless extensions.

eth0 no wireless extensions.

wlan0 IEEE 802.11b+/g+ ESSID:"my-essid" Nickname:"acx v0.3.36"
Mode:Managed Frequency:2.437 GHz Access Point: 00:25:43:DE:AC:2B
Bit Rate:24 Mb/s Tx-Power=15 dBm Sensitivity=1/3
Retry min limit:7 RTS thr:off
Power Management:off
Link Quality=31/100 Signal level=3/100 Noise level=0/100
Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0
Tx excessive retries:0 Invalid misc:0 Missed beacon:0

You can also find these parameters by logging into your wireless router's admin interface. You need to wire the computer to the router (or use another computer connected to the router) for this.

Thursday, April 03, 2008

Sessions in PHP (and cakephp Session abstraction)- how it works

The session in php is supposed to encapsulate the logic of retrieving the cookie values, mapping the cookie to a value at the web server backend, purging this backend data when cookies expire etc.

CakePHP provides a class called Session, that attempts to hide these details a step further. If you want to identify a web user across many web pages, all you need to do is enable sessions and then you can store information of the user in a Session object which will be always available at the web backend.

CakePHP provides a "helper" and a "component" that provides the next layer of abstraction to their basic Session object. If you're coding the "view" of the MVC, you use the helper - if you're coding the controller, you would use the "component".

This is all good for smallish-scale apps. As you start using more than one web server, you can't keep the cookie mappings in the web server, as the web requests are now going to one of many web servers. CakePHP has a mechanism using php session callbacks to allow you to store the cookie mappings in a central database.

This article attempts to walk you through the architecture of cakephp + php that allows this customization.
I would be referring to cake/libs/session.php so you should have it open in an editor as you go through this.

If we look in cake/libs/session.php, Session::__construct() function, we see that there is a call to $this->__initSession() followed by a call to session_start(). The __initSession() function sets some options in php that configures session handling at that level. Follow the switch statement on CAKE_SESSION_SAVE, and if it is set to 'database', you will see that session_set_save_handler() is used to tell php about some callbacks, namely:
__open : an initialization function
__close: a cleanup function, that is generally used to purge unnecessary data (gc)
__read: php passes this function a "session id" and expects the app to return the data that is mapped to this session id, which we shall call the "session value"
__write: php passes this function a "session id" and a "session value" and expects the app to store that mapping in the db

So, going back to the Session constructor, after the call to set the php callbacks, session_start() is called. This is obviously a function implemented in php core. It will now look at the cookie passed by the browser.

If this is a first time web user, there will be no cookie, so session_start() will generate a "session id" - an 128 bit value that is random enough to have a very low probability of collision. Then since session_start()
knows about the callback functions that were already set, it will call __open and __read

If you look at Session::__open() you will see it does nothing except return "true", signaling php that everything was ok on the initialization step. Now let's look at Session::__read().

As I said earlier, php passes a "key", which is nothing but the session id it just generated. The Session::read() code is expected to now retrieve the "session value" mapped to this "session id" by querying the db. Again if this is a first time user, there will be no value in the db and nothing will be returned to session_start(). Now session_start() will create a $_SESSION hash, that will be empty as nothing was returned to session_start() from Session_read(). It will also send this cookie value to the browser using a Set-Cookie response header. The cookie value looks like this:

CAKEPHP=session_id

Here the string CAKEPHP is defined in app/config/core.php and can be changed. For the curious, notice that this is passed into session.name in the Session::__initSession() function, that is how session_start() knows what name to use for the cookie. The session id in the cookie is of course what session_start() generated - that 128 bit random number we talked about.

At this point your application can write values to $_SESSION, whatever values you write will be saved to the database, upon a session_write_close() call to php. session_write_close() will use the 2 other callbacks, __write and __close to do that.

So at what point should the application call session_write_close()? It should be called before control leaves the current web request. One such method is when Controller::redirect() is called - see cake/libs/controller/controller.php. This function results in a redirect to the browser which takes the user to a different page, potentially hitting a different web server. So the current state must be saved to the database before that. The other method is the more natural path, of the request simply completing. The way CakePHP is designed, we're always guaranteed to have an instance of the ConnectionManager class running. The destructor of ConnectionManager calls session_write_close() - look in cake/libs/model/connection_manager.php.

So as you can see, if you use CakePHP, your app doesn't need to call session_write_close(), it is automatically handled for you by the framework.

So what happens inside session_write_close()? It will call Session::__write() passing the session id as the "key" and the $_SESSION hash as the "value". Session__write() first checks if a mapping exists for this session id, if so it updates it with the new values from the $_SESSION object - remember these may be values you set in you application, like user name, user email etc. For a first time user, there will be no mapping in the database, so Session::__write() inserts a new row.

Then session_write_close() will call Session::__close() which according to a coin flip (well actually using a function with lower odds than 0.5), will clean session mappings that have expired. How does CakePHP know if the cookies have expired? - interestingly enough the session cleanup here has nothing to do with the actual cookie expiration in the browser. CakePHP in fact uses a long lived cookie on the browser, but sets a configurable timeout for the mappings at the server. The timeout is calculated using the formula:

$timeout = CAKE_SESSION_TIMEOUT * factor,

where $factor is one of 10,100 or 300 for CAKE_SECURITY settings of 'high', 'medium' and 'low'. Both CAKE_SESSION_TIMEOUT and CAKE_SECURITY settings can be changed in app/config/core.php.

So we followed through what happens for a first time web user to your site - before the request leaves the page, a cookie is generated, sent to the browser, an empty $_SESSION object created that can be manipulated by the application, and then the $_SESSION object is saved to the database.

Now when the user visits the site again, the browser will send the cookie back to the site. Now session_start() has a cookie and it will extract the session id from it. Then when in calls Session::__read() it will pass the session id as the "key". The row will be found in the table and the session object will be returned to session_start(). session_start() will now create the $_SESSION object based on this value that Session::__read() returned.

Again, you're free to change, add values to the $_SESSION object and CakePHP guarantees that any changes will be written to the database before control leaves the web request logic.

CakePHP also provides some helper functions to write/read to/from the $_SESSION object. They are Session::write() and Session::read() respectively.

Hope this de-mystifies how sessions work a bit and helps in using the Session object with confidence.

Monday, March 10, 2008

Asynchronous file uploads


Here is a screen-shot of the async file upload I talked about in the earlier post.
The iframe is the "Upload File" UI element. It is embedded in the main form.
The main form has no idea of there being a file input element. All the file input logic is in the form element within the iframe. This way the file upload can even be re-used in different forms.

In fact livemocha has 2 forms where users can upload pictures. The first is when the user activates their account, the next is when they later would want to edit their profile information. I use the iframe code for both of those views. And that includes the controler logic that processes the image as well.

GMail like Asynchronous file upload from a form

Web forms that allow you to upload files typically insist on clicking on a "Submit" button to upload the file to the web server.
Most input data can be submitted to the web server from the client using Ajax, thus bypassing a "Submit" button. However file data cannot be serialized and passed to the web server with Ajax.
It is possible to use Javascript (NOT Ajax) and call form.submit() directly, thus bypassing the "Submit" button.
The only issue here is that, we have no control over how the submit() call renders data. One specific issue is that this call would cause the whole form to re-load which results in the user losing any other data she may have entered on the same page.

It is possible to use an iFrame within the form, include the file input field within the iFrame and call iFrame.form.submit() to upload the file. Then only the iFrame will be re-loaded and the rest of the form will remain intact.

GMail uses a technique similar to this, that allows them to "see" the attachment before the user clicks "Send" on the email - this is a very useful feature, as users immediately get feedback, GMail can perform validations - eg: virus checks - on the file as the user is composing the email etc.

Examine the html at http://livemocha.com/profiles/create to get an idea

Notice the "src" parameter points to a different form for the file upload part. That is the mini-form that posts the file data asynchronously (using javascript) to the server.

The javascript does the direct submit to the webserver bypassing the "Submit" button - in fact the mini-form inside the iframe does not have a Submit button as you can see. The javascript is triggered via the "onchange" notification on the file input field, i.e: when a file is selected for upload by the user.

This is a feature that is now available at http://www.livemocha.com/profiles/basic - check it out so you can see some more nice interactions. For example, you can immediately show the new picture uploaded to the user - normally this is difficult to do on the web browser, as on the client we have no access to the local file system.

Wednesday, December 05, 2007

xdebug on centos

these instructions are from : http://tech.nigel.in/

So for those of you who need to install Xdebug, here is what you need to do.
  1. You need to have PHP 5 and PECL installed and working with Apache already.
  2. You need to install PHP's devel package. This is needed for the command phpize to work. Note that you need to have the CentosPlus repository enabled for this. So from the terminal as root run the command:
    yum install php-devel
  3. Next you need to have GCC and GCC C++ compilers installed. The only way to install Xdebug on linux as of time of writing this is to compile the extension yourself. The good thing is that this turns out to be is really easy. Note that you also need Autoconf and Automake packages. Once again from the terminal as root run the command:
    yum install gcc gcc-c++ autoconf automake
  4. Next its time to get the Xdebug package for php itself, and compile and install it. That is as simple as typing the following command as root:
    pecl install Xdebug
  5. Now you need to configure your PHP to load the extension. On Centos, extensions are loaded from a folder most likely to be located at /etc/php.d In this folder you are most likely a bunch of .ini files corresponding to what ever extensions you have installed for use with PHP. You now need to create a file called xdebug.ini here as root. Next add the following lines to the file:
    ; Enable Xdebug extension module
    extension=xdebug.so

    ; Configure the extension [See Xdebug documentation for options to add here]
    xdebug.remote_enable=
    xdebug. .....
  6. Restart Apache with service httpd restart and check the output of phpinfo() to see the if Xdebug is loaded and configured correctly.