While working with Iframe and document.domain trick to get around the same-origin policy against cross domain restriction on web-browsers I came across an interesting oddity in Firefox (a hidden security feature if you will!).

The reference page for document.domain setting policy here says:

A script can set the value of document.domain to a suffix of the current domain

What is obvious that this prefix can go only upto anything below a top-level-domain (TLD). This page provides some more info about this (in the first two paragraphs).

Armed with this info I expected to be able to communicate between two of my apps, both hosted on google appengine (http://app1.appspot.com & http://app2.appspot.com). The idea was to set document.domain to appspot.com in both, the parent page on http://app1.appspot.com and in the child iframe loaded from http://app2.appspot.com.

Works great in Google Chrome! However, true to its spirit of being a pain in an unmentionable, the cross-browser beast rises again and bites me in the same unmentionable: It fails on firefox! On the JS document.domain = appspot.com on the page hosted on http://app1.appspot.com, it throws an error: “Illegal document.domain value” code: “1009.

A little more digging. I test using DynDNS. Now with my apps accessible at http://mydomain1.dyndns.org and http://mydomain2.dyndns.org it works! So it has to be a problem specific to appspot! Is firefox handling appspot.com specially? To be honest I found the idea ridiculous when some colleagues suggested this. But I still decided to give it a try: I search for appspot.com through mozilla source code at mxr. Lo and behold!, there is an entry for appspot.com here. Looking for effective tld names on google reveals the mystery.

The Mystery: Solved

As hinted here firefox needed to tell that both .com and .co.uk types of names are valid TLDs (top level domains). To do this they have included a special static file with firefox with a list of all TLDs. So for each entry X in this file you can strip document.domain to any sub-domain of X (y1.X or y2.y1.X or y3.y2.y1.X and so on) but not to X (even if X has dots(.) inside it). The evil piece is the appspot.com is also an entry in this file (so that firefox considers appspot.com as a TLD). As a result the Iframe and document.domain trick will not work for any app hosted on app engine.

I have been recently working on a small Facebook app FriendGeo, which is on the verge of being finished as I write. One conclusion: Facebook API is cool, the documentation is NOT. I had my full share of frustration trying to deal with hidden (or so it seems) features and, more painful, restrictions on Facebook API.

Anyway, coming back to the topic, my app needs to post an image to the user’s wall. True to its RESTfulness, the graph API allows you to do so by POSTing to the photos connection of the user. In a perfect world, a simple HTTP POST to https://graph.facebook.com/userid/photos with required parameters (the names of which you are left to guess or google!) should do.

A little bit of tinkering with google search query leads to forum posts like this. Fortunately, the PHP SDK for the Facebook API allows you to upload the photo by using a simple call to setFileUploadSupport. Unfortunately for me, I am using App Engine as my hosting solution and using Python for writing my backend and the official Python SDK for Facebook does not seem to be supporting any such provision. Solution: fall back on Python ‘batteries’ and use raw Python and HTTP.

To demonstrate how to do this with a self contained example I have written this small script that creates a test user and uploads a photo from disk to the test user’s account. The comments in the code below should be enough to guide you through the code.

import sys
import os
import itertools
import mimetypes
import mimetools
import urllib
import urllib2
import cgi

try:
    import json
except ImportError:
    from django.utils import simplejson as json

# this class allows you to upload a file to a URL
# using pure urllib
class MultiPartForm(object):
    """Accumulate the data to be used when posting a form.
       source: http://www.doughellmann.com/PyMOTW/urllib2/#uploading-files
    """

    def __init__(self):
        self.form_fields = []
        self.files = []
        self.boundary = mimetools.choose_boundary()
        return

    def get_content_type(self):
        return 'multipart/form-data; boundary=%s' % self.boundary

    def add_field(self, name, value):
        """Add a simple field to the form data."""
        self.form_fields.append((name, value))
        return

    def add_file(self, fieldname, filename, fileHandle, mimetype=None):
        """Add a file to be uploaded."""
        body = fileHandle.read()
        if mimetype is None:
            mimetype = mimetypes.guess_type(filename)[0] or 'application/octet-stream'
        self.files.append((fieldname, filename, mimetype, body))
        return

    def __str__(self):
        """Return a string representing the form data, including attached files."""
        # Build a list of lists, each containing "lines" of the
        # request.  Each part is separated by a boundary string.
        # Once the list is built, return a string where each
        # line is separated by '\r\n'.
        parts = []
        part_boundary = '--' + self.boundary

        # Add the form fields
        parts.extend(
                     [part_boundary,
                     'Content-Disposition: form-data; name="%s"' % name,
                     '',
                     value,
                     ]
                     for name, value in self.form_fields
                     )

        # Add the files to upload
        parts.extend(
                     [part_boundary,
                     'Content-Disposition: file; name="%s"; filename="%s"' % \
                     (field_name, filename),
                     'Content-Type: %s' % content_type,
                     '',
                     body,
                     ]
                     for field_name, filename, content_type, body in self.files
                     )

        # Flatten the list and add closing boundary marker,
        # then return CR+LF separated data
        flattened = list(itertools.chain(*parts))
        flattened.append('--' + self.boundary + '--')
        flattened.append('')
        flattened = map(str, flattened)
        return '\r\n'.join(flattened)

def get_client_access_token(app_id, app_secret):
    #create an access token acting as if we are a
    #desktop client (we in fact are!). app_id and
    #app_secrets are the parameters that you get
    #get when you create a new app
    params = {
        'grant_type': 'client_credentials',
        'client_id': app_id,
        'client_secret': app_secret
    }

    response = cgi.parse_qs(urllib.urlopen(
                "https://graph.facebook.com/oauth/access_token?" +
                urllib.urlencode(params)).read())
    access_token = response["access_token"][-1]

    return access_token

def create_user(app_id, access_token):
    #create a test user using desktop-client
    #access token. We need 'publish_stream' and
    #'user_photos' extended permissions
    params = {
        'access_token': access_token,
        'installed': 'true',
        'permissions': 'publish_stream,user_photos'
    }

    params = urllib.urlencode(params)
    url = 'https://graph.facebook.com/%s/accounts/test-users'%(app_id)
    request = urllib2.Request(url, headers = {}, data = params)

    response = urllib2.urlopen(request)
    response = response.read()

    response = json.loads(response)
    return response

def upload_pic(uid, access_token, pic_caption, pic_file_name, pic_file):
    #using access_token for the newly created test user upload
    #an image file from disk (or any other file like source) to
    #the user's account.
    #pic_caption: the caption to the pic
    #pic_file_name: name of the file (used only to guess mimetype)
    #pic_file: a file like object representing the file (one can
    #open a url and use the response here if the file to be uploaded
    #is on the web (and not on the disk)
    form = MultiPartForm()
    form.add_field('access_token', access_token)
    form.add_field('caption', pic_caption)

    form.add_file('source', pic_file_name, pic_file)

    request = urllib2.Request('https://graph.facebook.com/%s/photos'%(uid))
    body = str(form)
    request.add_header('Content-type', form.get_content_type())
    request.add_header('Content-length', len(body))
    request.add_data(body)

    return urllib2.urlopen(request).read()

def do():
    if not len(sys.argv) == 5:
        print "usage: python %s <app_id> <app_secret> <caption> <file_path>"%(sys.argv[0])
        return

    app_id = sys.argv[1]
    app_secret = sys.argv[2]
    p_caption = sys.argv[3]
    file_path = sys.argv[4]

    client_access_token = get_client_access_token(app_id, app_secret)
    user_info = create_user(app_id, client_access_token)

    u_id = user_info['id']
    print 'created test user account with id: %s'%(u_id)

    u_access_token = user_info['access_token']
    user_login_url = user_info['login_url']

    p_file_name = os.path.basename(file_path)
    p_file = open(file_path)

    upload_pic(u_id, u_access_token, p_caption, p_file_name, p_file)
    print 'login as the test user and view the uploaded pic: %s'%(user_login_url)

if __name__ == '__main__':
    do()

import urllib2

#size of each block we want to read. since we are unable to read all data at once, we will
#read this many bytes at once and write it to a local file then go back again fetch another block
#write to local file and keep doing this till there is data left to read
READ_BLOCK_SIZE = 1024*8

#use a url that will indeed send compressed data when asked for, some servers choose not to
#even if you are willing to accept compressed data
url = 'http://ax.itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=282935706'

request = urllib2.Request(url)

#let the server know you are ready to accept compressed data
request.add_header('Accept-encoding', 'gzip,deflate')

response = urllib2.urlopen(request)

#make sure the response is compressed
isGZipped = response.headers.get('content-encoding', '').find('gzip') >= 0

#file to write data to
outFile = open("bible.html", "w")

while True:
  data = response.read(READ_BLOCK_SIZE)
  if not data: break
  if isGZipped:
    #make a file like thing of the read bytes
    data = cStringIO.StringIO(data)
    #open the file-like-thing as a gzipped file
    data = gzip.GzipFile(fileobj=data).read()
  outFile.write(data)

This code, as I hinted above does not work, what one gets is a checksum failure error, something like

IOError: CRC check failed 0x29cd27b0 != 0x21b93d26L

This happens because gzip interprets the chunk you read as being a full gzip file (with an Alder32 crc at the end) which is obviously not the case (unless of course your whole file is less than your block size or you read the whole file into a string using read()). For more info on gzip format refer to the RFC.

A simple way around this problem comes from the fact that gzip is little more than a wrapper around zlib (primarily because most of gzip is more or less zlib based compression, except for some format details).
Use of zlib with a little modification does the trick:

import zlib

d = zlib.decompressobj(16+zlib.MAX_WBITS) #this magic number can be inferred from the structure of a gzip file
while True:
  data = response.read(READ_BLOCK_SIZE)
  if not data: break
  if isGZipped:
    #make a file like thing of the read bytes
    
    #no need for file-like-thing
    #data = cStringIO.StringIO(data)

    #open the file-like-thing as a gzipped file
    #data = gzip.GzipFile(fileobj=data).read()
    
    data = d.decompress(data)
  outFile.write(data)

That’s it, using this you can decompress any gzipped stream of data

:~$ ps --context -e | grep python

824 unconfined                      python run.py scripts/naivebayes/run.py

This gives me the complete command that was used to start the process and it becomes a lot easier to identify the process now

To count the number of upper case (capital letters) in a given text use this:

import re

re.findall("[A-Z]", text)

to count only those that are not the first letters of the corresponding words

import re

re.findall("\B[A-Z]",  text)

a lot of common Haskell functions are defined in the module Prelude (e.g. isLower, isUpper).The default module for WinHugs however is Hugs, hence if you try to access these functions directly (which every newbie does )..you get errors like this

ERROR - Undefined variable "isLower"

.
To correct load the module Hugs.Prelude

Hugs> :module Hugs.Prelude

Hugs.Prelude> isLower 'a'
True



If google is to be believed it is: RFC 1918 (rfc1918) – Address Allocation for Private Internets

Superclass:

public abstract class Super {
  public Super() {
    loadConfig();
  }

  protected abstract void loadConfig();
}

Subclass:

public class Sub extends Super {

  protected String var;

  protected void loadConfig() {
    var = "loaded_config";
  }
}

The idea is to enforce the constraint that the loadConfig method is always called in the constructor of the object, how the configuration is to be loaded exactly is left to the subclasses.

This however does not work a expected

public class Test {
  Sub s = new Sub();
  System.out.println("var value: " + s.var);
}

The output is not “var value: loaded_config” as a lot of us would expect. it’s “var value: null”.
To understand why, change the Sub class to this:

public class Sub extends Super {

  protected String var = setMeth();

  private String setMeth() {
    System.out.println("Setting member variable with setMeth");
    return "setMeth value";
  }

  protected void loadConfig() {
    System.out.println("Setting member variable with loadConfig");
    var = "loaded_config";
  }
}

Now the output should be:

Setting member variable with loadConfig
Setting member variable with setMeth
var value: setMeth value

What happens is that in the subclass constructor, the superclass constructor is called before the subclass object is created (that is why you always need to put calls to super() as the first statement). As a result, although the overriden method does it’s job fine in setting the value of the variable the way we want, just after this happens (from inside the super class ctor), the sub class object is initialized and the value for the member variable is overwritten (by the setMeth in second case and as the default null in the first).

The only way out of this (that I know!) is to move this obligation of calling the loadConfig method on to the subclass implementation code (which is clearly not anyone’s first choice )

usually it suffices to do this:

shashanksingh@nsl-31:~$http_proxy=http://username:password@proxyserver:80/

the placeholders having obvious meanings. A problem however arises if the password has an ‘@’ character inside it. This @, being a delimiter will mess things up, e.g., if the password were my@pass, the url will become

http://username:my@pass@proxyserver:80/

wget will then parse it and treat the first @ as if it were the delimiter to mark the end of the password and the beginning of the proxy address. As a result it will assume:

user: username
pass: my
proxy host: pass@proxyserver:80

and you will get an error like this

Resolving pass@proxyserver... failed: Name or service not known.

One possible solution that comes to mind is to escape the @. This somehow does not work (for wget at least). So

shashanksingh@nsl-31:~$http_proxy=http://username:\@password@proxyserver:80/

gives the same error (I wonder if I am guessing the escape character right, if you know why it’s not working please post in comments )

A work-around is to create a .wgetrc file. wget loads this file when run (you can find more details here).

A file like this works fine for me

http_proxy=proxyserver:80
proxy_user=username
proxy_passwd=my@pass

Place the .wgetrc file in your home directory ($HOME) or other possible locations as mentioned in the link above.

UPDATE: while escaping does not seem to work for wget, exporting a http_proxy in .bashrc seem to handle escaping using ‘\’. So if your password is @agrambagram, adding a line at the end of ~/.bashrc works

export http_proxy=http://username:\@agrambagram@proxyserver:80

Unfortunately people at postgreSQL seem to be suffering from a widespread tendency of treating a windows user as someone who needs to be punished for committing the cardinal sin of opting windows. I remember several of my batchmates having had their brains fried trying to install postgreSQL on their windows machines (before I stepped in to save them, of course ).

Anyway, enough digression! Here is how it worked out for me:

1. Get the server running:
   

   pg_ctl.exe start -D "C:\Program Files\PostgreSQL\8.3\data"

2. Open the file

   "C:\Program Files\PostgreSQL\8.3\data\pg_hba.conf"

   Initial Content:

   # TYPE DATABASE  USER CIDR-ADDRESS  METHOD
   
   # IPv4 local connections:
   host all all  127.0.0.1/32 md5
   # IPv6 local connections:
   host all all ::1/128 md5

   Changed to:

   # TYPE DATABASE USER CIDR-ADDRESS METHOD
   
   # IPv4 local connections:
   host all all 127.0.0.1/32 trust sameuser
   # IPv6 local connections:
   host all all ::1/128 trust sameuser

3. reload the server config:
   

   pg_ctl.exe reload -D "C:\Program Files\PostgreSQL\8.3\data"

4. run:
   

   psql -U postgres

   At this point if you get this error:

   psql: FATAL:  role "postgres" does not exist

   do this:

   createuser.exe -s postgres

5. run this to start command line client:
   

   psql -U postgres

6. run SQL to change password:
   

   postgres=# ALTER USER postgres WITH ENCRYPTED PASSWORD 'password';

   Note that the inverted commas around the password are required but are not a part of the password

7. Rollback the changes made in step 2
8. reload the server config again as in step 3
9. Curse the postgre config for one last time
10. Get going!