Tuesday, 30 November 2010

Tuesday, 2 November 2010

Bug Reports

Its really important to keeping things simple and clear. This article struck me as excellent, and so I'm posting it here, so I don't forget about it.


Tuesday, 24 August 2010

COM / DCOM In Python : The comtypes library

There are all kinds of crazy reasons you might want to use COM in Python. There are two main libraries for doing so. If you can use pywin32, then do- its high level, Pythonic and relatively functional. If you want to instantiate Word, or Excel, call a few methods, pass some simple data in, get some simple data back, fill your boots:

For more hardcore COM stuff you need to use the much lower level and partly undocumented comtypes library. The following bullets define "hardcore":
  • Subscribing to a COM event notification sink.
  • Building your own COM server (are you mad?)
  • Sending and receiving more complex datatypes.
It was my strong desire to automate certain tasks my group is responsible for by creating a DCOM client to hook into our management agents on the several hundred servers in our datacenter. This required an event sink.

I don't want to be unfair to comtypes, it is a great library, and when it works is very satisfying and robust. The main problems are the gaps in documentation. So without further ado, here's my pointers on using comtypes.

Download it and give it a try. Python 3.0 is unsupported afaik.
Do read the existing documentation. Carefully, what exists is of good quality.
Use the mailing list archive. Seriously, its the only way forward.
Have the IDL handy. The IDL is the file which defines the COM interfaces for your object, which the TypeLib is generated from. Most likely, if you need to use comtypes this file will be available to you, and will be a vital reference.

Turn on logging. If you're having issues you can turn on comtypes logging.

Read the source of comtypes. Once nice thing about Python is the source code is very readable. There are some useful comments in there also. On my machine the source can be found:
  • C:\Python26\Lib\site-packages\comtypes
Files which I found of interest were __init__.py, automation.py (types) and client._events.py (for events stuff).

Understand the type coercion. Much of the work that comtypes does for you is in the type coercion. If you get stuck wondering why your specific type of VARIANT is not getting coerced quite how you expected then  take a look in automation.py, and read the following Mailing list post (which should really be in the docs):
Keep your Sink in scope. This may be because I'm an idiot, but I was declaring my "advise" object in a constructor and then not keeping it. Make sure your advise object is still in scope when you call PumpEvents().

Understand the domain. Chances are you are going to have to learn about the bizarre world of C++ and COM. Frankly COM could only ever have seemed to make sense to a C++ programmer. Make sure you at least have a hazy notion of the following concepts, ideally in their C++ sense:
  • Interface
  • Pointer
  • Structure
  • Array
Also, realize why COM came about. Back in the day Windows had "dll hell" so Microsoft wanted a "code registry" which allowed for multiple versions and implementations, rather than just a system folder full of DLLs. Java was on the horizon with the idea of the "component model" and "code reuse", and Microsoft needed to compete. RAD languages such as Delphi were getting big too - so Visual Basic compatibility was another talking point. Server side Software Engineering problems which Windows Server was being utilized to solve were getting larger than a single machine: so it made sense to whack a bit of RPC in the mix (DCOM).

Built on ctypes. Be mindful the comtypes is build upon ctypes, especially when you get into type based confusion the docs there can be very enlightening.

Friday, 6 August 2010

Working with PayPal Web Payments Pro

For a web based store, to process credit card transactions is a requirement. PayPal offers some simpler integrations than fully blown gateways such as Authorise.net. My plan was to "ramp up" using PayPal to get a working flow, then migrate to a "proper gateway" later on if it seemed justified.

Getting PayPal up and running is simpler than it seems, however I found there is something "bewildering" about the PayPal website, the range of options they offer and their sandbox. This is to share the things I have learnt.

Inappropriate Options

Mass Payments is sending money to a large number of people. Not useful for me. Website Payments Standard offers lots of "clever" buttons which you can embed on a static site. Forget about it. Mobile Checkout is engineered for WAP. Not interested. Payflow Pro has PayPal act as an intermediary to a third party gateway. Can't see the point in this. Website Payment Pro Payflow edition is just a variant on Payflow, so again, not interested.

Website Payments Pro (Express Checkout and Direct Payment)

Once I focussed my mind on Website Payments Pro, things slotted into place. WPP consists of Express Checkout - where you dump your customer at PayPal and they use their PayPal account, and Direct Payment where you grab customer's credit card data your end, and post it to PayPal in the background. Direct Payment does not require a PayPal account - however it should be obvious that SSL is a requirement, as you are grabbing sensitive info your end.

Direct Payments is the real win here effectively its a very simple to use payment gateway at a reasonable cost from PayPal. However - the WPP offering is the combination of them both - Express Checkout and Direct Payments - so if you implement Direct Payments you are obliged to offer Express Checkout too - in order to redirect people into creating PayPal accounts I would warrant. This is not too much sweat however.

1. Creating a PayPal developer account.

First step is to create a PayPal developer account, this is not the same thing as the account you will use to access the API - this is the account you need to then create test API accounts. So this is a first step:
2. Create Sandbox account to test with.

Once you are logged into the PayPal sandbox "center" then you can create an account within the Sandbox to test with. There is a macro for creating preconfigured types of accounts- effectively you want the test version of a merchant account with WPP enabled. Select "add account" and then add a WPP test account. It will generate some rubbish email address based on what you enter. Emails sent to this address in the course of testing (i.e. notification of purchases etc) will go to an integrated webmail section of the Sandbox site. Don't forget the password you use to create the sandbox merchant account! It is different from the API password and you will need it to view the status of transactions during testing.

At this point also treat yourself by setting up an additional account to act as the customer when testing express payments.

Once you have created your Sandbox merchant WPP account, you can then browse the details of this account on the "API Credentials" menu item. You should have the following details:
  • API_USER_NAME: This is just a test Merchant username for API calls.
  • API_PASSWORD: The password.
  • API_SIGNATURE: This is the additional signature for accessing the API.
However, with these three items you should be set to access the API. I'll be doing so using Python.

3. Hit the NVP API webservice and do stuff.

At this stage its worth digressing on the sorts of flows which will need to be implemented. Step -1 is that your marketing will work, the customer will come to your site, feel excited / relieved to have found something they want to buy, and will use whatever system of button pressing you have implemented to add the item to their shopping cart. The following account begins at the point where they click "checkout".

Express Checkout:
  1. You offer the customer the option to enter credit card details on your site (Direct Payments) or use Express Checkout. They choose the latter.
  2. You POST using the NVP API to the SetExpressPayment method. This POST includes your API key, the amount of the transaction plus return and cancel URLs for PayPal to return your customer to.
  3. You receive an "express checkout token" in response. You construct a URL based off a static PayPal URI plus this token, and redirect your customer to it.
  4. The customer completes their purchase in PayPal by either entering their PayPal account details or creating a new PayPal account.
  5. PayPal will then redirect the customer back to your site, appending URL parameters containing the Express Checkout token and the Payer ID, which identifies the customer.
  6. You then call DoExpressCheckoutPayment method using the NVP API passing in the "express checkout token", the Payer ID you got back in step 5 and the same amount you passed along in the first instance. PayPal sends you a confirmation as a reply in NVP.
  7. The customer gets a confirmation email from PayPal, and the test merchant account has the transaction posted into it. You can check this from the Sandbox "center" site "Test Email" section.
So here's a sample script for this which can be run from the command line / IDLE etc. Note you need to copy paste the payer ID from query URL in the browser after you get back from PayPal into the script.

The above is heavily based on the example here
So that's Express Checkout - given this post has turned into a complete monster I'm going to leave Direct Payments for another day and another post. Hope someone finds this post useful!

Saturday, 31 July 2010

Production Flask app on CENTOS and Apache via mod_wsgi

Flask is a Python web framework which embraces simplicity and seems to work. It is based on Werkzeug so it is a WSGI application.

I've had some luck in the past using mod_wsgi to run Django apps, so I suspect getting a Flask app to run will be similar.

Using Google, I found the following in the Werkzeug Docs, which I reviewed for the following steps:

1. Install python2.7, mod_wsgi and Flask on production server.

Using Centos 5.4 (unbranded RHEL) comes with with python 2.4 and apache 2.2. Centos (and RHEL) use Python extensively under the hood (for Yum, among other things). I want to use a later version of Python, but one may not simply "upgrade" the version of Python- without breaking the OS - you need to install an additional version. I followed the steps in this guide, under the head "Install Python":
I chose to use Python 2.7 as its the last stable release of Python 2, and I don't intend going Python 3.* for a while. I hope I don't regret that. I needed to change a few things in obvious places as the guide covers Python 2.6.

Next I carried out the steps labelled "Install Setuptools" as this is necessary.

Next I carried out the steps labelled "Install mod_wsgi", restarted httpd and it started!

To install Flask I ran SetupTools explicitly from the location inside of /opt/python2.7/ to install 'pip' into the 2.7 Python distribution. Next I added an alias to pip to the ~/.bash_profile in the same way this was done for the python binary in the VenkysBlog page. Finally I used the source command to load the alias and ran:

pip install Flask

2. Create an appropriate wsgi file.

WSGI uses a file with the extension .wsgi as the interface between the wsgi container (in this case Apache / mod_wsgi) and the app. Flask has excellent documentation:

Steps are as follows:

  1. Place Flask application in a subfolder of the webroot (for testing I just used the "Flask is Fun" example: http://flask.pocoo.org/).
  2. Make a file called 'yourapp.wsgi' in the same structure - this is written in Python syntax - here you need to import your app and bind it to the name 'application'. mod_wsgi will then call this object in the appropriate way.

3. Edit apache configuration to invoke wsgi application.

Finally some directives need adding to httpd.conf. The LoadModule statement was added as part of step 1, however the wsgi file need to be referenced vs. the appropriate virtual root.

Great so everything worked! Wasn't that easy?

Tuesday, 1 June 2010

Getting VirtualEnv going

Everywhere I read the benefits of virtualenv, but each time I try it I get fed up as it seems a lot to get setup. Here's to trying again:.

yum - y python-setuptools
easy_install pip
pip install virtualenv

Wednesday, 5 May 2010

Installing Fedora 12 on an IBM / Lenovo Thinkpad X60s / X60

Having blasted my X60s and nearly pulled all my own hair out here is a recipe to do this. It is really rather straight-forward, and will leave your Windows in tact.

1. Grab your 4gb memory key and insert into a Windows box. If you have an x60 and no external drive you really you should have one of these: [ put it on your keyring! ]

2. Grab this from a Windows box: http://sourceforge.net/projects/unetbootin/

3. Select Fedora, Live and then OK.

4. When done, insert into your X60 and boot.

5. When logged in select "install to harddrive".

Job done!

Friday, 16 April 2010

Toggle NERDTree


Putty Colours Config

Just pop this here - for setup on Windows machine:


Don't forget to save this as your "default settings".

Another nice touch is turning font anti-aliasing on and selecting a funkier font.

Friday, 2 April 2010

Apache SSL configuration and Django


Using mod_wsgi to support Django development via Apache

I'm building a Django shop which integrates PHP components in the form of a blog (Wordpress) and needs testing with SSL. As discussed in the last blog entry- that means I can't well test holistically using only the Django dev server. On production I am using mod_python at the moment, which is inappropriate for development as I would need to restart the server each time I make a change.

I'm looking to follow a recipe, using mod_wsgi for two purposes:
  • To have a dev server running via apache, to integrate PHP and SSL components, as well as having "hot code replacement" to support development.
  • To evaluate mod_wsgi on the basis of the claim: "for hosting WSGI applications in conjunction with Apache it has a lower memory overhead and performs better than mod_python".

Step 1 will be to deploying using mod_wsgi on my dev laptop per the mod_wsgi docs.
  • sudo apt-get install libapache2-mod-wsgi [ubuntu]
  • restart apache
Step 2 is a standard integration with Apache. Per http://code.google.com/p/modwsgi/wiki/IntegrationWithDjango
  • create appropriate .wsgi file
  • edit apache.conf with directive (see below)
Step 3 is getting it to then run in daemon mode. This is a little like what mod_jk or ruby passenger / mod_rails does. It brokers requests off to a separate sub process which forks and brokers off to individual Python Django scripts.

Step 4, finally per: http://blog.dscpl.com.au/2008/12/using-modwsgi-when-developing-django.html, is to add a monitor.py (the source of which is here) and reference it in the .wsgi file.

Here are example apache / httpd.conf:

And dev.wsgi (based of putting a monitor.py in the root of the django app):

Apache Virtual Hosts

In putting together this online shop, I'm going to integrate a Wordpress Blog, and need SSL in addition to my Django app. The Django development server is not going to cut it.

In order to build out my Dev laptop I'm going to setup Apache with virtual hosts, and setup Django to run in a dev mode but behind Apache. This is a hit in time upfront, but should sharpen a few skills on the sysadmin side, make development testing and deployment more straightforward on an ongoing basis. I should consider future project also, so I need to specify the clients as well as the sub-project in the URL.


So, I'm going to add a few hosts into my laptops host file based name resolver. So using "sudo vi /etc/hosts" I add these as aliases for the IP Apache is listening on. hostname blog.client1.net www.client1.net

I test these out with a browser and they all take me to the same default page served by Apache on port 80. Per:

If Apache has no vhosts, it will use the main server's DocumentRoot directory (often set to /var/www/html).
Bingo. Now to change the behaviour of Apache.

Within a vhost block--between  and  tags in httpd.conf--many directives may be given, but only two are typically required: the ServerName and the DocumentRoot directives. As a matter of good form,vhost blocks and related directives should go at the end of the httpd.conf file. 
So using "sudo vi /etc/apache2/apache2.conf", added the following:

Friday, 26 March 2010


I've got some money I made from creating websites for people. I don't make much money making websites for people but my instinct is to reinvest it. I instinctively like Git-hub- see the wisdom of source control, benefits of Git, need for a private repository so I subscribed. We shall see how it goes.

I plan to fork. [http://help.github.com/forking/]

Then use a Wiki page to chart development progress.

Wednesday, 17 February 2010

Nothing new under the sun...

The web, so far, is semantically implemented exactly the same way as COBOL/3270 code, circa 1970. You have a dumb datastore, managed by bespoke code, talking to a semi-smart block mode terminal. 


Monday, 15 February 2010

Upgrading Centos 5.3 to 5.4

yum clean all
yum update glibc\*
yum update yum\* rpm\* python\*
yum clean all
yum update
shutdown -r now

Wednesday, 3 February 2010

RHCE Cramming Cheat Sheet

This week I'm taking the RHCE Fast track course. I have some Linux experience, but I am basically winging it. Others on the course had taken Red Hat's advised course of taking the RHCT first, which substantially shares a lot of content. First two days they were moaning we were covering old ground.

The key is knowing the commands associated with each learning objectice, I think. As then its possible to lookup the man page if unsure about the syntax. This seems like a great cheatsheet, for that purpose, for cramming:

http://conigliaro.org/wiki/rhce?do=export_html version to view from my phone :P

Test is on Friday coming...

Saturday, 23 January 2010

Android application with location awareness

Building on the last post, last week- I've developed further with my experiments with Android applications. It stuck me the most compelling aspect was the GPS capability, so I have progressed to a location aware base application.

The following blogs were all I really needed:
The first, I followed until it worked, and the second was a tweak to the design. I also added the project to Github:

Saturday, 16 January 2010

Android Development Environment on Ubuntu

I bought a Nexus One Google phone. Some day I thought I'd play around with the development possibilities, and in an idle moment last night, I wondered how hard it would be to get a "hello world" going.

I started with my laptop running Ubuntu 9.10, my Google phone and the USB cable to connect the two together.


Step one is to download the SDK. The link is indicated above. I just downloaded the Linux version and decompressed it into a folder under my user directory I set aside for Android stuff. I edited my ~/.bash.rc to add the tools directory to the path as they recommended also (see link below).


Next step was to install the ADT, which is an IDE based on Eclipse with all kinds of niceties for developing and compiling Android apps. Think of it like Visual Studio for Android. First step is to install the base version of Eclipse, so I can then add ADT as a plugin. So to begin, from the terminal:

sudo apt-get install eclipse

After that installs (over 200mb) I started up Eclipse (installs a link under "Programming" in the Gnome menu bar, which I moved to the desktop where I like my icons). As an aside I understand version 3.5.1 of Eclipse is required by the ADT, however by co-incidence this is the version which is installed by default in Ubuntu 9.10 when you apt-get eclipse. So those using another distro or version may need additional steps to ensure the correct version of Eclipse.

I then followed the steps below to install the ADT into Eclipse, basically going to help > install software and adding the URL Google provide for the ADT, and next, next finishing.


However there was one "gotcha", in that some third party libraries could not be found. I found the answer in a mailing list post (second link below) - basically I needed to add the Eclipse update site to the install of Eclipse - so the ADT can pull in some third party libraries. This allowed me to complete the ADT installation, which by the way took about 20 mins.


The final step in setting the ADT up is to associate the SDK with the ADT install. This is done by going into window > preferences in Eclipse, and on the Android page pointing at the location where the SDK is installed.

So, having downloaded the SDK, decompressed it, installed eclipse, installed the ADT into Eclipse and pointing it at the SDK I was ready to go? Not quite - Google have different version of the Android OS, and provide different modules for each. The Nexus One is Android 2.1, or API level 7. Frankly I don't really care about older versions, so I was just interested in the 2.1 SDK. I therefore followed these steps to install the 2.1 SDK:


All that being complete I followed the documentation to write a "hello world application".


As indicated in the tutorial, as a first step I ran the app on the emulator which I created as part of the tutorial. But really I was interested in seeing the application on my phone. The steps for that were here:


Again the Google documentation was excellent - however one Gotcha on Ubuntu was the adb server needs to be run as root, otherwise you see an access denies message. This can be done by starting the server as follows:

sudo /path/to/sdk/tools/adb start-server

And that was it! I had the "Hello world" on my phone. We were watching Raiders of the Lost ark in the background while I was tinkling on my laptop, and as a guide I started this when Ballac takes the statue off Jones in the Andes, and was completely finished by the time Jones miraculously arrives at the submarine base towards the end.

Thursday, 7 January 2010

A configuration file with an encrypted password in Python

I need to set configuration by environment in quite a few Python utils. I'd like to create an object with a filename as a constructor variable, and then get a dict of the values. One issue is the password -  better not to keep it in plain text. So I need to make them "encrypted", or a better phrase might be "non-human readable".


I got my head round the python unittest framework when writing this, and also got my head round the ConfigParser and optparse libraries.