Monday 2 November 2009

Automated Rails Website Clone / Mirror with WGet and FTP script.

As a bit of paid work outside of my day job I wrote a css layout for someone's business website. It was just a product presentation site. Eventually I was being asked to update it all the time so I wrote a rails app to allow stuff to be added by the customer in a 'wiki' style manner: [http://code.google.com/p/rapid-space/].

At first I had this served from my host, but it was a bit slow and occasionally my host had issues. So I've moved the rails app onto another domain, and each week I'm looking to configure a copy process to mirror the rails site, then FTP it up to my customer's own public host.
Clone locally
First step is to clone the site generated from the rails app into a static folder on my host, which is running CENTOS. I did this using wget:
  • [root@host ~]# wget --mirror -w 2 -p --convert-links -P foldertocopy2 http://rooturltomirror/
Annoyingly, the site came out unviewable: the HTML was OK, but the link to the CSS and the images were all broken. This was due to a random number appended to each CSS and Image URL such as ?323124124. These were generated using the Rails Asset Tag helper: http://api.rubyonrails.org/classes/ActionView/Helpers/AssetTagHelper.html
Given the scale of this site, there is no benefit to leaving the asset timestamps feature switched on, so I turned it off as follows ... in /config/environment.rb add:
Finally, I was missing files which were linked to via the CSS file, i.e. via @import url("importthis.css"); or images which are only referenced by the css file. Given these files are not subject to change, I'm going to overwrite them from a static folder when I upload to the public host- I can use the same mechanism to blank out the forms (which won't work on a static host).
FTP to the public domain.
I just tried to use the ftp client built into CENTOS for this, and echo the commands into it via the EOF mechanism in a shell script, per: http://www.cyberciti.biz/faq/linux-unix-autologin-cron-ftp-script/

However, using the standard ftp command you can't recurse through directories. I.e. you can copy all files in a directory, but not descend into sub directories. First I looked at a scripted option such as: http://expect.nist.gov/example/rftp . This approach made me nervous so I upgraded to lftp on my host ... [yum install lftp], and found I could upload the site as follows (with the command 'mirror -R'):
  • [root@host ~]# lftp public-host-ftp-server -u ftpusername, ftppassword -e "mirror -R /local/source /remote/destination"

No comments:

Post a Comment