Replication & backups with Ruby

If there’s one thing certain in life, it is the uncertainty. As you go higher up in the ladder of life, the fall grows steeper, risk becomes greater. Same rules apply in the digital world.

In the process of building and maintaining software, there are plenty of accidents, ways to screw things, foolish mistakes and enemies to sabotage work. So most would agree it’s sensible to have some solid backup strategy as your insurance policy in case a disaster strikes your budding app. But when you say backups & redundancy, it sounds really expensive and time consuming, isn’t it ? Well, Not anymore; with all the cloud services floating around it’s possible to have a good data backup plan with few additional bucks. So in this post I’m sharing a general backup approach easily implementable using wonderful Ruby Backup Gem that you could use or adopt according to your application needs and risk.

When it comes to data replication and recovery, there are various aspects you might need to look into depending on the nature/scale/risk of your data. The method suggested here assumes there are 2 servers for the master and slave where you can setup data replication with MySQL. Also this approach was largely inspired by the strategy adopted by Marco Arment for Instapaper.

Here is an overview of the strategy suggested in this post.

Backup Overview

Backup Overview

1) To achieve point-in-time recovery I’m using a simple master-slave database setup with MySQL. This is very straightforward and Here, here are some examples on how to set it up.

2) Install Backup Gem

gem install backup in both master and slave servers.

3) Setup MySQL binary log syncing from master to slave

– Setup Backup config in master (This will create a ~/Backup/config.rb file)

  sudo backup generate --databases='mysql' --storages='s3' --compressors='gzip'

– Additionally create a default config file defaults.rb (Put mail alert, twitter alert configs here)

– To sync binlogs every 5 mins put this code in ~/Backup/config.rb. Do usual SSH key copy procedure to avoid password prompting when rsyncing.

– Update crontab to sync MySQL logs every 5 minutes

0,5,10,15,20,25,30,35,40,45,50,55 * * * * /bin/bash -l -c 'cd /home/username/Backup /usr/bin/backup perform --trigger sync_logs --config-file
/home/username/Backup/config.rb'

4) In the slave, follow the same process as in the master and setup the backup directory + config.rb. Additionally create a default.rb to store common configurations.

– Create config.rb

  backup generate --databases='mysql' --storages='s3' --compressors='gzip'

– Add email, twitter notification settings in defaults.rb

5) MySQL binlogs will be synced with S3 every half an hour. For this add half_an_hour.rb to your ‘Backup’ directory (All this can be put in config.rb as well. But for the sake of clarity I’m separating them based on the frequency).

6) Daily backup a full copy the database to local disk. For this use the daily.rb script.

7) Weekly store a full copy of database in S3. Use the weekly.rb script for this purpose.

8 ) Use Dropbox to store a copy every month. Plus if one of your workplace machine is synced with Dropbox account it will get synced to the local machine automatically and you can burn them to disks.
Use monthly.rb for this task.

9) Finally update the crontab in your slave to run your backup scripts according to frequencies you intend.

#sync every 30 minutes
0,30 * * * * cd /home/username/Backup  /usr/bin/backup perform --trigger sync_backup --config-file /home/username/Backup/half_an_hour.rb
/var/log/cron/cron.log  /var/log/cron/error.log
 
#backup daily
0 6 * * * cd /home/username/Backup  /usr/bin/backup perform --trigger daily_backup --config-file /home/username/Backup/daily.rb  /var/log/cron/cron.log
/var/log/cron/error.log
 
#backup every week
0 6 1,8,15,22 * * cd /home/username/Backup  /usr/bin/backup perform --trigger weekly_backup --config-file /home/username/Backup/weekly.rb  /var/log/cron/cron.log
/var/log/cron/error.log
 
#backup monthly
0 6 26 * * cd /home/username/Backup  /usr/bin/backup perform --trigger monthly_backup --config-file /home/username/Backup/monthly.rb  /var/log/cron/cron.log
/var/log/cron/error.log

So at the end of this process you will have several redundant copies of your database as well as MySQL transactions up to last 5 minutes. In case of an emergency (ie: disastrous SQL query) you could pick up the nearest full backup and apply MySQL binary logs one up till the disaster occurred and you are good to go.

As a last note (not the least) kudos to the team behind Backup gem. You guys made my life lot easier!

Elections & Prosperity

Election times are upon us…. well, not that there weren’t enough in the past few years, but only that an election which people care at least a dime is nearby and thus some changes to the Sri Lankan social behaviour.

Larger than life political faces materialising in every nook and corner of the country. Roads that were meant to be built in years are built in days. Living cost that was said to be impossible to come down even by godly interventions are coming down like melting butter. Roads that were closed due to terrible security hazards are being opened. Rulers’ godly status has been threatened and thus travelling alongside us – the mere mortals, in the road. Common mortals whom were lethargic due to the lack of gossip matters have again got something to shout about. Oh, almost forgot about free comedies and the entertainment…who needs Mr. Bean ? Considering all this, one might say that we should hold elections every month to see some positive developments in the country.

This is what I call the election prosperity.

But with such a polarised and biased grounds, is there any meaning to all this hype ? That’s another story for another time.

Sri Lanka, a new future ?

Last week was a very memorable one for all Sri Lankans, in whatever way you look at it. End of a 30 year old conflict is not something you forget that easily IMO. Younger generation today (me included) hadn’t seen a day without a war. Each month (some times every week) there were bomb blasts going on…people dying here and there… so it’s not a great wonder to feel relief after such a devastating conflict.

But I can’t really enjoy it either.

The cost of ending this brutal war has been massive. The war has brought so much grief to most families in north..family members dead .. closer to 300,000 displaced in camps…more than 6000 army soldiers have sacrificed their lives (in last 3 years of war alone, according to recent stats)…most people have lost whatever they had..so it’s not a party time either.

At most it’s bitter sweet.

But due to all these sacrifices from Sri Lankans, it has brought a new chance to look at the future in a new light. But it should be remembered specially the tri-forces who had put their life at risk and some who have laid lives, secretary of defence for his unwavering goal to finish the war and president for supporting it. They have done what four Sri Lankan presidents, eight governments, and 6 cease-fires could not. No matter whatever blunders the government has done, I don’t think any one can really blame government for going at active war…because they have tried everything possible through peace processes to come to a solution. Specially previous government through its MoU and even President Chandrika Bandaranayaka earlier with her willingness to give north to LTTE for 10 years without an election shows the desperation Sri Lanka wanted to have a peaceful solution. It’s no wonder because with the ongoing war country was basically at a stalemate when it came to the economy and growth. Prabhakaran never gave a compromise (even backing down a notch) from what he wanted and there was a limit the government could promise, which was a federal solution..which he despised. So when one party is not willing for a compromise, it’s a fact that there’s a conflict or a war in this case whether you like it or not.

But here we are…so what should be done ?

To where ?

First and foremost, there are about 300,000 displaced people in the north that are in very hard conditions. Also according to stats, it takes around Rs 3 crore per day and around 1 billion Rs per month to maintain these camps. Without a doubt they have to be resettled back asap and hopefully within 6 months as suggested by the government. But not only resettling them, rest of the country have to provide all the help they can to bring back these destroyed cities and villages to normalcy as much as they can be. Also it’s required to have a rapid development plan in these areas to bring them to the level of rest of the country.

But the real rock and hard place will be to bring peace through some political solution that’s acceptable to everyone.

Peace and a new future ?

As many believe Prabhakaran may have been a very skilful military leader…but that’s the only thing that can be said about him. If he had an iota of political sense most of this destruction could have been avoided through any of 6 peace processes underwent. Then what about all lives he took of intellectuals (specially that of Tamil leaders that didn’t go well with his ideology, or terror) ? That will take many years to fill that gap.

Through terror he shut down most of powerful moderate Tamil voices and except for a handful all were fearful of raising their voices. Personally, I feel there’s a lot of resemblance between the character Lord Voldemort of Harry Potter series and Prabhakaran in many ways. Both were megalomaniacs who thought they were like deities and made people even fearful of uttering their names much less voicing different opinions.

So now with the end of the war, the bottleneck is gone and Tamils have the freedom to express their opinions without a hindrance.  Also it’s government’s responsibility to let the freedom go to people.

People should use this to send representatives (preferably not idiots like some of those who are now in Parliament) who really care about them. Then use them to bring their grievances to discussion table and resolve it politically and peacefully.

The responsibility of the government is to use this golden opportunity provided by sacrificing thousands of lives by soldiers to bring all parties together (mainly UNP, as they have suggested their agreement to support the government in a political solution to get necessary votes in the parliament) instead of watering its self image (it may be justifiable to a certain extent because this is politics after all, but extreme is always ugly).

Also general public should think in terms of their contribution to the country. If a soldier can put their lives at risk and work 24 * 7 without a holiday, shouldn’t the rest at least work their 8 hour work load per day dutifully ? What right do we have to talk about soldiers if we can’t even do that ? [“Even after increasing it from 116,000 to 200,000, a soldier still cannot get seven days leave a month, he pointed out.”]

So we are at a cross road…I really hope it leads to somewhere better than past three decades.

My best 7 free JS modal boxes

I have been working on a new iteration of HOT (follow @hotelotravel for more info) for last few weeks and thought of changing the existing JQuery UI Dialog box for something bit fancy and solid (on other hand I may have just wanted to get a break from usual PHP stuff and to play with JQuery a bit after some time). I did have some popular JS modal box names such as Lightbox, Facebox and Thickbox that I wanted to test and found some few new names on the way.Certainly there will be many more modal boxes out there that I’ve missed and not to mention my requirement will be different from yours, but here is the gist of each modal box in my opinion of those I’ve tested so it may help someone to pick the right one at the right moment.

JQuery UI Dialog

JQUery UI in Hotelotravel

JQUery UI in Hotelotravel

This is the dialog box that I’ve used in most cases and of course it’s great. Few years back when I started working with it I noticed few issues when closing the dialog and such but by now they have all been fixed. Also it’s continuously maintained by JQuery community so you can be sure it is solid. What I like mostly about it is its simplicity as well as customization power through various callbacks when you need more action (you can define what to do when a drag starts, drag stops, box shows up, box closes. etc) through various option settings.

This is all you have to do to get a simple dialog box if you have a div with the id of “myDialog”.

$("div#myDialog").dialog();

Another perk of this modal box is its file size, which is quite small (about 10KB and minimized version is about 6KB) and when doing a complex site with numerous CSS and JS scripts, size of each file becomes crucial to maintain a small load size to reduce the load time as well as save server bandwidth. Only seemingly downside of this box is that it doesn’t come with any fancy preloaded stuff (themes, effects, preload-images.etc) but for someone interested (and with a bit of JS and CSS knowledge) can customize them.

LightBox

Light box

Light box

This is focused mainly on presenting pictures and does a good job at it. If you are interested in creating something like a picture gallery without touching much JS, this could be the ideal JQuery plugin for you. Also it has a relatively low size with a size around 19KB and packed version is about 6KB. But since I was looking for something more with raw customization power, this wasn’t the choice for me.

FaceBox

Facebox

Facebox

Another popular choice for a modal boxes and it deserves the name. It has a very small file size and a simple code. It also comes with a default theme and can be a convenient choice for hasty tasks or people 😛 But the downside I noticed is that it gives very small customizing power to the user through JQuery code (which is the case with Light box also btw).

It’s simple to a fault and you just have to name the class name of the link you are going to pop the box as “facebox”. It could have had few more options such as to define the basepath of the project as I couldn’t find a way to define it for pre-loading images without hacking the code nor a way to give width and height manually or pass callbacks. Also this project seems to have been abandoned for a while now and if you are considering adapting this box for your whole site check it thoroughly.

ThickBox

ThickBox 3.1

ThickBox 3.1

ThickBox is really a cool JS box. It’s simplicity and extremely small file size makes it very adorable. This modal box gives some customizing power but still focuses mainly on simplicity and link naming as “thickbox” which is its magic word. However as mentioned earlier this has a better customizing power through JS than FaceBox or LightBox so it’s more flexible. With some hacking you can also give your own callbacks and options as you like.

LightWindow

LightWindow

LightWindow

This is an all purpose, very fancy looking modal box done using script.aculo.us and prototype libraries. It can host all kind of media types and even flash clips which is really impressive. But the obvious downside is its huge file size of about 60KB which is a huge amount when considering this will be just a small part of Hotelotravel.com and for general use it’s not tolerable IMHO. It is not compressed or packed by default so you can manually minimize it using something like YUI compresser to bring it to somewhere around 30KB but even then it’s too large for me. But this is ideal for a site that has tons of ajax stuff and in need of a very customizable modal box with lot of options and callbacks that can be used throughout the site.

SimpleModal

SimpleModal

SimpleModal

This is the smallest JQuery modal box I’ve seen and if you need to customize your modal box to the extreme I’d suggest this. Downside is that you will have to write a lot to get something done through this modal box but extreme small size complements that.

NyroModal

NyroModal

NyroModal

This is also a very good looking JQuery based modal box with a reasonable file size(about 14KB for packed version) and gives considerable customizing power to the user. Also it comes with a packed theme and all that so you can use it easily without much coding which is a plus. In fact, I had some trouble deciding whether to use this one or JQuery UI for my task but finally settled on using JQuery UI because my familiarity with it and community backing. So I guess in my case JQuery UI is the rightful winner 🙂
[digg=http://digg.com/programming/Best_7_free_JS_model_boxes]

Tagged , ,

Drupal with Dreamweaver

I had to migrate and setup the whole workspace at my old desktop machine last week after the breakdown of my HP notebook. So while setting it up I thought this little piece of trick could help someone who is fond of Dreamweaver and looking for an IDE to code Drupal.

As a side note, one could accuse me of promoting unethical software and in fact I don’t refute it (even though it is not my intention). I also know there are many great open source editors for coding PHP like Eclipse-PDT and even Gedit which could be transformed to a very helpful IDE. But on other hand I’m kind of addicted to DW after long years of working with it and if someone who already has DW looking for a way to use it for coding Drupal this could be helpful.

The most frustrating thing when coding Drupal modules or themes with DW is that it doesn’t recognize it as a PHP script and doesn’t give any validation or auto completion features like it usually does for php scripts. To fix this there’s little configuration to be done.

First find the installed directory of DW (if it’s Windows it most probably in ~/Program Files or if it’s Linux+wine it should be in your virtual windows environment) and go to Configuration->DocumentTypes. There should be a xml file called MMDocumentTypes that holds configurations regarding which language should be used for a given file format.

Find the line that says,
‘<documenttype id=”PHP_MySQL” servermodel=”PHP MySQL”….’
and depending on weather you are using windows or mac, append ‘,module,install,theme,inc’ to existing winfileextension or macfileextension values.

So for windows it should look like this.

<documenttype id=”PHP_MySQL” servermodel=”PHP MySQL” internaltype=”Dynamic” winfileextension=”php,php3,php4,php5,module,install,theme,inc” />

Now restart DW and open your module or theme file. Ta da!..it should now work as a normal php script.

Hope this will be helpful to someone looking for a way to convert Dreamweaver to a more Drupal friendly place.

Tagged , ,