A typical use-case is when we have a couple (or more) web servers independent of each other (say, webmail web servers: webmail.mydomain.com), I’d add multiple A records for the same domain so that DNS resolution happens in a round-robin manner at the client’s end. i.e.,

dig +short webmail.mydomain.com
184.xx.yy.154
184.xx.yy.155

The client uses the first IP it gets during resolution and as the number of clients that are resolving your domain (and making HTTP requests) grow, you’ll start seeing a more or less equitable distribution of HTTP requests hitting each of your hosts.

These IPs are meant to float and be handled by your spread/wackamole tools. (i.e., they’re not hard-configured into network config files like permanent configs: i.e. /etc/sysconfig/networking/ifcfg-*). Say, the .154 IP was on host A and .155 on host B, host B goes down, spread daemon on host A detects that host B isn’t responding to "are you alive?" requests and instructs wackamole daemon on host A to take over the IP that host B had (.155). Sometimes – depending on the router in your environment – one might have to send a gratuitous ARP packet to the router and hook this up with wackamole’s "post-up" action.

This post is about how I couldn’t find usable RPMs for spread/wackamole (and was in a time crunch to shave that yak) and looked for an alternative.

Pacemaker and Keepalived are known entities in the market. So is UCARP (as userland implementation of BSD’s CARP for Linux). Being on a time crunch and noticing how the former options seemed a little complex at first sight, I settled on deploying UCARP.

The configurations on the Internet typically show how one IP is floated around between two hosts. Now this doesn’t let me have DNS-based round-robin’d "load" balanced incoming requests. So here’s how I configured UCARP on host A (assuming you have installed from EPEL repo as `yum install ucarp’):

[root@web02-dal07 nvenkateshappa]# cat /etc/ucarp/vip-001.conf
# Virtual IP configuration file for UCARP
# The number (from 001 to 255) in the name of the file is the identifier

# In the simple scenario, you want a single virtual IP address from the _same_
# network to be taken over by one of the routers.
ID="001"
VIP_ADDRESS="184.xx.yy.154"
PASSWORD="love"
BIND_INTERFACE="eth1"
SOURCE_ADDRESS="184.xx.yy.179"

# In more complex scenarios, check the "vip-common" file for values to override
# and how to add options.

And on host B:

[root@web01-dal07 nvenkateshappa]# cat /etc/ucarp/vip-001.conf
# Virtual IP configuration file for UCARP
# The number (from 001 to 255) in the name of the file is the identifier

# In the simple scenario, you want a single virtual IP address from the _same_
# network to be taken over by one of the routers.
ID="001"
VIP_ADDRESS="184.xx.yy.154"
PASSWORD="love"
BIND_INTERFACE="eth1"
SOURCE_ADDRESS="184.xx.yy.180"
OPTIONS="--shutdown --preempt --advskew=10"

# In more complex scenarios, check the "vip-common" file for values to override
# and how to add options.

The above vip-001.conf on the two hosts is for managing the first floating IP, and the following are for the second: vip-002.conf

Copy over the same configs on each host, change ID to 002, VIP_ADDRESS to 184.xx.yy.155 and swap the OPTIONS line.

The –advskew option (advertisement skew) is what gives a sense of affinity for your virtual IPs.

Let me know in what other interesting use-cases you’ve used UCARP in.

The cause seems to involve a couple of things: 1. c6 having adopted a stronger file digest algorithm (sha256 as opposed to md5 in c5) and 2. compressing the payload with xz (as opposed to nothing in c5).

This is easily remedied by passing relevant options to `rpmbuild’ and `createrepo’.

If you’re using fpm in your CI, you can now append the following to your fpm command invocation:

--rpm-rpmbuild-define '_source_filedigest_algorithm md5' \
--rpm-rpmbuild-define '_binary_filedigest_algorithm md5' \
--rpm-rpmbuild-define '_source_payload nil' \
--rpm-rpmbuild-define '_binary_payload nil' \

And invoke createrepo as `createrepo -d -s sha1 –update /path/to/rpms/for/c5′

i.e.

dig +short @8.8.8.8 qa.example.com

<returns public IP>

dig +short qa.example.com

<returns private IP>

If there aren’t too many consumers for this name resolution, we could’ve done with putting in entries into /etc/hosts on each consumer host. But our consumer hosts included a lot of android phones. And we didn’t want to root them all to be able to modify their /etc/hosts.

If we were running our own DNS server in our DMZ, we could’ve configured the infamous split-DNS setup through BIND or tinydns. (Un)fortunately, we didn’t.

The first "workaround" to this was to maintain a duplicate zone for example.com on our local DNS server (the one served by our DHCP server) and override the records as required. This would soon start to suck.

A colleague of mine – who didn’t take my word that the above two methods are our only options available – persevered through the PowerDNS docs and found an option where it could serve off the host’s /etc/hosts file. Now what was brilliant about this was, adding a ’192.168.1.223 qa.example.com’ into /etc/hosts effectively solved the problem we had!

We didn’t really needed a full-fledged DNS server like PowerDNS and I looked if dnsmasq could solve the same problem – and it does.

[root@blr-vbox1 ~]# egrep -v ‘^#|^\B+’ /etc/dnsmasq.conf
bogus-priv
resolv-file=/etc/resolv.conf.isp
interface=eth0
no-dhcp-interface=eth0
no-dhcp-interface=lo0
conf-dir=/etc/dnsmasq.d

Our ISP’s DNS servers go into /etc/resolv.conf.isp

Ensure that the host that’s running dnsmasq has only ‘nameserver 127.0.0.1′ in /etc/resolv.conf and put in all your entries to be served into /etc/hosts.

Ever since I got back to B’lore last November, I’ve been commuting to work on a car. It’s a 2006-07 Hyundai Getz and still works quite well. What makes it an unpleasant and at times depressing ride is when one is stuck inside it in B’lore peak hour traffic. One can’t hit the second gear for an hour or more. So much wasted time sitting idle on a seat waiting on slow minds – who don’t realise how traffic rules are meant to help them be efficient on the road – to make way. Oh, but the rules themselves are harebrained in certain scenarios: for instance, traffic lights in a junction that turn green in a round-robin fashion with equal time intervals for all roads on that junction.

Last week I acquired a Trek 7.1 FX. It’s a hybrid that’s supposed to roll well on city roads. I took it out for a test spin to work on Saturday to test the waters: to see if the goal of riding nearly 12 kilometers is doable at all without sweating too much. Turns out that it’s possible.

Two years ago, at around the same time, I had acquired a Hercules ACT104. It’s supposed to be an MTB. But I didn’t know at that time that good MTBs don’t come that cheap at all. That cycle was a deadweight to ride with. The weekend before last, I had taken the ACT104 out in an attempt to see if I could ride all the way to office. I had to give up at one-third the distance. It was too heavy or just didn’t roll well. Previously at college, I was a proud owner of a BSA Mach III. I could literally glide on it at times. All this made me start looking at cycles with narrower tires.

My ride to work is nearly 11 kilometers and the ride back is almost 18 kilometers (I ride to NGV/Koramangala and then toward home). I’m not sure how tired I am as I write this, but being just the first day, I’m looking forward to improvements.

P.S. The Trek cost me my 6 month petrol bill (assuming the petrol price doesn’t increase in the next 6 months)

P.P.S. The Nissan Leaf isn’t available here. The Prius is twice the amount it costs in the US.

UPDATE on 22nd:

Some of you might be looking for details such as:

1. I wear a dry-fit jersey while I ride and carry a casual t-shirt (which I change into while at office) in my backpack along with my laptop. I don’t sweat as much thanks to the awesome weather of BLR and I don’t have a shower at work. (Among the metros in India, I believe only BLR has this weather advantage.)
2. The time to commute on bike almost equals that of the car. Varies a bit now and then depending on the traffic. At least with a bike one can pull it off onto the footpath and walk it up until there’s road to ride.
3. I park the bike at the basement at work. I use a coil-like lock and lock it to something.
4. Being the second day now, I think goggles for eye protection and a decent pollution mask are necessary.

India is quite densely populated too – at least in the metros – and yet we’re stuck decades behind countries like Romania (which recently surpassed the USA as having a better average Internet speed). I wonder if this is a general trend in tropical countries. The lack of long-sightedness or long-term thoughts or decision-making. Maybe we’re all too comfortable around here (compared to Scandinavia or Siberia, for instance).

The concept of a “good service” is poorly understood in nations with high population densities. It’s a matter of numbers you see. Having been close to one of the largest online service providers (for canned commodities such as web or mail hosting), I’m all too familiar with the sort of attitude an Indian businessman possesses when he markets a service. You’re not important. You’re just another node in the huge graph. If I lose you as a customer, there’ll be another. I can go on losing a customer per day because I’m used to the fact that there’s always another customer (or two) who’ll take your place. So why are you special again?

This is applicable to practically everything around these parts in the mid-sector. What I mean by mid-sector is: the sector that addresses the mass. The middle-class.

A middle-class-style lunch house or restaurant would not have the sort of waiters who’d even recite the menu politely to you. You’ll have to put in extra efforts to get attention. You’re just another customer (of the numerous that go in and out each day).

Oh, and hygiene. Since you’re replaceable or not too important as an individual. I wouldn’t take extra care when it comes to hygiene or even fresh food ingredients (especially the non-vegetarian processed foods). Don’t expect that chicken salami sandwich to be fresh all the time, they will not get rid of it by themselves. And forget about expiry dates on them.

Same applies to roads. Nobody seems to have put in a long-term vision and taken the stand to take the initial losses in order to gain the long-term profits. There was an estimate someone had put out stating the astronomical losses India faces per year solely due to lack of decent roads. Day to day commutes to work and back home is quite taxing – both mentally and physically.

Oops. Let me get back to the Internet service provider issue now.

So, FTTH is still quite costly around here. Sure, there are cables laid around here and there. (And by nature, the first telecom entity that bids and gets fiber optics laid out is always the loss-maker. But someone’s gotta do it). I don’t know if the likes of BSNL are sitting on their butts waiting for people to finally give in to the insane prices or just underprice the damn thing (and take the initial – short term – losses) and reap the long term benefits of increased subscription-base.

Sure, this doesn’t always work: Volkswagon Polo, for instance, underpriced their 10L factory costing car to 6-7L but still haven’t gained the sort of traction the Hyundais or Swifts do. But it’s worth a try – especially in an increasingly net-savvy metropolitans.

Where I live, it seems only Airtel and BSNL are the only ISPs around. We did have BSNL earlier and not knowing that it had better speed plans, I took the plunge of getting rid of them and signing up for Airtel (whom I’ve had a good experience with while I was at Mumbai last year). The 4mbps connection is what I had and was quite happy with it until this August. When the rains began pouring in Bangalore.

The signal-to-noise ratio would deteriorate quite badly whenever it rained. The issue was quite obvious – anybody with half-a-brain would attribute this to poor sheathing of the copper cable – which isn’t in my home – out there anywhere between the CO and my home.

Half of August (beginning late July) had near to no connectivity and almost everyday I had to call up the Airtel customer service (a. annoying ringtone and b. poor internal communication). I resorted to trying to get in touch with their “social” faces: @Airtel_Presence and the airtelpresence email contact.

I got a call on two occasions from two different people (Airtel_Presence), the first time, I had to explain how this wasn’t an issues that requires modem reinstallations or any of the usual circus the technicians tend to do (especially around these parts, you know “Dell Support, how can I help you?”) and the second time, I had to report back saying “nope, hasn’t been fixed”.

I ran out of patience, signed up for BSNL somewhere in early August and placed a cancellation request with Airtel. Someone calls me up again (and again) asking me why I was going to cancel the connection and I regurgitate the same set of sentences and so did they “Sir, I assure you that this will be fixed”. Which rarely does.

It seems that they finally figured what the real issue was. Perhaps a lot of complains from around this locality might’ve hinted at the root cause. And soon enough, the connection started faring better since mid-August. But on one bad day in late August, the issue resurfaced. And I raised a cancellation request again. The next day things were back to normal so I got the cancellation request cancelled. Or so I thought. Until last Wednesday.

The Airtel Presence folks mailed me back:

Dear Customer,

Thank you for writing to Airtel.

This is with reference to your e-mail, where in; you requested regarding
the cancellation of services towards your Airtel landline connection
0804168XXXX.

However; later you confirmed us that you would like to continue your
patronage with Airtel. Please be informed that we are glad to have you
with us and look forward to an opportunity of offering you the best of
our world class products and technology in the times to come.

It is our privilege to have you as our valued customer and would like to
thank you for your continued support. We look forward to a warm and
fruitful long-term association with you.

Should you wish to take a landline/DSL connection with Airtel in future,
kindly contact the undersigned at airtelpresence@in.airtel.com.

Regards
XXXX Kachroo
Airtel Presence (Airtel Customer Service Team)
Bharti Airtel Ltd
airtelpresence@in.airtel.com

I was somewhat impressed at this point. “Man, Indian service providers are finally taking things like Twitter and customer service seriously”. I guess I was happy too soon.

Last Wednesday, my line goes dead because someone de-activated my connection due to some miscommunication (or so they claim).

I reply back (5 days ago):

Looks like there’s been a miscommunication on your side. Airtel has
discontinued my connection without any intimation. As a person working
in the IT field and relying on being able to connect to corporate VPNs
at odd times, this is very a very unwelcome experience. Please
re-enable my connection ASAP.

and got no reply. I call up the customer service each night after work, hoping for a positive response and on day one, support executive A listens to the issue, but forgets to log a request (I have no request number with me). Day two, I re-explain the situation to another support executive B and get a request number who says that “the issue would be fixed” (standard phrase around these parts) by 23rd September and gives me another request number.  So I don’t call them on Friday, Saturday or Sunday.

There goes another night of unproductive work. Honestly, why is it so
hard for an Indian company to take service seriously. Does it always
have to be the likes of an Amazon?

I call them up on Monday morning to see what the status with the previous request is. And support executive C has no information regarding this request number. So she takes a fresh request and gives me yet another request number and states that “the issue would be fixed” (what did I tell you?) by 5:30pm the same day. I sighed a breath a relief. “Finally I’d get my Internet back” and go out. I reach back home with great expectations by 8pm only to be disappointed again.

I call up the service line again, speak to support executive D, who puts me on hold but the call got cut.

I call up again, speak to support executive E, told him that the previous call got cut – “could you please check request number xyz which I got this morning.” Puts me on hold, call gets cut again.

I call up again, speak to support executive F, requested him not to cut the call and check my request number xyz – which he claims didn’t exist. He puts me on hold and the call got cut again.

I call up yet again, speak to support executive G, told him how three calls of mine were cut; so he escalates to his senior (or so he claimed) and I speak to person H who couldn’t find anything regarding request number xyz (so I was under a false hope all day… expecting “the issue to be fixed” by 5:30pm. I explain the whole situation again explaining how all of this has been a giant miscommunication and that I didn’t had to have my connection de-activated and requested him to contact a backend team person immediately and flip that switch which gets me my connection back (all the hardware apparatus is here BTW, even the “Link” LED is blinking just fine – barring the “Data” LED).

Support executive H has given me yet another request number and claims that “the issue would be fixed” by 22nd September. And that I should expect a call from the backend team soon.

I plan to call them up again and quote the latest request number I have to see if their CRM if fucked in head. Because it has eaten up two request numbers of mine already, it’d sure be a hat-trick if this one gets eaten to.

So that, gentlemen, is a primer on Service Providers in India.

Note-to-self: visit the nearest BSNL office if the third request number really does get eaten up tomorrow and find a data card for my sister who is being affected the most by this because she’s working on her final year project and relies on a lot of Javascript/NodeJS foo that’s not really selling in books yet – making online references mandatory for her.

UPDATE (2011-09-23):

Hello again,

Despite your phone calls and promises on Tuesday that the “issue would
be fixed” in two days, on calling up the customer service today I
found out that the comments were added in the wrong category and
tickets were closed leaving no traces of the grievance I’ve been
facing since last Wednesday – been more than a week now.

I have no words to describe this situation at the moment. I seriously
hope things will improve – at least in terms of available alternatives
to the horrible levels of service you seem to provide.

I am in no state to receive a call from you ever again only to tell me
“issue would be fixed” in yet-another-two days. Save it. Don’t call if
you can’t get this resolved ASAP.

Yet-another-new-request number issued: 4543080 (I hope this isn’t fake)

Nope. Hasn’t been fixed yet.

Yesterday’s news about Oracle planning to put Opensolaris to rest wasn’t as much of shocker as it was a late announcement.

The other news about Oracle trying to sue Google for Java is a little, what should I say, not completely serious?

Oracle seriously suing Google or what?

In other news, Babylon 5 is a good show. Vishnu recommended that I watch this show and I’m liking it a lot. In one of the episodes, there’s this surveyor who takes contracts to survey planets for various purposes and ends up at a mysterious place called Sigma 957. Suddenly, Sakai (the surveyor)’s ship encounters something never heard of before. Not in the database. Her ship loses almost all its power and leaves her exposed to a lone death – until the troops G’kar had sent for came to her rescue. G’kar had earlier warned her not to go there stating that it was a mysterious place and that Sakai would be better off not going there.

When Sakai returns and tells G’kar of what had happened:

Catherine Sakai: While I was out there, I saw something. What was it?
G’Kar: [pointing to a nearby flower] What is this?
[Upon closer inspection, an insect is visible.]
Catherine: An ant.
G’Kar: “Ant”!
Catherine: So much gets shipped up from Earth on commercial transports, it’s hard to keep them out.
[As Catherine is talking, G'Kar carefully picks up the ant.]
G’Kar: I have just picked it up on the tip of my glove. If I put it down again and it asks another ant, “What was that?”, how would it explain? There are things in the universe billions of years older than either of our races. They are vast, timeless. And if they are aware of us at all, it is as little more than ants…and we have as much chance of communicating with them as an ant has with us. We know. We’ve tried. And we’ve learned we can either stay out from underfoot, or be stepped on.
Catherine: That’s it? That’s all you know?
G’Kar: Yes. They are a mystery. And I am both terrified and reassured to know that there are still wonders in the universe…that we have not yet explained everything. Whatever they are, Ms. Sakai, they walk near Sigma 957. They must walk there alone.

And here we are. Calling out for a national bundh because of a petrol-price hike. And me enjoying the peace due to it.

Recent updates include:

a. moving to a newer apartment near Powai lake in Mumbai.

Moved in with my former (first one at college) roomie – Koushik Mandal(!) and his department-mate (and my batch-mate from Hall3), Sidhant Dash. They’re cool folks (barring Mandal’s mandalism at times). Each of us were living with other non-IITK, non-Y4 batch folks earlier – this was the primary criterion that prompted the move for each of us.

b. managing to read fiction.

I read ‘Blind Willow, Sleeping Woman’ – a collection of short stories and ‘Hard-boiled Wonderland and The End of The World’ (novel) by Haruki Murakami. He’s a Japanese author and I read the English translations of these. The last fiction I read before these was Agatha Christie’s “The Murder of Dr. Roger Ackroyd” four years ago! I’ve been a passive consumer all along – anime, movies, TV dramas, you know, the kind that don’t require the mental weightlifting as in the case of active media – books.

In a recent Wired article, two abilities were distinguished – “The ability to scan and browse is as important as the ability to read deeply and think attentively.” Being the kind of person who’d been chasing after Google search results until recently, I might’ve adopted the former as my dominant mode of thought and maybe reading “larger” material might improve the situation. Not that this is primary motivation to read- just that it’s a welcome throw-in that comes with it, I guess. Looking forward to another Murakami piece – ‘Kafka on the shore’.

c. weekly anime-sessions.

Vishnu, Karan and I meet up on weekends trying to catch up with anime. I haven’t been able to watch anime like I used to in the past year. But these weekly sessions are fun.

We’ve been watching ‘Aura Battler: Dunbine’, an 80′s show, it’s about 49 episodes long and I think we’ve reached 38 episodes through!

The past 3 weekends I’ve been introduced to the new ‘Dr. Who’ show and I’ve found it a really good British sci-fi show so far. I’m usually wary of insanely long shows – long as in several seasons over the years. For one, I fear getting really sucked into it and wanting to watch it all or read it all. The same reason that keeps one away from, say, ‘One Piece’. Maybe I should stop being such a “perfectionist”.

It’s been a while since I’ve posted an “updates” post so it looks like I’m going to have to recap quite a bit.

First off, today marks two months of my stay at Mumbai. It’s been an interesting experience in as far as the passage of time is concerned. I feel a lot busier than I used to and I think that’s a good thing. I’m beginning to make fuller use of the day unlike my (extremely) lazy days at college a year ago. Interestingly, today also marks the Hindu new year in the Western and Southern parts of the country (unsure about the rest).

And more interestingly, yesterday the GATE results were announced. Although I had registered for it, I didn’t appear for the exam. I was convinced I was way under-prepared and I took off to Mumbai. My juniors at IITK got their scores: Shashank did surprisingly well (99+ percentile), then it was Praneeth followed by Surya. Going by the numbers they’ve scored and the first-hand report on the sort of importance Theory of Computation holds in this paper, I’m slightly better informed now (would’ve been more better informed if I actually wrote the exam!) and this gives me more direction towards what to prepare for by next year’s GATE.

Rachit, a former wingmate of mine, flew from his Uni in Paris to Mumbai on 1st of March for a short detour homeward and then onto his new job at Paris. We had a reunion of sorts: Mandal (puts up at Kandivali), Kanodia (somewhere in Navi Mumbai, Ghansoli IIRC), Saurabh(!), Rachit and I. We had a good time and went down to Churchgate for what would be my first encounter with “Bademiya” – a smallish outdoor eatery specialising in primarily non-vegetarian food. It was busy. Really busy. But good too.

And last weekend was a weekend out with Mitesh (who so gracefully came all the way from Pune), Rachit (who had just returned from visiting his home), Saurabh, Mandal and I. We met on Friday, had a good night and the morning Rachit left to fly back to Paris. We had a good time on Saturday – watched “Alice in Wonderland” in IMAX! – Followed by another visit to Bademiya (Mitesh’s first visit).

I quite liked the movie. Before the actual movie started, a bunch of trailers (2D and 3D) were shown and boy is the IMAX 3D experience going to get big or what. Must thank Avatar to kick things off in India. Really. I liked the white queen in the movie. The movie was so sparkly and all, could be thought of as a *cough*acid trip*cough* (not that I know). It’s a must watch, really. It’s amazing how novel experiences can really brighten up our minds, raising our bars of satisfaction every time we confront something new.

Being a Linux user for a while now, there have been very few instances when I haven’t cursed Flash for what it is. An abomination on the web. I have not investigated whom exactly to blame here – the list of shared objects in `ldd /usr/lib/flashplayer-mozilla/libflashplayer.so` or the actual shit that’s in libflashplayer.so or the browser’s interface with this blob.

Awesome Web 2.0(tm) sites such as slideshare use it and trying to view a presentation fullscreen is more painful than watching your laptop die due to overheating. I’ve given up on watching youtube on the browser too. These days I wait for the video content to load into /tmp and play Flash* file with dear old mplayer. Such is life. Can’t wait to get my copy of Windows 7 license now.

Yes, I’m beating a dead horse here but I post this at the backdrop of the news of Mr. Jobs having announced that the iPad(sic) isn’t going to have Flash on it. I don’t know if I’m ever going to use this piece of locked hardware ever – maybe it’s useful in the hospitals – but the “no-flash” stance makes me happy.

I sincerely hope that HTML5 and related bling really does take off and take Flash off of my laptop.