April 4, 2006

AOL and the Offshoring question

I mentioned the AOLT All Hands yesterday. It was the first "public" showing for our new CTO (and her senior management team, myself included), and I thought she came off quite well: Talked the talk very crisply.

Now its time to walk the walk :)

However, there was a question that came up during the Q&A which I felt was very important, emotionally, and under-answered during the session. It was a tough question, and my post here is in no way intended as a slight on the answer given at the All Hands (which I did not deliver) - but with some time to reflect, I wanted to try and address it again. "Answering" it is probably beyond my reach.

The gist of the question was:
Q: There is a lot of fear and a lot of talk about outsourcing [and offshoring]. There have been a lot of initiatives to start outsourcing [and offshoring] certain jobs to Bangalore. But what’s not clear is if outsourcing working for AOL. We have several examples in AOLT now where we have successfully outsourced functions or systems and it appears that we should have enough data to be able to quantify those results. What are the key business metrics any organization should be looking at before they say, this is working, or no, it’s not working? Is AOL really saving money from these outsourced initiatives, and if so, can we get some information or data on that?

I think the underlying question can be fairly summarized as: "I understand that offshoring saves us money, on paper - i.e. cost per hour - but do we know (and how do we know) that it saves us money really, i.e. cost per unit of productivity?"

Or, perhaps, even more succinctly: "If its not working, let's stop doing it." (which isn't really a question :))

Here's the problem (and I realize this will seem like I'm dodging, ducking, dipping, diving and, uh, dodging): its just the wrong question.

If "they" are not as productive in our execution (which, of course, "they" are, mostly, though not always and not uniformly - just as anywhere) then that's STILL not a reason to not continue offshoring where it makes sense on paper. We should, naturally, know and deal with productivity issues, such as they may be - but it doesn't change the idea that we, as a Company, and as a managment team, should pursue offshoring and outsourcing.

The issue is, of course, that this is a really touchy subject - and for good reason - it triggers patriotic sensibilities, as well as real job security sensitivities.

However (and this is a big "However", because I don't want to or mean to dismiss these concerns lightly) we have to recognize that there are smart people ALL over the world. It behooves us to figure out how to be advantaged by them - this was certainly a major point that Eric Schmidt, CEO of Google, made when he visited, and he's right.

I'm not saying people are smarter elsewhere, but, tough though it may be to admit, all over the world there are people JUST as smart, and JUST as talented, and JUST as effective. To compete at the scale that AOL does (and needs to) - to compete at a global level - we need to be able to leverage smart resources all over the world.

No one asks, "Why do we offshore [sic] to Mt. View, California?" after all. As AOL, we need to be wherever the talent is, and that includes Bangalore.


Anonymous said...

There's a couple of factors that complicate working with BDC at AOL.

1. The AOL engineers in Bangalore are quite junior. When you go to Bangalore, this stands out: almost all of the engineers are in their twenties. They don't have any architects or principal software engineers. Which means that they need lots of support and mentoring from senior engineers in the US, which means lots of expensive travel time by those engineers.

2. AOL builds highly integrated applications. To do that, you need lots and lots of communication between development teams. When those teams do not share common work hours, this is difficult. Everything takes longer.

I've been out to Bangalore twice now, and I have a great deal of fondness and respect for the engineers out there. They are a bright young bunch. As you say, there's no qualitative difference between working with them and working with a group of similarly experienced American developers. (And many of the missteps that they take remind me a lot of mistakes that I made when I was a bright young engineer.)

But it is different than working with Mountain View. Because Mountain View has their own architects. And Mountain View time is only three hours different than Dulles time - not 10.5 hours.

So is BDC worth it? I don't know. I'd have to look at the numbers. All of the numbers - the travel costs, the costs of miscommunications, the costs incurred by the increased level of organizational friction that is inevitable in inter-continental partnerships.

There's only one thing I'm sure of: it isn't an easy question to answer.

Anonymous said...

Cannot comment on the engineering aspect, but having worked as a technical support consultant for aol in varying capacities (for the last 5 years), I can assure you the customers do not like Outsourcing.

When it first started happening in 2001 or 2002 we had to spend the first 5 minutes of each of our calls listening to people rant about talking to someone in India or Pakistan as they said (I didn't know we had people in Pakistan).

Even today, 4-5 years later. The most common first response I get from a customer when I give my scripted welcome is "Thank God you speak English."

I don't know about you but how productive can off shoring be if the people feel this way about it. Not very in my opinion.

Again I cannot speak for the engineering aspect of this, I do not know if there are a shortage of computer scientists in the US. As a CS student with full classes and waiting lists to get into them I seriously doubt it.

But as far as the call center goes, lets be honest, outsourcing has nothing to do with finding world class talent.

It's about saving money: anyway the company can, and at any cost. It makes the company look "lean" on paper.

It just so happens that so many companies did this at the same time, the American public had no recourse or ability to leave said companies as their competitors were doing the same thing.

Let's face it, American corporations simultaneously said "F.U." to the American public, and because they did it as a group, the public has no leverage to do anything about it. Corporations 1, America 0 in my opinion.

Again I speak from what I deal with on a day to day basis. And of all people, I got to listen to people tell me off because my company "outsourced." The irony is overwhelming.

I am not a dreamer, I don't think this is going away anytime soon. But I do see it as the victory of quantity over quality, at least as customer support goes.

0 said...

I think it is effective - cost effective that is (Indian salaries can vouch for that lol). Integrating things and transferring and sharing knowledge and information is not at all expensive in this day and age. I think the problem is reservation - thinking they can't pull it off, preconceptions of sorts or maybe just being territorial ... IMHO maybe it shouldn't be about "will it work out" but how do WE "make it work". it is possible, and I think it is working, with a little support and opening minds to new possibilities. And for that matter are all projects delivered cost effective anyway - is there a metric to define that especially if it is an extension of what is being done at HQ?

Anonymous said...

if it wasn't cost effective it wouldn't be a thriving business and it is.

Anonymous said...

It may work for other companies, but for AOL... I don't know!

riz said...

I think the comment about offshoring and language implies more to customer service, where your language skills have an impact on your job performance. I don't think this is valid case for engineering. Computer Science is largely language-agnostic.

The comment about AOL building highly integrated applications is an interesting one. I think the importance is in building a highly integrated experience, which does not necessarily require a highly integrated codebase. I think that is why we are building the OCP, and partly why we cannot continue on the WAOL platform.

Anonymous said...

"...almost all of the engineers are in their twenties...

Oh come on Joe ... being in your twenties is not quite a yardstick to judge an engineer with :P I totally agree with your points regarding increased communications overhead with offshoring ... but it is getting better and easier to manage offshore projects in truly "follow the sun" type development manner.

There are several aspects of "IT" that can be "oursourced" or specifically "offshored" and also there are several ways of going about it. Companies typically outsource:

- Maintenance/support
- Application development (aka Engineering)
- QA
- Help Desk / Customer Support
- Misc: hosting, data centers, backoffice, etc.

Also, they can either OWN the offshore operations, or outsource to a provider. At AOL, I think we're doing reasonably okay with the first type, but I've seen the latter botched up for some projects (but then again, we've botched up "on-shore" outsourcing as well IMHO). The reasons for outsourcing remain multi-faceted -- getting access to world-class resources is one, but the fact is that perceived cost-savings remains another large factor that companies, including AOL, associate with outsourcing.

Roopa as far as salaries are concerned, as the world converges towards a "global workforce", standards of living in the subcontinent, China etc WILL rise and so will the salary levels become more uniform across the globe. Already there's evidence of that happening a la IIT/IIM grads.

Anon 1".. our calls listening to people rant about talking to someone in India or Pakistan as they said (I didn't know we had people in Pakistan).

I seriously doubt we have or had any operations in'll be a few more years with the political situation stabilizing before there can be a Karachi Development Center (or call center for that matter) ... even though there certainly is potential there.

Anonymous said...

Doing overseas engineering work is difficult unless you work in a waterfall process, and have specs that are extremely detailed. In an innovative environment, that's tough to achieve.

I am 100% sold that overseas QA is a good thing, simply for the acceleration that it brings a dev cycle. You'll always need local QA engineers to sit with the product and engineering teams to really grok what's being built. The local QA folks do the test planning, build the test cases and manage the BDC resources.

For overseas QA, the time factor actually helps. It's great to spin a build, go home, have overnight testing occur and when you come in the next morning - you've got a (hopefully small) pile of bugs to scrub.

Some things are still challenging, like the early AM or late PM calls, getting "not a bug" bugs in the queue - but on the whole, I found that overseas QA accelerates the dev cycle and the rewards justify the risks.

Anonymous said...

Oh come on Joe ... being in your twenties is not quite a yardstick to judge an engineer with :P

Yeah, I didn't mean that to come off quite how it sounded.

What I meant is that, quite aside from the age issue, the BDC engineers are a lot more junior than those in Dulles, Mountain View, or Dublin.

At most AOL campuses, a development project is probably led by a principal SE or architect. The majority of developers will be at least Sr SE's.

In Bangalore, the tech lead is likely to be a Sr SE, and the majority of engineers will be SE's. That's a big experience difference.

Of course, no one's really asking me whether our use of BDC development is cost effective. That decision is made above my pay grade.

But it is part of my job to make integration work. And based on what I've seen (which amounts to seeing the results of three different BDC development teams), the secret is a whole lot of mentoring by senior folk in Dulles, California, or Dublin.

A BDC development team, like any AOL team building an application, needs someone senior to spend a lot of time on the project. Since there are few if any senior engineers in Bangalore, that means someone from another AOL site.

Further, the BDC teams have some special issues when it comes to integrating with other teams. That means that they need someone to serve as their technical ambassador to the other AOL groups - someone to track down dependencies and figure out how integration will work. That also means someone fairly senior from a different AOL site.

Finally, the BDC folk, like junior engineers anywhere, prefer to solve problems than to find out how others solve those same problems. They'd rather design a new solution than research an existing solution that can be reused. They haven't yet learned the advantages of laziness. They need someone to keep a close eye on what they are doing to make sure that they don't reinvent the wheel. And especially to make sure that the wheels that they do reinvent are compatible with AOL axles.

All of this means that a rather senior engineer has to spend a lot of time mentoring the BDC dev team. Probably more time than he would expect. And he must have the discipline to stick close to the team, in spite of the distance. (This last is one area where I've failed in the past, incidentally.)

And because of the distance, that becomes a lot more difficult.

I'll be glad when BDC has their own architects. That's when we'll really start to see major gains from using BDC teams.

Sree Kotay said...

Wow, some deep good comments all around. Nothing said I completely disagree with in any way.

I didn't really mention cost, because I think it certainly (obviously) part of the underlying motivation. I just wanted to pointed that (a)its not the only reasons, and (b)it IS a good reason.

My main point, which I don't think was lost, and certainly could be found in the echo of most of the comments, was that it is part of the task to maximize our investment in Engineering dollars, and that certainly includes leveraging (and developing, as Joe points out) talent wherever and however one can.

I have to admit, I don't buy the highly integrated thing - feels like a dodge. Its true for some tasks, at some levels, but there's a LOT of work to do, and we have many THOUSANDS of technology folk.

Certainly, as Rizwan pointed out, my post was really directed more at development than customer services tasks, but I do want to caution against snap judgements. Things like customer satisfaction, net promoter index, etc. ARE measurable objectively for those type of services even more so than typically developer productivity for product development is.

Should we NOT higher somebody in the US for a call center, then, if they have a funny accent?

Anonymous said...

To the point about language and customer service, I've had plenty of bad experiences with customer support provided by good old Americans here in the states, and great support from our friends offshore. It's really a factor of training and skill.

I know plenty of people who work (not for AOL) on phones in California. One of them is Indian, and often the first question she fields is, "Are you in Bangalore?" While she doesn't seem to mind it much, I find it incredibly disrespectful. After all, when we get someone with a European accent, we don't say, "Are you in Morocco?" Or, "Are you in Taiwan?" for the Asians. Or even more extreme, "Are you in South Africa?" The last of these callers would likely be slapped with a lawsuit, but I know that there was a day when my friend had to endure four calls in a row that started off with the location question.

It's a global economy and a global workforce. We need to go withere the skills are, and the skills are worldwide. It presents a number of challenges as to how you hire, manage, and work with those teams, as Joe and others point out. But these are questions we have to learn to answer, whether those colleagues are somewhere in America, Asia, Europe, or... who knows, maybe some offplanet remote site someday :)

Sree Kotay said...

Just to echo Edwin (as I often do :)) - if you treat outsourcing as just a cost exercise, you're not reaping the real opportunity.

Anonymous said...

Great article! Thanks.

Anonymous said...

Thanks for interesting article.

Anonymous said...

Excellent website. Good work. Very useful. I will bookmark!

Charmingirl said...

Wholesale lingerie directly from China?
As a famous brand and specialized manufacturer of sexy clothing in China. Charmingirl supply the international market with fashionable sexy lingerie and sexy costume since 2002. With advanced technology,all our products are of high quality. Now we have clients all around the world. Lingerie Wholesale and OEM are welcomed!
As a Lingerie Manufacturer, Charmingirl has standard workshop and production line, professional designers and experienced workers.
We do Wholesale Underwear,
Lingerie Wholesale, including corset and bustier,
Sexy Lingerie Wholesale, including bikini, underwear
Lingerie Wholesale, and Babydolls, Sexy Lingerie Wholesale, and
Sexy Lingerie Wholesale including sleepwear,clubwear.
Lingerie Wholesale from China: Lingerie China, you will find the
Leather Lingerie and PVC Lingerie, also you can buy
Christmas Costume and Xmas Lingerie
for your Christmas Lingerie Christmas day.
Our Wholesale center: Sexy Lingerie Wholesale can do Lingerie Wholesale online.
Halloween Costume,
also wholesale Adult Costume with fashion Babydoll Babydoll, and bra and panties Bra and Panties, Sexy Uniform Sexy Uniform is also our major products.