# Sunday, 27 May 2012

The announcement that Facebook intended to acquire Instagram made headlines. While the focus of these headlines primarily centered around the $1B valuation and 50 Million user audience, the true story went largely untold.

Instagram's story marks an inflection point in computing history whereby a start-up embracing cloud infrastructure is no longer the exception, but now considered the rule for building a successful company.

Consider the fundamentals of their meteoric rise over 2 years:
  • 2 people start with an idea
  • Create an iPhone application with hosting on Amazon Web Services (AWS)
  • Distribute the app on Apple's app store
  • Grow an audience
  • Evolve and enhance the product
  • Leverage elastic scalability on AWS
The costs for a v1 product were extraordinarily low and gave them an opportunity to pivot at any point without being grounded by capitalized infrastructure investments (i.e. building their own colo servers and doing their own hosting).

At the time of Instagram's acquisition they had added only 11 more employees, mostly hired in the few months leading up to their acquisition.

Some tips and lessons learned for other start-ups considering embracing the cloud as their development and hosting platform:

Choose a platform and master it: Instagram selected Amazon Web Services, but there are many options available. Don't hedge your bets by designing for platform portability. Dig deep into the capabilities of your chosen platform and exploit them.

Think Ephemeral: Your cloud web servers and storage can disappear at any point. Anticipate and design for this fact. If you have a client server background, then take a step back and grasp the concepts of ephemeral storage and computing before applying old world concepts to the cloud.

Share Knowledge: This is still a new frontier and we're all constantly learning new tips and tricks about how to best utilize cloud computing. Many people at Facebook (including myself) were fans of the Instagram Engineering blog long before the acquisition. Share what you've learned and others will reciprocate.

Build for the long term: The Instagram team did not build a company to be sold. They built a company that could have continued to grow indefinitely, and perhaps even overtaken competing services. There are so many legacy business processes and consumer applications whose growth is restricted by their architecture. Embrace the cloud with the intent to build something enduring and everlasting.

Sunday, 27 May 2012 10:26:04 (Pacific Daylight Time, UTC-07:00)
# Sunday, 30 October 2011

Integrating CRM with ERP/Financial systems can be a challenge. Particularly if the systems are from 2 different vendors, which is often the case when using Salesforce.com CRM.

At Facebook, we've gone through several iterations of integrating Salesforce with Oracle Financials and the team has arrived at a fairly stable and reliable integration process (kudos to Kumar, Suresh, Gopal, Trevor, and Sunil for making this all work).

Here is the basic flow (see diagram below):

1) The point at which Salesforce CRM needs to pass information to Oracle is typically once an Opportunity has been closed/won and an order or contract has been signed.

2) Salesforce is configured to send an outbound message containing the Opportunity ID to an enterprise service bus (ESB) that is configured to listen for specific SOAP messages from Salesforce.

3) The ESB accepts the outbound message (now technically an inbound message on the receiver side) and asserts any needed security policies, such as whitelist trusting the source of the message.

4) This is the interesting part. Because the Salesforce outbound message wizard only allows for the exporting of fields on a single object, the ESB must call back to retrieve additional information about the order; such as the Opportunity line items, Account, and Contacts associated with the Order.

In Enterprise Application Integration (EAI) speak, this is referred to as a Content Enrichment pattern.

5) An apex web service on Salesforce receives the enrichment request, queries all the additional order details, and returns a canonical XML message back to the ESB.

6) The ESB receives the enriched message and begins processing validation and de-duplication rules, then transforms the message into an object that can be consumed by Oracle.

7) The ESB then inserts the Order into Oracle.

8) The Oracle apps API inserts/updates the various physical tables for the order and throws any exceptions.

Sunday, 30 October 2011 17:15:23 (Pacific Standard Time, UTC-08:00)
# Sunday, 21 August 2011

Dreamforce 11 is just around the corner and fellow Facebook Engineer Mike Fullmore and myself have been invited to speak at the following panel:

Enterprise Engineering
Friday, September 2
10:00 a.m. - 11:00 a.m.
Can you really develop at 5x a regular speed when you're at enterprise scale? In this session, a panel of enterprise technical engineers will discuss engineering best practices for the Sales Cloud, Service Cloud, Chatter and Force.com. Topics include security, sandbox, integration, Apex, and release management.

Speakers: Mike Leach, Facebook, Inc.; David Swidan, Seagate Technology LLC; Mike Fullmore, Facebook, Inc.

In case you're not able to attend, here are the high level points from our presentation. :-)

Moving Fast on Force.com

Facebook has been using Salesforce for several months to rapidly prototype, build, and deploy a number of line of business applications to meet the needs of a hyper-growth organization. This presentation shares some best practices that have evolved at Facebook to help develop on the Force.com platform.


Before sharing details about Facebook's processes, methodologies, and tools; it's important to point out that the people on the enterprise engineering team are what really make things happen. Each Engineer is able to work autonomously and carry a project through from design to deployment. Great people instinctively take great pride in their work and consistently take the initiative to deliver awesomeness. I would be remiss not to point them out here. All these Engineers operate at MVP levels.
The effort that goes into recruiting a great development team should not be underestimated. Recruiting an awesome team involves several people doing hundreds of phone screens and dozens of interviews. Facebook is in a unique situation in its history and we don't take it for granted that we have access to unprecedented resources and talent. It's actually very humbling to work with such a stellar team at such a great company.

("yes" we're still hiring)

Business Processes

Projects and applications generally fall into one of 9 major process buckets. Engineers at Facebook seeking to have a high impact will typically either have a breadth or depth of knowledge. Some focus on the long-term intricate details and workflows of a single business process while others are able to move around and generally lead several, concurrent, short-term development efforts in any business area.


Each Project has it's own development sandbox. Additionally, each Engineer may also have their own personal sandbox. When code is ready to be deployed, it's packaged using the Ant migration tool format and typically tested in 2 sandboxes: 1 daily refreshed staging org to ensure all unit tests will run and there are no metadata conflicts, and a full sandbox deploy to give business managers an opportunity to test using real-world data.

Change sets are rarely used, but may be the best option for first time deployments of large applications that have too many metadata dependencies to reliably be identified by hand.

The online security scanner is used as a resource during deployment to identify any potential security issues. A spreadsheet is used for time-series analysis of scanner results to understand code quality trends.

Once a package has been reviewed, tested, and approved for deployment; a release Engineer deploys the package to production using Ant. This entire process is designed to support daily deployments. There are typically 3-5 incremental change deployments per week.

Obligatory Chaotic Process Diagram

"Agile" and "process" are 2 words that are not very complimentary. Agile teams must find an equilibrium of moving fast yet maintaining high quality code. Facebook trusts every Engineer will make the right decisions when pushing changes. When things go wrong, we conduct a post-mortem or retrospective against an "ideal" process to identify what trade-offs were made, why, and where improvements can be made.

All Engineers go through a 6 week orientation "bootcamp" to understand the various processes.

Typical Scrum Development Process

The development "lingua franca" within Silicon Valley, and for most Salesforce service providers, tends to be Scrum. Consultants and contractors provide statements of work and deliver progress reports around "sprints". Scrum training is readily available by a number of agile shops.

This industry standard has been adopted internally and keeps multiple projects and people in sync. Mike Fullmore developed a Force.com app named "Scrumbook" for cataloguing projects, sprints, and stories.

A basic Force.com project template with key milestones has been created to give Project Managers an idea of when certain activities should take place. Whenever possible we prefer to avoid a "waterfall" or "big bang" mentality; preferring to launch with minimal functionality, validate assumptions with end-users, then build on the app in subsequent sprints.

Manage The Meta(data)

The general line of demarcation within IT at Facebook is:
  • Admins own the data
  • Engineers own the metadata
The Salesforce Metadata API is a tremendously powerful resource for scaling an enterprise, yet remaining highly leveraged and lean. We've developed custom metadata tools to help us conduct security audits and compare snapshot changes.

(Credit to our Summer Intern, Austin Wang, who initiated the development of internal tools! )

Change Management

The advantage to using Salesforce is the ability to use declarative configuration and development techniques to produce functional applications, then use more powerful Apex and Visualforce tools to maximize apps around business core competencies. "Clicks over code" is a common mantra in Salesforce shops, and Facebook is no exception.

A change management matrix is a useful tool for determining when "clicks-over-code" is preferred over a more rigorous change management process.

Sunday, 21 August 2011 11:52:04 (Pacific Daylight Time, UTC-07:00)
# Sunday, 22 May 2011
DocuSign's Hackathon on May 15th and 16th, 2011 brought out some serious talent to DocuSign's new office in downtown San Francisco. For 2 days, developers took a shot at the $25K purse in 3 categories; consumer, enterprise, and mobile.

My app, SocialSign, was based on the T-Mobile Fave 5 campaign:
  • Select 5 Facebook friends
  • Sign an unlimited voice/text contract through DocuSign
  • Merchants manage the contract in Salesforce.
I admittedly spent about 75% of the hackathon working with the Facebook app canvas and Salesforce Sites APIs, which probably explains why SocialSign placed as runner-up in the consumer category (other developers got far more creative with the actual Docusign API).

In the end, it was a great opportunity to demonstrate the feasibility of conducting social commerce through Facebook using Salesforce CRM to manage contacts, orders, and contracts. Thank you DocuSign for hosting such a great event!

(Video demo of SocialSign available here and embedded below)

Sunday, 22 May 2011 20:24:38 (Pacific Daylight Time, UTC-07:00)
# Tuesday, 26 October 2010

Career__c career = [SELECT Id, Name FROM Career__c WHERE Culture__c='Cool' AND Location__c='Palo Alto, CA' AND Description__c LIKE 'Salesforce%' AND PerkId__c IN (select Id from Perk__c) LIMIT 1];
system.assertEquals('Software Application Developer', career.Name);

Join an incredible team that is shaping the future of Facebook with the development of enterprise apps in the cloud.

Apply for this position

Force.com Application Developer
Palo Alto, CA

Facebook is seeking an experienced application developer to join the IT team and participate in the development, integration and maintenance of force.com applications supporting Facebook data centers consigned inventory asset tracking. This is a full-time position based in our main office in Palo Alto.

•    Technical design, configuration, development and testing of Force.com custom applications, interfaces and reports;
•    Model, analyze and develop or extend persistent database structures which are non-intrusive to the base application code and which effectively and efficiently implement business requirements;
•    Integrate force.com applications to other facebook external or internal Business Applications and tools.
•    Develop UI and ACL tailored to facebook employees and suppliers.
•    Apply sound release management and configuration management principles to ensure the stability of production environments;
•    Participate in all phases of software development/implementation life cycle including functional analysis, development of technical requirements, prototyping, coding, testing, deployment and support;
•    Participate in peer design and code review and analyze and troubleshoot issues in a complex applications environment, which include Salesforce.com, force.com, Oracle E-Business Application Suite R12 and custom built lamp stack based tools and systems. 
•    Research and understand force.com capabilities to recommend best design/implementation approaches and meet business requirements.
•    Plan and implement data migration and conversion activities;
•    Provide daily and 24x7 on-call support

•    Bachelor's in Computer Science, any engineering field, or closely related field, or foreign equivalent;
•    Passionate about Salesforce and building apps on the force.com platform
•    At least 6 years of design, configuration, customization and development experience with Salesforce in general and Force.com
•    Strong development background with APEX and VisualForce.
•    Strong knowledge of Salesforce/Force.com API and toolkits for integration.
•    Strong understanding of RDBMS concepts and programming using SQL and SOQL;
•    Background in database modeling and performance tuning;
•    Knowledge of best practices around securing and auditing custom applications on Force.com
•    Background in design and development of web based solutions using Java, JavaScript, Ajax, SOAP, XML and web programming in general.
•    Strong experience with business process and workflow and translating them into system requirements.

Tuesday, 26 October 2010 08:15:01 (Pacific Daylight Time, UTC-07:00)
# Saturday, 23 October 2010

Short Version: The rumors are true. I accepted a position at Facebook with a focus on Salesforce.

(That's about it. Read the long version for more details)

Long Version:
I started working at Facebook in August with a focus on designing and developing apps on Force.com within the IT group. The group is very focused on embracing cloud-based solutions to maintain agility and efficiency during a hyper-growth phase.

My wife and I moved from Portland, OR to San Carlos, CA as part of the transition and simply love it. We've actually been considering a move to Silicon Valley for 2 years for the weather, family, and, well... this is the center of cloud computing. The Facebook position gave us the perfect opportunity to make the jump.

Why Facebook? The 3 primary motivations for working at Facebook are the cloud, a progressive IT group, and company culture.

The Cloud
Facebook's infrastructure is one of the most under appreciated cloud architectures in existence today.

When you consider that Salesforce can support 1.5M customers on 1,000 servers (a 1,500:1 ratio) you understand why we are at an important inflection point in IT infrastructure and must consider cloud-based solution providers at every opportunity. Multi-tenancy is just much more efficient.

Then when I look internally at Facebook supporting 300M users on 30,000 servers (a 10,000:1 ratio) the efficiencies of scalable cloud-based architecture become even more obvious and impressive. (I'd love to share how we accomplish this feat, but you'll just have to apply here first :-). It truly is amazing and gets more efficient with each release)

Facebook as an enterprise class cloud provider will become even more apparent within 10 years as the user base grows and takes their social graph with them into every facet of their life (phones, devices, automobiles). When Mark Zuckerberg coined the term "social utility" back in 2007, the average response was "Huh?... It's just a website where I communicate with my friends".

But as the social graph gets consumed via web service APIs and millions of people communicate through the service, the role of Facebook (and the reasoning behind Mark's comment) becomes much more like an AT&T for the web rather than just a website.

The focus and dedication to cloud-computing is so serious, that we will build all our own datacenters from the ground up going forward, starting first in Prineville, OR.

Progressive Information Technology
I work in the IT group at Facebook in a ground floor opportunity at a rapidly growing company. We have an opportunity to learn from past enterprise IT architectures and challenge the norms using cloud computing and open source. The infrastructure put in place today will very likely still exist in 10+ years.

Unlike traditional IT roles, Facebook IT is responsible for the core product; our datacenters, servers, and technical operations. Our group is aligned very closely with the product Software Engineers.

Every IT asset; from CRM, ERP, Financials, HRM, SCM, ESB,... and a dozen other TLAs, are in it's first or second generation deployment. There are "build or buy" decisions being made constantly. My role often times involves demonstrating how to leverage what we've already "bought" (using Force.com for a variety of specific solutions).

Facebook takes a progressive stance on utilizing Salesforce as a platform and I've additionally been challenged with projects and opportunities that leverage my strengths outside just Salesforce development.

I'm probably the youngest 40 year old most people will ever meet still enjoy the occasional hackathon, drinking lots of coffee and embracing the most progressive idea or concept emerging at the time in software development.

There is no other company that I can think of than Facebook that is the epitome of this type of culture. There is a kindred spirit amongst all Engineers at Facebook that is hard to explain. I've participated in 2 hackathons so far and will never miss future events.

Most people don't perceive Facebook as a technology company, but the internal esprit de corps is very focused on building cool stuff, simply for the enjoyment of building stuff. Facebook is a technology company.

Side Projects
Maybe I'm being over ambitious to think I'll have any spare time to work on side projects, but here's a status of projects I continue to support and take great interest in on the side.

Passage .NET Portal Framework: This platform is still OEM'd by a couple partners who do the heavy lifting. The XOS framework has really proven that object oriented databases can be scalable and flexible. A couple more partners/resellers are in the process of developing Passage based solutions, so I don't think the end is in sight.

i-Dialogue: Other partners and associates have taken over the day to day hosting and support of i-Dialogue portals and sites. I still respond to any dlog related questions within 48 hours. Another developer has taken over all custom development, but I still enjoy occasionally optimizing the framework and have committed to participating in quarterly reviews/updates/patches.

Cool Sites: Similar to i-Dialogue, this native AppExchange app is primarily driven by partners. Cool Sites was developed to give my creative and web development friends (who rely primarily on HTML/CSS/JS skills) access to the rapidly growing Salesforce customer base.

There is a backlog of Cool Sites plugin requests that involve porting existing i-Dialogue components to Visualforce (Event calendar, knowledge base, shopping carts, partner finder, maps, etc...). Word on the street is that Salesforce acquired a CMS company and is preparing a similar offering for Dreamforce 2010, so I'm in "wait and see" mode re: the future of this project.

Cool Trends: I started this Salesforce native app after working with the Google Maps API on the Azure proof of concept app last Winter. It's a very simple data warehouse (only 2 custom objects) with time-series analysis charts for reporting day-over-day trends.

If time allows, I'll try to submit this to the AppExchange before Dreamforce 2010.

Cool Mesh: After years of designing and developing single-tenant enterprise web applications, the entrepreneur in me began to pursue the "next big thing". Cool Mesh is a massively scalable, multi-tenant, open-source, clustered computing project loosely based on the following ingredients:
  • Amazon EC2
  • Unix
  • Apache
  • Node.js
  • CouchDB / Cassandra
  • Immutable storage
Don't ask me what I'd do with a working prototype. Probably just re-invent business models I've worked on in the past :-)

Chatter Bot: On the heels of receiving an award for Chatter Bot, I started delving into "The Internet of Things" and envisioning what Cloud 3.0 might look like. An idea emerged, based on Cool Mesh, that continues to intrigue me. Still dabbling.

Music in Schools (non-profit):  I still support or volunteer for 3-5 non-profits on Salesforce and really believe in their vision. Immersing myself in a music education non-profit is where I'd like to start devoting my time next. I'm new to the bay area, so there's some research to do. Who knows; if I don't find a perfect NPO, then I'm open to launching one myself.

I'll be wandering the halls of Dreamforce 2010. Don't be a stranger. If you read this blog, then I hope we'll meet at the Tweetup :-)

Saturday, 23 October 2010 13:53:24 (Pacific Daylight Time, UTC-07:00)
# Saturday, 10 July 2010

Facebook has nearly 500 million users (at the time of this writing) and are poised to transform how customer relationships are acquired, cultivated, and supported.

Consider the evolution of Contact management over the past 50 years:
  • Paper: Write names, addresses, birthdays, and other notes down on paper
  • Rolodex: Everyone trades business cards and keeps a local paper copy
  • PC Software: Rolodex moved to PC (Spreadsheets, Goldmine, Act!)
  • Client / Server Software: Many people in same office share common contact database (Siebel, MS CRM)
  • Software as a Service: Contacts moved to Internet hosted server. Accessible from anywhere (Salesforce.com)
  • Crowd sourced: 3rd parties pooling contact information to improve data quality, keep lists up to date (Plaxo, Jigsaw)
  • Social Networking: Contact maintains singular identity. Always up to date. Control of privacy and disclosure (LinkedIn, Facebook)
The consumer now has more power than ever. Consumers now generate more information about themselves than can be generated by 3rd parties.

Buying a list of leads that may be interested in buying camping gear will have no where near the effectiveness of publishing an ad on Facebook targeted at consumers with a self-identified interest in camping (See The Role of Advertising at Facebook).

Consumers will increasingly defer to their friends recommendations on which restaurants to visit, which shoes to buy, and which cars to drive.

CRM systems no longer contain the master records for Contact information.

Businesses must evolve past collecting and cleansing Contacts and instead collect meta-information that refers to customer-managed online profiles, such as Facebook and LinkedIn, for these resources are now truly authoritative. When there is a conflict between CRM Contact information and an online social network profile, the social profile will be master record.

Websites must evolve to become applications. Web forms soliciting contact information will become a thing of the past. Consumers will not have the patience to do anything more than a single click to identify interest in a product or service.

The video embedded in this blog post (and available here) demonstrates a Cool Sites application integrated with Facebook Connect and Salesforce CRM.
Saturday, 10 July 2010 16:31:06 (Pacific Daylight Time, UTC-07:00)