# Sunday, 21 August 2011

Dreamforce 11 is just around the corner and fellow Facebook Engineer Mike Fullmore and myself have been invited to speak at the following panel:

Enterprise Engineering
Friday, September 2
10:00 a.m. - 11:00 a.m.
Can you really develop at 5x a regular speed when you're at enterprise scale? In this session, a panel of enterprise technical engineers will discuss engineering best practices for the Sales Cloud, Service Cloud, Chatter and Force.com. Topics include security, sandbox, integration, Apex, and release management.

Speakers: Mike Leach, Facebook, Inc.; David Swidan, Seagate Technology LLC; Mike Fullmore, Facebook, Inc.

In case you're not able to attend, here are the high level points from our presentation. :-)

Moving Fast on Force.com

Facebook has been using Salesforce for several months to rapidly prototype, build, and deploy a number of line of business applications to meet the needs of a hyper-growth organization. This presentation shares some best practices that have evolved at Facebook to help develop on the Force.com platform.

People

Before sharing details about Facebook's processes, methodologies, and tools; it's important to point out that the people on the enterprise engineering team are what really make things happen. Each Engineer is able to work autonomously and carry a project through from design to deployment. Great people instinctively take great pride in their work and consistently take the initiative to deliver awesomeness. I would be remiss not to point them out here. All these Engineers operate at MVP levels.
The effort that goes into recruiting a great development team should not be underestimated. Recruiting an awesome team involves several people doing hundreds of phone screens and dozens of interviews. Facebook is in a unique situation in its history and we don't take it for granted that we have access to unprecedented resources and talent. It's actually very humbling to work with such a stellar team at such a great company.

("yes" we're still hiring)

Business Processes

Projects and applications generally fall into one of 9 major process buckets. Engineers at Facebook seeking to have a high impact will typically either have a breadth or depth of knowledge. Some focus on the long-term intricate details and workflows of a single business process while others are able to move around and generally lead several, concurrent, short-term development efforts in any business area.

Sandbox->Staging->Deploy

Each Project has it's own development sandbox. Additionally, each Engineer may also have their own personal sandbox. When code is ready to be deployed, it's packaged using the Ant migration tool format and typically tested in 2 sandboxes: 1 daily refreshed staging org to ensure all unit tests will run and there are no metadata conflicts, and a full sandbox deploy to give business managers an opportunity to test using real-world data.

Change sets are rarely used, but may be the best option for first time deployments of large applications that have too many metadata dependencies to reliably be identified by hand.

The online security scanner is used as a resource during deployment to identify any potential security issues. A spreadsheet is used for time-series analysis of scanner results to understand code quality trends.

Once a package has been reviewed, tested, and approved for deployment; a release Engineer deploys the package to production using Ant. This entire process is designed to support daily deployments. There are typically 3-5 incremental change deployments per week.

Obligatory Chaotic Process Diagram

"Agile" and "process" are 2 words that are not very complimentary. Agile teams must find an equilibrium of moving fast yet maintaining high quality code. Facebook trusts every Engineer will make the right decisions when pushing changes. When things go wrong, we conduct a post-mortem or retrospective against an "ideal" process to identify what trade-offs were made, why, and where improvements can be made.

All Engineers go through a 6 week orientation "bootcamp" to understand the various processes.

Typical Scrum Development Process

The development "lingua franca" within Silicon Valley, and for most Salesforce service providers, tends to be Scrum. Consultants and contractors provide statements of work and deliver progress reports around "sprints". Scrum training is readily available by a number of agile shops.

This industry standard has been adopted internally and keeps multiple projects and people in sync. Mike Fullmore developed a Force.com app named "Scrumbook" for cataloguing projects, sprints, and stories.


A basic Force.com project template with key milestones has been created to give Project Managers an idea of when certain activities should take place. Whenever possible we prefer to avoid a "waterfall" or "big bang" mentality; preferring to launch with minimal functionality, validate assumptions with end-users, then build on the app in subsequent sprints.


Manage The Meta(data)

The general line of demarcation within IT at Facebook is:
  • Admins own the data
  • Engineers own the metadata
The Salesforce Metadata API is a tremendously powerful resource for scaling an enterprise, yet remaining highly leveraged and lean. We've developed custom metadata tools to help us conduct security audits and compare snapshot changes.


(Credit to our Summer Intern, Austin Wang, who initiated the development of internal tools! )

Change Management

The advantage to using Salesforce is the ability to use declarative configuration and development techniques to produce functional applications, then use more powerful Apex and Visualforce tools to maximize apps around business core competencies. "Clicks over code" is a common mantra in Salesforce shops, and Facebook is no exception.

A change management matrix is a useful tool for determining when "clicks-over-code" is preferred over a more rigorous change management process.