After Mike Otey and I spoke with the MySpace team about the explosive growth of their implementation of SQL Server, we followed up with Ted Kummert, senior vice president of the business and platform division at Microsoft, to get Microsoft’s take on this partnership. To learn more about how the MySpace team stayed on top of their burgeoning customer base and SQL Server load, see the January 2010 SQL Server Magazine cover story, “MySpace Answers the Question: Why SQL Server?” InstantDoc ID 103082.

SQL Mag: Thanks for taking the time to talk with us. The MySpace team has quite a story to tell. They say that from a transaction and data volume standpoint they may be the largest implementation of SQL Server.

Ted Kummert: This is certainly one of the largest implementations on SQL Server. At a petabyte of data storage and in terms of instance count MySpace is one of the largest public-facing Internet sites in the world. It’s exciting! It’s a great company and a great example of a very high-scale, leading-edge mission-critical application.

Increasingly there are more companies with implementations like this. It’s been one of our points of emphasis, as we come from SQL Server 2005 to 2008, to invest in the tools to allow the platform to scale to the highest levels. Increasingly we’re seeing this type of solution on our platform. Historically you had to go to IBM or Oracle, and now we’ve got the full application platform that can scale to any type of solution. We think it’s great for customers because you no longer have to go to those other guys and pay them lots of money.

SQL Mag: The MySpace team loves Service Broker for handling asynchronous rights and managing transactions across databases. We’d like to know if you agree with what they say about their implementation: “MySpace is handling one of the most extensive Service Broker implementations in SQL Server right now. You'd have to verify that with the Microsoft folks, but I'm pretty confident that we're probably one of their biggest customers as far as Service Broker is concerned.”

Ted Kummert: That’s absolutely true. They’ve built a very large distributed data service. It’s 400-plus instances of SQL Server. The interesting part of the story is that while they were growing so dramatically they ran into some transactional integrity issues across that distributed data service. They were on their way to building their own infrastructure top to bottom to provide for reliable messaging across the data service. But with SQL Server Service Broker they saw something that could give them the underlying infrastructure to do that. It’s been interesting to watch how Service Broker became the core platform service within their environment and how it’s enabled them to add new features more rapidly. It’s given them transactional integrity across the distributed data service. And, based on the way they’ve implemented it, it’s given them a lot of agility in innovating around it and moving quickly. One of the challenges when you scale to this high level is: How do you extend your implementation while still maintaining the reliability and the scalability of the core system that you’re extending? We think that MySpace has done great work in this architecture in designing their Service Dispatcher so they have something that enables them to do that.

SQL Mag: We understand from the MySpace team that Service Dispatcher was certified in Microsoft labs.

Ted Kummert: We’ve had a long partnership with MySpace. It goes all the way back to prior to SQL Server 2005. We partner with them as they think about adopting new features. I’m told that today they’re onsite working with us on Windows Server 2008 R2 and SQL Server 2008. They’re also part of a select group of customers that we work with very closely to consult with on a more long-term basis about where we’re headed. What’s in the future? What are the things that are important to them? And we give them an early view into where we’re headed so they’re deeply involved. When you talk about a company that has a mission-critical application like this, this level of partnership is something that is very important. We work hard to engage with them and with customers like that at all levels.

SQL Mag: Will MySpace extensions like Service Dispatcher or Tier Hopper eventually find their way into SQL Server?

Ted Kummert: Well, our approach is to build the enabling platform and help our customers build these innovations. It’s a natural thing as we work with customers like MySpace for them to suggest features that would make those types of implementations easier. Our objective is to provide those broadly enabling features. The platform capability to make those types of services easier and easier to implement is on our roadmap, and we’re not saying that we explicitly have a plan to incorporate those capabilities end-to-end.

SQL Mag: Seems like there’s a lot of possibilities in this space for vendors.

Ted Kummert: Our approach is to enable an ecosystem to innovate around a core platform. We can’t and won’t get to solving every problem. That’s why we take an approach of building a platform that’s broadly applicable with the type of tools you need in order to build solutions. That gives a lot of partners the ability to innovate and add value in specific ways that we can’t get to in that release. The explicit part of our approach is to build the enabling platform.

SQL Mag: The MySpace team mentioned that a key reason for choosing SQL Server was that SQL provides a rapid development environment that compares favorably with open source competitors.

Ted Kummert: That’s right. We consistently hear from customers that, in terms of time to solution, developer productivity is a pretty big component of the expense in these custom application projects. Visual Studio and .NET, and the flexibility of SQL Server, give them a broad set of tools oriented around getting developers more productive to solve those types of problems.

SQL Mag: One of the things that the MySpace people mentioned was that they received some significant help from the SQLCat team. Could you talk about the role of the SQLCat team?

Ted Kummert: We have a team inside my group that has existed for years and years within the SQL Server organization that’s oriented around working with customers at the leading edge on their implementations. That team is the SQLCat team—the customer advisory team. We’ve had a partnership with MySpace at that level and that partnership continues. With a customer like MySpace, having a relationship at multiple levels is important. On a day to day level I don’t have a lot of interaction with the MySpace team, but certainly they know that I’m there should there an issue or a need arise.

SQL Mag: The MySpace team told us about their growing pains. They started small, under a million customers, and grew and grew. They started on the Standard edition and then had to move to the Enterprise edition to handle their clustering issues.

Ted Kummert: The numbers in this application are just mind-boggling. Petabytes of data. They’re not quite to a trillion rows, but there are some 800 billion rows of data in this system. I think there’s something in the neighborhood of a thousand databases on over 440 instances of SQL Server. Per day 300,000 new users. Those numbers are incredible. This growth trajectory has been from the beginning with MySpace. They’ve been continuously innovating to meet those demands. It’s been an exciting thing to watch and partner with them on.

SQL Mag: We were blown away by these numbers as well. Are there other companies that you’re working with that are experiencing this sort of growth?

Ted Kummert: Well certainly if you look across the workloads of SQL Server, from data warehousing to OLTP we see all ranges of applications on the platform today. We see data warehouses that are growing at significant rates--tens of terabytes, substantial end-user concurrency and growth rates. We have other examples of very high transaction throughput OLTP systems. There are many examples. MySpace is one of the largest publically facing sites in the world, so it’s unique in those characteristics and dimensions. And there’s other things—financial services, health care, retail, that have their own uniqueness. But we do see, generally across the board, mission-critical applications in every workload and we do see those growing and customers addressing those growth challenges.

SQL Mag: We’re watching with interest as Microsoft works out problems with large customers and sets a roadmap for SQL Server Enterprise going forward.

Ted Kummert: Certainly if you look at our roadmap, you see one of the largest areas of investment in SQL Server 2008 was in data warehouse scale. A lot of the features apply to scalability across the whole platform—scaling storage, compression, scale in terms of performance.

SQL Mag: What kind of features are in SQL Server 2008 that can help customers like MySpace manage multiple servers?

Ted Kummert: In 2008 one of the biggest areas of investment was in an infrastructure called policy-based management. This enables customers to manage more by policy and to detect and enforce things at a policy level across their environment. That makes it easier to manage a multi-server environment. In SQL Server 2008 R2 we continue our investments in the manageability area. The multi-server management features give customers who manage SQL Server environments with lots of servers a dashboard to give them a view of all their assets and the visibility they need to efficiently manage larger and larger farms. This is one of our biggest areas of investment.

We’ve got a pretty exciting year ahead of us!