Project Dallas

In December of 2019, I found myself celebrating the release of SQL Server 2019 and catching up with all my colleagues on the SQL Server team in building 43 on the Microsoft campus in Redmond. It was a heady time for SQL Server and me. I had just traveled some 50,000 miles over the last calendar year getting the message out on SQL Server 2019. I had launched my second book, SQL Server 2019 Revealed (https://aka.ms/sql2019book), telling the entire story of our latest SQL Server release. I felt proud of the release and was able to tell all the great customer feedback and stories to many in the engineering team. On that trip I spoke to a large crowd of all the engineers who worked on the release in an internal meeting organized by the famous Slava Oks, Distinguished Engineer at Microsoft and Lead Development Manager for the SQL Server 2019 release.

I spent some of my time in Redmond catching up with my longtime friend from CSS, Robert Dorr, who now worked for Slava. I spent time at lunch telling Bob all the tales from my travels and my experiences launching the product and authoring the book. SQL Server 2019 was tagged project Seattle by Slava, Travis Wright, and Tobias Ternstrom. I told Bob at lunch, “Hey, I think you and I should come up with the project name for the next version of SQL Server.” Bob started a year after me, so together we represented 55+ years of experience with SQL Server. That day Bob (as he often does) was sporting a Dallas Cowboys jersey. I proposed the project name should be called Dallas. After lunch I stopped by Slava’s desk and pitched the idea. He told me, “It makes sense. This new version should be a tribute to the two ‘Bobs’ who have worked so hard for SQL Server over the years.” Slava sent out an email the next day to our teams, and thus the project name Dallas was born. Sometimes that is how project names are created at Microsoft.

Even though we had just shipped SQL Server 2019, discussions had already begun about what type of new functionality could be part of project Dallas. For me, my focus would change drastically. On this same trip, I had meetings with my manager, Asad Khan, Vice President of Program Management (PM) across all of Azure SQL and SQL Server, which would lead me to start working on Azure SQL. But I still had my ear to the ground on project Dallas.

Already in late December of 2019, my colleague Amit Banerjee was putting together a proposal for the project. As we turned the corner to start 2020, little did I or anyone know all of us would have to adapt to a worldwide pandemic. In all honesty, the pandemic accelerated work for our teams in the cloud, which led to a slowdown in planning for project Dallas. The good news is that based on our engineering systems, much of the work we design and build in the cloud just naturally applies to SQL Server; Intelligent Query Processing (IQP) and SQL Ledger are great examples.

The focus on the cloud worked out very well for me personally, as I was already on the path to work on Azure SQL. I didn’t realize that the pandemic would speed up that work in the form of a partnership with Anna Hoffman for the Azure SQL Workshop (https://aka.ms/azuresqlworkshop), which then led to the popular Azure SQL Fundamentals learning path (https://aka.ms/azuresqlfundamentals) and a huge series of videos called Azure SQL for Beginners (https://aka.ms/azuresql4beginners). And since we had moved to all digital for our content, why not a third book? Thus was born Azure SQL Revealed (https://aka.ms/azuresqlbook), which launched in the fall of 2020.

But what about project Dallas?

Throughout calendar year 2020, project Dallas work had started under the leadership of Slava on the development side and Amit for program management. One example is degree of parallelism (DOP) feedback under a project called Gaia. Gaia had big ambitions and is an endeavor we are still working on, but the DOP feedback concept was born out of it, and you will hear more about this in Chapter 5 of the book. Other capabilities like Parameter-Sensitive Plan (PSP) optimization, cross-platform backups, buffer pool scan, Query Store hints, multi-write replication, SQL Ledger, T-SQL language improvements, and more were all under way. We were actively working through improvements for project Dallas but had not really landed on the full plan for a release. Again, the cool thing about how we do things now is that many of these features were designed for Azure SQL, but we knew because of our engine compatibility they could be fit into Dallas at any time.

As we moved further along calendar year 2020, it became clear to me that project Dallas would not make it into an actual release until calendar year 2022 at the earliest. I had meetings with Slava, Amit, Asad, and Travis Wright late in 2020 to discuss the plan. All of us agreed, along with our Vice President of Azure Data, Rohan Kumar, that we needed a new major version of SQL Server sooner rather than later. But we also agreed that trying to do anything sooner than 2022 would not result in a great release. In late calendar year 2020, I would hear for the first time the words SQL Server 2022.

Becoming SQL Server 2022

When this book is released, I will have been at Microsoft 29 years, all working on SQL. Throughout these years I’ve seen many changes, reorganizations, and transitions, more than I can even remember. But two changes happened early in calendar year 2021 that surprised me and gave me concern for project Dallas. Slava Oks left the SQL team to work in Azure (note: Slava would eventually leave Microsoft in the fall of 2021), and Amit Banerjee left Microsoft. In addition, Travis Wright pivoted from a role in the program management leadership team to work as a development manager for Azure Arc. Losing someone like Slava was pretty hard for me personally. I have worked with Slava in SQL for 20 years. All the journeys we had together (Slava is the one who taught me debugging in case anyone ever wanted to know that small fact; I thought I knew what debugging was until I met Slava) from SQL Server 2005 to SQL Server on Linux and of course SQL Server 2019. I also would deeply miss Amit. I actually mentored Amit when he first joined Microsoft and then enjoyed all those years as colleagues in engineering. I was really looking forward to partnering with Amit to release Dallas. But I knew the opportunity he was moving to was important for him and his family. I was happy and excited for these folks, but as you can imagine, I quickly was in a few meetings with Asad Khan on how to handle the situation and ensure Dallas moved forward.

With a goal for a Private Preview in the fall of 2021 (we made the decision to announce this at the Microsoft Ignite virtual event, which we knew at this time was in early November 2021), we had to get more organized than ever and start cranking out some builds. Joe assigned Pedro to be our de facto release program manager holding the reins to ensure everyone was meeting release criteria and getting features lined up for an announcement.

Our marketing team assigned Matthew Burrows to be the lead marketing manager for SQL Server 2022. Matt quickly started building weekly virtual team meetings to talk about SQL vNext (we always call a version vNext until we 100% agree on the exact name) including myself, Joe Sack, Pedro Lopes, James Rowland-Jones, and Sonya Waitman (Sonya works with Matt in marketing and was so valuable throughout the entire release). We would soon add another party to this band, Kendal Van Dyke. Joe hired Kendal to run our Early Adoption Program (EAP) since our first announcement would be a Private Preview program, which required customers to register with Microsoft to test and try out early builds. Kendal was a longtime veteran of Microsoft as a customer engineer and a well-respected person in the community and within Microsoft. I quickly met with Pedro and Joe and asked how I could best help us march toward Private Preview. They both responded very quickly, “Build us a deck!”

So in the late spring of 2021, I started “doing my thing” to build an “NDA” (for anyone with a nondisclosure agreement with Microsoft) deck we could use. The very first attempt for an overview of SQL Server vNext looked like the following Figure 1-1.

An illustration of the features of S Q L server v Next. It includes cloud connected, intelligent database, security, data platform, and analytics at scale.

You can see in Figure 1-1 there are not really any specific features listed, just themes. Plus, you will notice at the time we were still debating the future of Big Data Clusters (BDC). Project Austin, named by James Rowland-Jones, was an early proposal to reinvent BDC with serverless SQL and Spark.

I knew all the planned features for Dallas, but at this time I didn’t have a 100% commitment that we could talk about all of them yet. One thing I knew is that these “pillars” represented the major categories for features in the release.

To build a foundation for a true public deck on SQL Server vNext, we first needed acknowledgment on the name. We got that from Matthew Burrows about 1 month out from Ignite. SQL Server 2022 would be the name. We also got a much clearer picture of all the features we knew would make the release, so I built this first version of an all-up SQL Server 2022 slide with major features with some rebranding for categories which you can see in the following Figure 1-2.

An illustration of the features of S Q L server 2022. It includes Azure support, an enhanced database engine, analytics at scale, an intelligent database, and extended T S Q L support.

With this in mind, we all decided to keep the announcement for SQL Server 2022 very close to the chest. I was still presenting virtually and even in person (yes, it is true I started presenting at in-person events such as the SQL Server and Azure SQL Conference as early as June 2021). And I would always get asked the question, “When is the next version of SQL Server coming out?” or even better “Will there be a new version of SQL Server?” Both fair questions since we were very quiet on this subject, and by the summer of 2021, it would have been almost 2 years since we released SQL Server or even talked about it.

I then built out more slides talking about each of these areas, including all the features we knew would be in the release. I consulted with Pedro regularly on exactly what we could commit would be in the release, even if it were not immediately in Private Preview builds.

Announcing SQL Server 2022

Preparing for an announcement of this magnitude is no simple task, and our entire team worked tirelessly in October and right up to the announcement for Ignite to have everything in order.

Asad Khan and Anna Hoffman recorded their Ignite presentation announcing SQL Server 2022 (you can see their original session at https://learn.microsoft.com/en-us/events/ignite-nov-2021/azure/breakouts/brk225/).

Joe, Pedro, and Kendal worked to make sure everything was lined up to have the Private Preview build and EAP registration setup.

Matthew and Sonya worked to get our main website (https://aka.ms/sqlserver2022) up and running and build a blog, which would be authored by our Corporate Vice President of Azure Data SQL Engineering, Peter Carlin (my manager, Asad Khan, works for Peter, and Peter works for Rohan Kumar) at https://cloudblogs.microsoft.com/sqlserver/2021/11/02/announcing-sql-server-2022-preview-azure-enabled-with-continued-performance-and-security-innovation.

As we prepared for the big launch announcement, the famous Microsoft Mechanics team (led by Jeremy Chapman) reached out to me about recording a Mechanics video on SQL Server 2022. Microsoft Mechanics (https://aka.ms/microsoftmechanics) is just a cool team. The preparation that goes into the demos, the script, and the animations is a pure marvel. I recorded the session weeks before the announcement in my home in North Richland Hills, Texas, in one take! The result was a very popular video at https://aka.ms/sqlmechanics22. It was the #1 watched video produced for Microsoft Ignite. These types of videos are also a forcing function to get demos working even with early builds. I was able to collaborate with the Managed Instance team to get the link feature for Managed Instance working (along with Anna who demonstrated this in Asad’s session) with very early builds of both SQL Server 2022 and SSMS (SQL Server Management Studio). I worked with our team to get Synapse Link working and was able to craft up a fun demo (with a big thanks to Anna for building a machine learning model in Spark) with a bit of a Texas twist! And then a demo for what you will probably find out as you read this book is my favorite feature: Parameter-Sensitive Plan (PSP) optimization. PSP is “all engine,” so I spun up an Azure Virtual Machine with “pre-CTP” bits and was able to modify a demo that Pedro Lopes and my good friend from development Jack Li gave me.

In the weeks leading up to the announcement, Joe Sack asked me to do a few NDA presentations on SQL Server 2022 to some of our key partners to prepare them for our launch. It was pretty fun to do these as it was the first time I had actually told anyone about SQL Server 2022 outside of Microsoft.

One last piece of promotion for the SQL Server 2022 Private Preview launch was a partnership with Intel for an article in WIRED magazine. The resulting article at https://www.wired.com/sponsored/story/data-is-driving-the-future-of-business was wildly popular and received a lot of views at launch and for months afterward.

One other interesting twist for me was a trip to Redmond the week before Ignite in late October 2021. Due to the pandemic, I had not traveled to Redmond since February of 2020. I went to Redmond for a few reasons. First, I needed to record a webinar in our Microsoft studios on SQL Server 2022 with Matt Burrows and Anna Hoffman. You can see that webinar on-demand at https://aka.ms/sqlserver2022webinar. The second reason was to record a demo for the PASS Summit keynote with Rohan and Peter Carlin. It was fun to record this because I recorded the demo using the PSP demo I did for Microsoft Mechanics, but the twist was having Conor Cunningham interrupt me during my recording. We kind of planned this out, but funny enough we captured it in the first take. It was fun to do since both of us were used to being on stage live with Rohan for the PASS keynote, which couldn’t happen in 2021. You can watch the recording of that keynote at https://youtu.be/Ydlg1KpmrKU?list=PLoGAcXKPcRvYWLmrDZJ9XTdSJAduAefm7. The third reason I traveled to Redmond was to meet some of my colleagues in person for the first time in almost 2 years. It was great to see folks in person again. While I was up there, Rie Merrit, who runs our MVP program, asked if I would do a preview announcement of SQL Server 2022 for the MVPs since we were making our big announcement the next week. It was really fun to do that session because it was the first time I had told a fairly large audience about SQL Server 2022. I’m used to keeping this type of confidential information, but it is still hard. For example, we had reserved slots for the PASS Summit the week after Ignite for SQL Server 2022 sessions but couldn’t even tell the PASS event organizers the details of our sessions. In addition, Thomas LaRock had invited me and others to come speak at their SQL Live event in Orlando in November (https://sqllive360.com/ECG/live360events/Events/Orlando-2021/SQL.aspx), and I had reserved a session called “Microsoft SQL Engineering Session Under Wraps.” The organizers were asking me, “Can’t you tell us anymore about this?” and my answer was “Not really.” At the MVP session, I could hear Thomas in his head going, “Ah, that is why Bob couldn’t tell us.” I had the same issue with sessions I was doing at the SQL Server and Azure SQL Conference (formerly SQLIntersection) in December in Las Vegas.

It was a whirlwind week, which also marked my 28th work anniversary at Microsoft. From SQL OS/2 to now, I would have never dreamed I would be still working on SQL Server.

Finally, on November 2nd at Microsoft Ignite, Scott Guthrie announced SQL Server 2022. We launched all of our other assets along with Asad’s session as well at Ignite. I also feel that LinkedIn can be a powerful method to make announcements, so I crafted a LinkedIn article the day of launch at www.linkedin.com/posts/bobwardms_activity-6861322082387664896-mBzK.

I felt the reception of SQL Server 2022 Private Preview was very positive. I think many in the industry and community quite frankly wondered whether we would ever build another major version of SQL Server. I personally saw reviews and comments of surprise that we not only were building another version but also we were including some very fairly major enhancements including all the Azure-connected features, built-in query intelligence, SQL Ledger, and data virtualization. It was a little awkward to make this announcement because we started with Private Preview, which is not how we announced SQL Server 2019. There was a precedence for this with SQL Server 2017, but I realized the frustration this caused for some who were anxious to download the bits and see what was inside.

Our sessions were very well attended, and I thought the survey results showed folks were excited. This is because during the virtual Microsoft Ignite, we didn’t really get to go into many details about all the features of SQL Server 2022. PASS was our first chance to do this for the community in a public fashion. If you registered for that event (the general sessions were free), you can watch the recordings of our talks. But you can also see the original decks we used at https://aka.ms/sqlserver2022decks (use the Private Preview folder to see our original decks). If you look closely at the first slide for these decks which you can see below in Figure 1-3, you will notice an image of downtown Dallas, Texas, paying tribute to project Dallas. The backstory there is that for the webinar filmed in October 2021, our production team built a cool intro slide and it had a downtown city on it but it was Chicago (just to show a cool downtown image – they didn’t know the project name). I asked the team to change to an image for downtown Dallas. Kind of my “Easter egg” in the original decks paying tribute to our project name.

An introduction to Microsoft S Q L server 2022 original deck with a skyscraper view of a city on the right. The name of principal architect Bob Ward is mentioned.

As we were moving toward the end of calendar year 2021, someone told me that our PASS Summit sessions were only available for free to those who signed up for the event. So I asked Rie Merrit if she knew anyone who would host me for a free virtual presentation on SQL Server 2022. Rie quickly found the Virtual DBA Fundamentals group, so I did an overall SQL Server 2022 presentation for them, which you can see at https://youtu.be/M1h4kSZYdu4. Shortly after this I found out that PASS had posted all of our SQL Server 2022 sessions on PASStv at https://www.youtube.com/user/SQLPASSTV/videos.

All was going well into December of 2021, and we were riding high and ready to crank it up in 2022. And then one day Joe Sack called me to tell me he was leaving Microsoft. As much as I was happy for Joe to pursue new opportunities, privately I was crushed because not only is Joe a friend but I really felt he would be vital to make SQL Server 2022 a success. But SQL Server is more than one person. It is the legacy and tradition of one of the most successful products in the history of Microsoft. I knew we still had all the right people to bring it home for a successful release in 2022.

Private to Public Preview

After the announcement of Joe Sack leaving, I received another call from my longtime friend at Microsoft, Ajay Jagannathan. Ajay told me that he would be taking over from Joe to manage all the program management for SQL Server (as well as Azure Arc data). We have a deep bench at Microsoft, so knowing that Ajay and Pedro would be leading our efforts to release SQL Server 2022 buoyed my spirits.

Early in the year, we faced some challenges (which also caused some challenges for me for this book) and encountered some surprises. The biggest challenge was that several features we had announced for SQL Server 2022 were not available yet in CTP builds. CTP 1.0 was the first build to come out in November in 2021 followed by CTP 1.1 in December 2021 and CTP 1.2 in January 2022. You can see my SSMS connected to the very first official CTP build here in Figure 1-4.

A screenshot of the C T P build of the S Q L query window. On the left, the object explorer files are listed with a S Q L query result on the right.

Unfortunately, not all features were fully baked at this point, and some were not even in these early CTP builds. The good news is that our PM team had started to document existing features in a GitHub repo created by Kendal Van Dyke. Here is a look at the GitHub repo we used during Private Preview in the following Figure 1-5.

A screenshot of the S Q L server early adoption program repository. It lists the releases and the working GitHub files with a few folders on top.

With these challenges also came some great momentum. I found many internal Microsoft customer teams and sales groups wanting me to present to them SQL Server 2022. The word had spread fast about the new capabilities of SQL Server 2022, especially the cloud-connected features. Folks at Microsoft wanted to learn this quickly so they could start educating their accounts.

Then I found out that I would hit the road again at more events than I could have imagined. SQLBits was back in person in London in March 2022. The SQL Server and Azure SQL Conference was held in Las Vegas in April 2022. And Dell Technologies World was back in person in Las Vegas in early May of 2022. At each of these events, I gave the message of SQL Server 2022 and all its rich capabilities (with the amazing assistance of so many of my colleagues to make it all happen).

Then as we prepared to launch the first Public Preview (CTP 2.0) of SQL Server 2022, my friend Pedro Lopes announced he was leaving Microsoft in May of 2022. Could our team take these losses of key experts and leaders for SQL Server and still deliver? As much as I was saddened to see Pedro leave, I realized something as I saw other people on our team step up. We all have a great passion to do what is right to build and deliver a great product. One that matters. One that has value.

The Path to General Availability

We moved forward and announced the first Public Preview at the Microsoft Build virtual conference in late May of 2022. You can see myself and others present at Build at https://aka.ms/sqlserver2022build. The summer moved fast. I presented SQL Server 2022 both internally and externally at events like HPE Discover, Microsoft Inspire, the Data Platform Summit, and various other user group and Data Saturday events. Along the way we updated our CTP build to 2.1, and then came the first Release Candidate build in August of 2022. When a Release Candidate build comes out, you know the General Availability is imminent. As I make my final updates to this chapter the General Availability is very close. I’m excited to know we will “make the name”.

For me it was an interesting challenge for the book because I wanted to squeeze every possible update to our features that came in preview builds right up to the product launch. I believe I was able to accomplish that, but you will find a few “at the time of the writing of this book” moments in some of the chapters.

Introducing SQL Server 2022

Now that you know the history and “behind the scenes” of how we made it to the release of SQL Server 2022, let’s step through an overview of what is in SQL Server 2022 to set the stage for the rest of the book.

Built on a Foundation

One of the lessons I learned as I toured the world in 2018 and 2019 presenting SQL Server 2019 at big events and directly to customers was that I forgot to talk about previous releases and the foundation of SQL Server 2019.

An illustration titled the next step for S Q L server. It includes the S Q L servers 2016, 2017, and 2019 with the S Q L features of each year.

Customers I would talk to were not even on SQL Server 2016 and were not aware of features like Query Store, Polybase, Always Encrypted, or Intelligent Query Processing. This wasn’t their fault but mine for not getting the word out more about SQL Server 2016 and 2017. SQL Server releases are cumulative, so everything you learn about in this book for SQL Server 2022 comes with features as you see in Figure 1-6. SQL Server 2019 Revealed can give you deep insights into what was in SQL Server 2019 that you also get when you upgrade to SQL Server 2022. More importantly, we built new features in SQL Server 2022 as enhancements to these capabilities or used them in new and innovative ways. With this context in mind, let’s look at the major new enhancements for SQL Server 2022.

Wheel of Power

In 2019, I built a slide for my presentations on SQL Server 2019 with “quadrants” to show the major capabilities of the release. (You can see this slide in the first chapter of SQL Server 2019 Revealed.) After presenting SQL Server 2019 several times, I started to call this my “camera slide.” That is because anytime I would display it at an event, everyone took photos of it (which is pretty humbling). But the concept is the same. Create a single slide that captures the major capabilities of the release.

For SQL Server 2022, I did something similar with a different twist. Figure 1-7 represents what I call the “Wheel of Power.”

An illustration of S Q L server 2022 capabilities. It comprises of S Q L server 2022 in the center connected with features of extending T S Q L, cloud-connected, built-in query, security, and data virtualization.

The “dash” lines represent areas where SQL Server can connect to sources outside of the server or VM. Not every area has the same investment, but this provides a nice categorization of our features. And as you read the rest of the book, you can see several chapters are organized around the spokes of the wheel.

Getting Started with SQL Server 2022

You now have read an overview of what is possible with SQL Server 2022 and know that the following chapters of the book help you learn details of each of these areas of the Wheel of Power. You may be asking, “How can I get it and just start using it?”

A Cloud-Connected, Intelligent, and Industry-Proven Data Platform

I often called SQL Server 2019 a modern data platform because that release can be used for so much more than just what a database engine provides. SQL Server 2019 provides capabilities needed for a data platform including data virtualization, a robust engine, and machine learning services

SQL Server 2022 takes a hybrid data platform to the next level. SQL Server 2022 is connected to the cloud on your terms in new and innovative ways never seen before. SQL Server 2022 comes with built-in query intelligence, solving some of the most common and expensive query tuning problems you and developers face today. And SQL Server 2022 comes with an industry-proven database engine with innovations in security, scalability, availability, and data virtualization and enhancements to the most popular database language in the world: T-SQL.

Ajay Jagannathan, our Principal Group Program Manager over SQL Server, gave me his perspective:

This is a special release of SQL Server. During planning for every major version of SQL Server, we spend considerable cycles to look at several key themes that will define the release. This includes industry leading innovations, cloud born as well as mission-critical features and most importantly customer and community feedback. This has helped us in shaping SQL Server as the most popular commercial relational database on the planet. For SQL Server 2022, it was critical for this release to be an integral part of the Microsoft Intelligent Data Platform by bringing some of the latest innovations such as Link feature for MI, near real-time analytics with Azure Synapse Link and governance with Microsoft Purview. It doesn’t stop there, our customers will also get to leverage advancements to SQL Server’s industry leading mission critical capabilities spanning much desired themes around T-SQL language, intelligent query processing, performance, security, availability, and scalability to name a few. And by becoming the first Azure Arc integrated release out of the box our customers can leverage Azure benefits anywhere, in their environment of choice through hybrid and multi-cloud deployments. Not only am I proud of the work done by the entire SQL Server product team to bring this release to our customers, partners and community, my dear friend and colleague Bob Ward does a phenomenal job in this book to help deep dive into several areas of the product and simplify it for the audience. Reading this book will give passionate SQL Server users a clear understanding of the features and the ability to maximize these capabilities in their environments. I can’t wait for our customers to get started with SQL Server 2022 and hear all the cool scenarios where they are leveraging the features to solve their business problems.

If you want to learn more about the installation and upgrade experience of SQL Server 2022, Chapter 2 can be used to start your journey.