Changing Nature of Database Management
A few months ago I was presenting to a large audience on the topic of running SQL Server in the cloud in IaaS and PaaS. I was touting the auto install, auto management, and the auto patching capabilities. I mentioned the capability of Azure IaaS to auto provision an AlwaysOn cluster. I demoed the simple step of picking a ‘template’ and auto installing SQL Server. A gentleman sitting in the audience raised his hand and interrupted me, asking “Is the cloud going to get us DBAs out of a job one day?” There was silence for a few minutes in room……
Database Management, as we know today, has evolved over the last few decades. Starting with punch cards, programming circuit boards, physical manipulation of storage devices, etc. we have now arrived at the era of cloud computing. Provisioning a server today simply means going to a Web portal and clicking ‘Next’ a few times. Administration and Management tasks which used to a few hours just a few years ago, now have been reduced to few minutes or have been totally eliminated as a result of cloud computing.
Database Management as a trade is changing and in a few years’ time we could possibly see even more automation coming from platform related capabilities. This poses an interesting dilemma to DBAs, Server Admins, and professionals in similar areas who provide administration and instance / database level management and support. What should they do so they adapt and change with the changing times and technology landscape? How should they keep their skills relevant? Let’s explore what the change in technology is, how it came about, and what it means to the DBAs today, more important, what should a DBA do to stay relevant and in a job.
Cloud Evolution
Public Cloud
The cloud has evolved significantly over the last decade or so. My first exposure to the cloud was through a SaaS email application. We all are familiar with email and productivity apps in the cloud. Then a few years ago an online ‘book seller’ shook the industry by introducing IaaS in the public cloud at a large and affordable scale. There were several Data Center provider vendors even prior to AWS, which was a type of IaaS, however, AWS brought to the market lot of the auto provisioning and auto install capabilities through a simple web portal. They also made public cloud available to the masses. You could provision and rent just one server or hundreds of servers. Microsoft Azure and Google soon followed suite.
Private Cloud
While the public cloud was catching on, similar progress was being made in the private cloud area. VMWare and Microsoft Hyper-V provided the virtualization and the management capabilities to enable a cloud like environment on-prem. Instance provisioning, template based software deployment, auto patching, monitoring, HA and DR type of functionality became routine and part of the virtualization technologies. Administrators did not had to design and build any more for those abilities. They had to do lot less, since the inherent cloud technology provided them with those abilities.
Technologies such as the Windows Azure pack delivers Microsoft Azure technologies for you to run inside your datacenter enabling, rich, self-service, multi-tenant services and experiences that are consistent with Microsoft’s public cloud offering. The Windows Azure Pack integrates with System Center and Windows Server to help provide a self-service portal for managing services such as websites, Virtual Machines, and Service Bus; a portal for administrators to manage resource clouds; scalable web hosting; and more. (As per: https://www.microsoft.com/en-us/cloud-platform/windows-azure-pack )
Cloud Impact on DBAS
In summary, the cloud has and will continue to have an impact on what the DBA is required to do and how much management and maintenance is required from the DBA. We suspect, a lot of the basic DBA management functions will get automated by the cloud platform.
Technological Advances
Along with the cloud evolution, technological advances from a feature and capability perspective added a new and interesting dimension to the DBAs. The feature advancement and capability to some extent made query tuning, IO management, disk configuration etc. less pronounced and somewhat less demanding. Just to be clear, I am not saying that a good design or a good IO configuration or efficient disk layout will not be necessary any more, all I am saying is more capabilities got added in the platform that helped significantly with performance, such as the few mentioned below.
Columnstore Indexes
SQL Server 2012 made the first attempt in introducing the Columnstore Index. It was basic and was an additional index (non-clustered) sitting on top of the table. The table had go in the ‘Read-only’ mode as a result of this. In SQL Server 2014, the columnstore index could be made the clustered index, thus giving you the compression advantage and the performance. After the first physical IO, the entire index could then be in-memory.
The columnstore index is predominantly meant for the scan centric DW / Reporting type of queries. I have seen/heard query times come down significantly as a result of this index for very large reporting / DW type of queries.
In-Memory Computing
SQL Server 2014 added In-Memory OLTP tables and indexes to the mix, project Hekaton. Though severely restricted in usage scenarios due to limitations such as no constraints, no FKs, limited datatype support etc., the overall performance improvement can be huge, anywhere from 2x to 40x (workload and IO dependent).
Appliances
Along with software feature improvement, came the advent of Appliances, hardware based devices for specific workload scenarios, such as the APS (formerly called PDW) for DW type workload. Through MPP type architecture, very large DW query performance got significantly improved and queries that used to take hours got reduced to minutes.
Feature and Appliance Impact on DBAs
Through the feature advancement and the advent of the appliances, some of the DBA tasks in the performance tuning area got a little simplified. A DBA no longer had to worry about the complex query tuning intricacies in terms of indexing, statistics, data distribution, query and join structure, etc. In fact, Microsoft proudly claims regarding its APS appliance that it will significantly reduce and to a large extent even eliminate query tuning etc.
Again, just to reiterate, I am not suggesting that software features or appliances makes good design or good architecture redundant, all I am saying is these improvements add more capability and power for query performance.
How does a DBA stay relevant?
Without a doubt, the nature of Database Management has changed and cloud and technology improvements have injected a lot of automation. The basic nature of the DBA role will change, if it has not already. As a DBA, you need to be prepared for the day when your companies’ entire infrastructure or a significant portion of it could be hosted on a public cloud. For some companies, this may never happen, or may only happen partially. However, in any case, it is important to understand what it means to you as a DBA and even more important to assess the changes that are required to retool, to stay relevant in an ever changing technology world. I don’t think there is a reason to panic, but there definitely is a reason to be aware of the changing world and based on your specific situation, to proactively plan and adjust so you are not left without a job.
Some of my thoughts and suggestions to DBAs from a retooling perspective are as follows.
Key Skills to be successful
Let’s examine the key skills necessary to be successful in today’s IT world
Soft skills
No surprise here, soft skills such as good written and oral communication skills will always be important. Ability to present information, either in a formal presentation or as an informal discussion always makes a big difference. I have seen people with good communication skills do well, compared to their peers with similar technical skills but not so developed communication skills. At the end of the day, the ability to convince others to go along with your idea or the ability to garner support for your initiative will always be critical.
Brand Management
I don’t think people (especially technical people) give enough thought to their personal Brand. I am fully aware most of the technical folks hate this topic or even the idea, but unfortunately brand or perception carries a lot of weight.
Actively work on your Brand image. Position and portray yourself as a very flexible technical resource who understands business very well and can adapt, change and retool if and when necessary. An example of this will be bringing up newer technologies to the management’s attention and how it can be applied to the business. Always remember, the technical discussion has to be use case driven, not purely technology driven. The application of technology to solve a business problem is more important than the technology itself.
Technical Adaptability
Essentially the ability to adapt to new technologies and not show resistance to change. Learn as fast as you can and learn the newer cooler stuff.
Business Enablement as opposed to DBA stuff
Think in terms of Business Enablement / Initiatives, as opposed to purely technical activities. For example, if cost savings is a business initiative, think how a specific feature can help cut cost. Maybe a new type of compression to save on disk space etc.
Data Orientation as opposed to technical orientation
Data has been and will always be the most important asset for any organization. Data could be in the form of customer records, sales purchase history, product lists, web logs etc. If you retool yourself simply to understand all the strategic and tactical data sources in your organization, their attributes, life cycle, etc., you can have a data oriented conversation and add value as the data person, as opposed to the technical DBA.
Architecture and Planning as opposed to Maintenance and Uptime
Let’s assume for a moment that you agree with me that the traditional DBA functions such as server/instance management, maintenance, backups, HA/DR etc. are provided by the cloud platform. What is and cannot be provided by the cloud platform is the Data Architecture and the Design / Planning that goes with it. Data Architecture will always remain critical and fundamental. The most efficient organizations I have seen have the most optimized and efficient Data Architecture.
Data Management as opposed to Server / Instance Management
Managing the life cycle of the data will be an important skill. Fits in with the Data Architecture point above. Data Management will include things such as Data Lifecycle, Retention, ETL, DW, Reporting, BI, and Analytics.
Analytics
Being skillful or having knowledge of Data Analytics (traditional BI as well as Analytics, Machine Learning, Data Mining etc.) will go a long way. This could be a good area to research and get trained on.
In Conclusion
As the Greek philosopher Heraclitus quoted ‘Change is the only constant’. Our world is changing, don’t resist, get ready. Evaluate the pace of change at your organization and proactively plan and retool. And if your organization is not changing or unwilling to change, maybe time to consider a new employer.
