Data Engineering: Management Requirements

Data engineering is becoming one of the most important roles in today’s IT industry. Every business is relying on data and the ability to analyze and transform this raw date into trustworthy and valuable management information.

As a data engineer you need to ensure that up to date data is properly received, transformed, stored, and made accessible to other users.

That’s easier said than done… To do data engineering well for your company you need to make sure your activities are in line with the overall company goals and objectives. Once you understand the context in which Data Engineering operates you can set up your management metrics and KPIs to make sure it is 100% clear to every stakeholder involved what the value add is of data engineering.

To understand the depth of management requirements for Data Engineering, you may want to perform a Self Assessment. This will give you a current status overview, a line in the sand to have clarity and understanding where data engineering adds value. It will also visualize where the weak spots / areas for improvement are.

When you look at self assessment questions, it’s important to answer them based on your own personal opinion and experience. This becomes even more important when you fill out the Self Assessment with your team. Each individual will answer the questions differently – but keep in mind that the ultimate answer to each of these questions is:

‘In my belief, the answer to this question is clearly defined’

You can go even further and ask for documented evidence, rather than just opinions. This will move the questionnaire more into an auditing realm as you require evidence to substantiate the answers.

Some of the most important management requirements for Data Engineering are listed below. For each of these questions, think about your current role and try to answer them truthfully.

Are these requirements identified, assessed, implemented and documented? Or is there room for improvement of Data Engineering processes in the organization?

The management requirements are across 5 different phases, which coincide with the general life cycle of a business process. These phases loosely align with Deming’s Quality cycle: Plan – Do – Check – Act (PDCA for short).

  • Plan what you are going to do
  • Do what you planned for
  • Check / study and analyze the results of what you did in the previous step
  • Act accordingly – improve the activities, measurements and expected outcomes.

Phase 1: Recognize the value of Data Engineering for the overall business

What does Data Engineering success mean to the stakeholders?

How to keep the information needed to operate and maintain a product aligned with the changing product over its life cycle?

The questions to ask here are all around the alignment of Data Engineering with the overall business goals and targets. Do we need Data Engineering, and if we do what exactly does that mean? Will Data Engineering actively support the profitability of the business, or is Data Engineering critical to the viability of the business?

Phase 2: Define what Data Engineering means within the context of our business

Are roles and responsibilities formally defined?

Is it clearly defined in and to your organization what you do?

Are there any constraints known that bear on the ability to perform Data Engineering work? How is the team addressing them?

Once there is clarity on the need for Data engineering, it is now time to design and organise the management structure around Data Engineering. That is why these questions revolve around roles and responsibility and clear boundaries and constraints.

Phase 3: Measure & Analyze How Data Engineering is currently performed

Have you identified your Data Engineering key performance indicators (KPIs)?

Have you captured the workflow (which conceptualizes the data inputs, transformations, and analytical steps to achieve the final data output)?

Is someone responsible for migrating data sets that are in old/outdated formats?

In phase 3 the actual work on Data Engineering has begun so the management requirements are all around the collection of data about the Data Engineering processes. There need to be clear metrics and KPIs in place based on the outcomes of the questions asked in phase 1 and 2.

This is the only way we can ensure that the metrics are managed in the context of the overall business goals. What gets measured gets done so this step is important to avoid creating a Data Engineering process that drains resources without adding any value.

Phase 4: Improve the Data Engineering processes

What evaluation strategy is needed and what needs to be done to assure its implementation and use?

Who will be responsible for making the decisions to include or exclude requested changes once Data Engineering is underway?

It will come as no surprise that once we measure and analyze our results, we now enter a phase of process and quality improvement.  Don’t just focus on improving the processes around Data Engineering but make sure you also focus on the improvement of the tools used to perform Data Engineering, the training and education needs of the staff involved with Data engineering and the overall design and architecture of the infrastructure used. Make sure you involve your suppliers and vendors in this process to check how they align with your management requirements.

Phase 5: Control & Sustain the Data Engineering Objectives

What role does communication play in the success or failure of a Data Engineering Project?

What are the short and long-term Data Engineering goals?

This final phase is all about the future. It’s one thing to create a Data Engineering role in your organisation as a one-off project, it’s something totally different to have a solid, consistent and sustainable process in place to ensure Data engineering is performed as a mature business aligned and robust function of the organisation.

Planning strategically for the future and aligning your Data Engineering goals are a critical component of this phase and – done correctly – will help with your competitive advantage in a fast changing, chaotic world.

Article by Ivanka Menken, CEO The Art of Service, author of Data Engineering Complete Self Assessment Guide.


Author of Data Engineering - Management Requirements

Ivanka Menken is a serial entrepreneur and the owner and Co-Founder of The Art of Service since 2000.  Ivanka specialises in creating organisations that manage their services in a sustainable and customer driven manner. With 20+ years of management consultancy experience and an education degree, Ivanka has been instrumental in many organisational change management projects in The Netherlands, USA, Canada, New Zealand and Australia for both government agencies and private corporations. Ivanka beliefs that education and training is at the foundation of every successful enterprise. Ivanka has been a guest lecturer for a number of Queensland universities on the subject of IT Service Management and Organisational Change Management and proudly featured as one of “Australia’s 50 Influential Women Entrepreneurs” in 2016.

While running The Art of Service, Ivanka authored a number of publications on IT Service Management, Cloud Computing and Customer Service. She also completed her Entrepreneurial Masters Program at MIT and served on the board as the second ever female President of the local Entrepreneur’s Organization chapter.