The app for independent voices
Today, data engineers are busier than ever
Companies seem to think the engineer title means they can fix or build anything π¨
The result? Overworked staff with unclear direction
In one of my recent newsletter articles, I talked about this and tried to define what the role should, shouldnβt be and the grey zone for Data Engineers
π οΈ ππππ’π§π’ππ ππππ ππ§π π’π§πππ«π’π§π πππ¬π©π¨π§π¬π’ππ’π₯π’ππ’ππ¬ β The role requirements that are relatively standardised across organisations
- ETL/ ELT Process & Schema Design
- Data Pipeline Development
- Data Integration
- Orchestration
- Performance Optimisation
- Version Control & Documentation
π ππ«ππ² ππ«ππ πππ¬π©π¨π§π¬π’ππ’π₯π’ππ’ππ¬ β These tasks represent things data engineers should be supported on by other roles, but often find themselves doing it alone
- Data Architecture Design
- Data Modelling
- Platform Infrastructure
- Reverse ETL
- API Development for Data Access
- Data Catalogue Integration
- Data Privacy & Security
π« πππ ππππ ππ§π π’π§πππ«π’π§π πππ¬π©π¨π§π¬π’ππ’π₯π’ππ’ππ¬ β The things that Data Engineers often are asked to do because of a lack of resources or non-existent org structure/ operating model. For each activity, the DE should play a role but should not own it
- DataOps
- Requirements Gathering
- Data Quality Monitoring/ Observability
- Data Democratisation
- Operational Tooling Support
- Data Governance Adherence
There may be responsibilities and tasks missing in each of these categories (please comment if you have additional suggestions), but this shows how stretched engineers are and the huge expectations inherent in the role
Check out a recent article I did on this in The Data Ecosystem newsletter (thedataecosystem.substaβ¦)
