They want to automate this, but also understand that this needs to be in a Data Lake/Warehouse at some point. And this initiative is to start that out to :
- Extract external data automatically on a schedule – Must understand how to get data from external sources
- Load this data into a Data Lake
- Transform this into their landing tables (i.e. a warehouse)
- Work with Analyst to provide a data dictionary
- Allow Analysts to query this data in the landing zone to prepare their results (power bi)
This WILL expand, so the implementation MUST be extensible. At some point, we would like to use Azure Synapse to leverage the full capabilities of scale, analytics and ML.
Who You Are:
The individual to support this should be able to:
- Look architecturally at the different Warehouse methodologies and implement the one that makes the most sense, but has an eye to know how this will extend to a more mature architecture.
- Deep understanding of Data Warehousing Methodologies, and Data Lake Methodologies
- Understands Azure Synapse in order to help lay out a path forward
- Work closely with our Data Architect to ensure alignment
- Work independently, and communicate effectively.
- Ability to be collaborative and think outside of the box. The customer is looking to us to provide solutions. We are not here to simply take orders.
- Ability to analyze and a strong understanding of creating reports
- Strong, intimate knowledge of Azure Data Factory for Orchestrations
- Hands on experience building pipelines using various connections via Azure Data Factory
- Deep understanding of data warehouse modeling (star schema, snowflake, etc.)
- Deep understanding of Data Lake Architecture
- Familiarity with writing complex reports
- Familiarity with BI tools such as PowerBI
- Good communication skills and can communicate technical designs to a non-technical audience