What You’ll Do
- Design, develop, and preserve scalable knowledge pipelines and workflows to ingest, rework, and retailer giant datasets.
- Collaborate with knowledge scientists, analysts, and software program engineers to know knowledge wants and ship efficient options.
- Optimize and improve current knowledge processes for efficiency, scalability, and cost-efficiency.
- Implement knowledge high quality checks, validation, and monitoring to make sure knowledge accuracy and reliability.
- Develop and handle knowledge warehouses, databases, and different storage options.
- Guarantee compliance with knowledge governance and safety insurance policies.
- Keep up-to-date with rising applied sciences and finest practices in knowledge engineering and apply them as applicable.
An Best Candidate Ought to Have
- Bachelor’s or Grasp’s diploma in Pc Science, Engineering, or a associated subject.
- Confirmed expertise as a Information Engineer or in the same position and expertise with ETL.
- Proficiency in programming languages reminiscent of Python and expertise in SQL
- Huge knowledge instruments: Information- and Delta-lakes
- Cloud: Naked-Metallic, Hybrid infrastructure
Good to Have
- Expertise working with media recordsdata (transformations)
- Torch dataset expertise