Senior Data Engineer 

Description

The Senior Data Engineer will be responsible for the design, develop, and maintain:

Best Practices for Data Architecture and Design Patterns for Data Engineering Use-Cases.

Real-time data feeds from database sources like MySQL, Oracle, and MS SQLServer using the “Change Data Capture Engine” aka “CDC”.

Sub-Second Real-Time Data Pipelines using AWS Kinesis, Glue, Spark, Kinesis, Lambda .

Batch Pipeline Orchestration using on Apache Airflow and Jenkins.

Auto Scalable platform using Kubernetes on EKS.

Org Wide Master Data Management, Data Catalog Engine, and Data Quality Engine.

Code Repo Pipeline to automate Continuous Integration and Continuous Deployment (CI / CD).

Structure Tables with Partitioning and Clustering to increase Cost & Performance  Benefits.

Guide Data Analysts and Data Scientists to write efficient queries and workloads.

Data Sharing with On-Demand Encryption/Decryption which can operate at Scale.

Running Containerized ETL workflow at scale.

Qualifications

Bachelor Degree of Computer Science, IT or equivalent

at least 3-4 years of data engineering experience

Have strong fundamentals in Computer Science concepts like Cloud Computing Architecture, Distributed Computing, High-Velocity Data Processing, Lambda Architecture, etc…

Strong Data modeling and managing Distributed Computing Platforms for Data Processing.

Advance knowledge of SQL and writing resource-efficient queries.

Have at least 2+ years of professional programming experience in Python.

Have at least 2+ years of experience in running a data processing pipelines on either of these: Google BigQuery, Redshift, Hadoop, Presto, Spark, or KSQL.

Have at least 2+ years of experience in writing Sub-Second Real-Time pipelines using Google DataFlow(Apache Beam), PubSub, Kinesis Stream, Lambda, etc...

Have a good understanding of how Kubernetes clusters work and scale on-demand.

Have adequate experience using Containers for Data Engineering workload.

Implemented manual or automated tools for Data Quality, Catalog, and Lineage.

Uphold the sense of Frugality across Data Engineering teams.

Have Good Interpersonal and Presentation Skills to explain and promote Best Practices across the organization with both technical as well as non-technical stakeholders.