Job Title      : EMR_Spark SME
Experience  : 5-10 Years
Location      : Bangalore

Job Description :

Technical Skills:

• 5+ years of experience in big data technologies with hands-on expertise in AWS EMR and Apache Spark.
• Proficiency in Spark Core, Spark SQL, and Spark Streaming for large-scale data processing.
• Strong experience with data formats (Parquet, Avro, JSON) and data storage solutions (Amazon S3, HDFS).
• Solid understanding of distributed systems architecture and cluster resource management (YARN).
• Familiarity with AWS services (S3, IAM, Lambda, Glue, Redshift, Athena).
• Experience in scripting and programming languages such as Python, Scala, and Java.
• Knowledge of containerization and orchestration (Docker, Kubernetes) is a plus.
• Architect and develop scalable data processing solutions using AWS EMR and Apache Spark.
• Optimize and tune Spark jobs for performance and cost efficiency on EMR clusters.
• Monitor, troubleshoot, and resolve issues related to EMR and Spark workloads.
• Implement best practices for cluster management, data partitioning, and job execution.
• Collaborate with data engineering and analytics teams to integrate Spark solutions with broader data ecosystems (S3, RDS, Redshift, Glue, etc.).
• Automate deployments and cluster management using infrastructure-as-code tools like CloudFormation, Terraform, and CI/CD pipelines.
• Ensure data security and governance in EMR and Spark environments in compliance with company policies.
• Provide technical leadership and mentorship to junior engineers and data analysts.
• Stay current with new AWS EMR features and Spark versions to recommend improvements and upgrades.

Requirements and Skills

• Performance tuning and optimization of Spark jobs.
• Problem-solving skills with the ability to diagnose and resolve complex technical issues.
• Strong experience with version control systems (Git) and CI/CD pipelines.
• Excellent communication skills to explain technical concepts to both technical and non-technical audiences.

Qualification:

• Education qualification: B.Tech, BE, BCA, MCA, M. Tech or equivalent technical degree from a reputed college.

Certifications:

• AWS Certified Solutions Architect – Associate/Professional
• AWS Certified Data Analytics – Specialty