Spark Scala Developer

Infosys is seeking an experienced Spark Scala Developer to design, develop, and optimize scalable big data solutions. The candidate will work on building high-performance batch and real-time data pipelines leveraging the Hadoop ecosystem and distributed computing frameworks such as Spark. The role involves working closely with data engineers, architects, and business stakeholders to deliver robust, scalable, and efficient data processing systems.

Required Qualifications:

• Candidates authorized to work for any employer in Canada without employer-based visa sponsorship are welcome to apply. Infosys is unable to provide immigration sponsorship for this role at this time.
• Candidate must be located within commuting distance of Mississauga, Ontario, or be willing to relocate to the area.
• Bachelor’s degree or foreign equivalent required from an accredited institution. Will also consider three years of progressive experience in the specialty in lieu of every year of education.
• At least 4 years of Information Technology experience.
• 4+ years of experience in Big Data technologies.
• Strong expertise in Apache Spark (Core, SQL, DataFrames, RDDs), Scala programming, and PySpark.
• Hands-on experience with Kafka, the Hadoop ecosystem (HDFS, Hive, Impala), and NoSQL databases such as HBase, MongoDB, and Couchbase.
• Strong understanding of distributed computing concepts and data processing frameworks.
• Experience building ETL/data pipelines for large-scale datasets.
• Proficiency in SQL and data modeling.

Preferred Qualifications:

• Hands-on experience with data lakes, data warehouses, and scalable ETL pipeline design, including batch and real-time processing architecture.
• Strong understanding and practical exposure to Agile software development methodologies, including Scrum, and SDLC practices.
• Proven experience in the Banking domain, supporting use cases such as fraud detection, risk analytics, regulatory reporting, and customer insights.
• Excellent analytical, problem-solving, and communication skills, with the ability to translate business requirements into scalable technical solutions.
• Demonstrated ability to work effectively in cross-functional, multi-stakeholder environments, collaborating with Business, Data Engineering, and Architecture teams.
• Experience with real-time data streaming frameworks such as Kafka and Spark Streaming for low-latency processing.
• Understanding of data modeling concepts, including dimensional modeling and snowflake schemas, to support analytics workloads.
• Experience and desire to work in a global delivery environment.

Key Responsibilities:

• Design and develop large-scale data processing pipelines using Apache Spark, Scala, and PySpark.
• Build and optimize batch and real-time data processing workflows using Spark, Kafka, and the Hadoop ecosystem.
• Develop Spark applications using RDDs, DataFrames, and Spark SQL for complex transformations.
• Develop and optimize PySpark applications leveraging joins, Spark DAG execution flow, stage optimization, transformation techniques, and streaming with dynamic allocation and failover handling.
• Implement streaming pipelines using Kafka and Spark Streaming or Structured Streaming.
• Develop and maintain HDFS-, Hive-, NoSQL-, and Impala-based data lake solutions.
• Convert existing SQL/Hive workloads into optimized Spark jobs for improved performance.
• Work with ETL pipelines to ingest, cleanse, transform, and process large datasets.
• Optimize performance through partitioning, caching, serialization, and tuning techniques.
• Handle data formats such as Parquet, ORC, Avro, and JSON.
• Integrate multiple data sources including streaming systems, flat files, RDBMS, and APIs.
• Collaborate with cross-functional teams to understand business requirements and translate them into scalable technical solutions.
• Ensure data quality, reliability, and performance monitoring across pipelines.
• Participate in code reviews, design discussions, and best practices implementation.

Key Skills:

• Distributed Data Processing.
• Spark Optimization and Performance Tuning.
• Real-time Data Streaming.
• Data Modeling and ETL Design.
• Problem-solving and Analytical Thinking.
• Strong Communication and Stakeholder Management.

Nice to Have:

• Exposure to Machine Learning pipelines or MLOps workflows.
• Experience with the Databricks platform.
• Experience with AWS or GCP.

Summary:

This role requires a highly skilled Spark Scala Developer with strong expertise in big data engineering, streaming systems, and distributed computation, capable of building scalable, high-performance data platforms supporting enterprise analytics.

The job entails sitting as well as working at a computer for extended periods of time. Should be able to communicate by telephone, email, or face to face. Travel may be required as per the job requirements.