Big Data Technology Lead

Return to search Apply Now
Published:   2021/12/07


Full time



Required language

Japanese: Business
English: Business



  • Lead an engagement in the space of Big Data with 10-15 members.
  • Interact directly with end-customer as well as lead and manage technical team.
  • Report into Enterprise Architect from technical point-of-view and Delivery Manager from delivery perspective.
  • Experience with building Big Data platforms and areas of data governance, data security, data quality and related tools.
  • Create and maintain optimal data pipeline architecture.
  • Assemble large, complex data sets that meet functional / non-functional business requirements
  • Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
  • Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS big data technologies.
  • Work with QA Test analyst to ensure test coverage (Including Integration & Regression testing)
  • Develop new program logic and/or assembles standard logic modules to create new applications


[Qualifications & Experience]

  • 10 Years of overall technology experience
  • At least 5+ years of hands on experience with Spark, Scala
  • Experience with relational databases (Preferably Oracle)
  • Programming SPARK in Scala & proficiency in SQL to write complex SQL queries
  • Strong data analysis and troubleshooting skills
  • Domain knowledge of Capital Market is plus
  • Knowledge of shell scripts and other languages including Python, R, Java is plus
  • Good understanding of AWS & Open Source big data technologies Airflow, Terraform, AWS Glue, AWS Lambda, Spark SQL
  • Ability to implement both batch and streaming data pipelines in the AWS and change data capture (CDC) experience
  • Experience with Databricks preferable
  • Good knowledge of Linux OS and Shell scripting
  • Experience working on complex distributed information systems
  • Experience with version control systems, preferable SVN, git.