Teradata Think Big Senior Data Engineer in Germany
Think Big Senior Data Engineer
As a Senior Data Engineer I, you will provide technical leadership to clients in a team that designs and develops path-breaking large scale cluster data processing systems. You will mentor sophisticated organizations on large scale data and analytics and work with client teams to deliver results.
Additionally, as a senior member of our Consulting team, you will help Think Big, A Teradata Company establish thought leadership in the big data space by contributing white papers, technical commentary and representing our company at industry conferences.
Design and develop code, scripts and data pipelines that leverage structured and unstructured data integrated from multiple sources. Software installation and configuration. Participate in and help lead requirements and design workshops with our clients. Develop project deliverable documentation. Lead small teams of developers and coordinate development activities. Mentor junior members of the team in software development best practices. Other duties as assigned.
Proven expertise in production software development
7 years of experience programming in Java, Python, SQL, or C/C
Proficient in SQL, NoSQL, relational database design and methods for efficiently retrieving data
Strong analytical skills
Creative problem solver
Excellent verbal and written communications skills in German and English
Strong team player capable of working in a demanding start-up environment
Experience building complex and non-interactive systems (batch, distributed, etc.)
Preferred Knowledge, Skills and Abilities:
Prior consulting experience
Experience with Hadoop, Hive, Pig, Avro, Thrift, Protobufs and JMS: ActiveMQ, RabbitMQ, JBoss, etc.
Dynamic and/or functional languages (e.g., Python, Ruby, Scala, Clojure)
Experience designing and tuning high performance systems
Prior experience with data warehousing and business intelligence systems
Professional or academic background that includes mathematics, statistics, machine learning and data mining
Prior work and/or research experience with unstructured data and data modeling
Familiarity with different development methodologies (e.g., agile, waterfall, XP, scrum, etc.)
Broader experience with spring ecosystem including spring-batch, spring-mvc, and spring-hadoop
Standards-based REST implementation
Configure a Jenkins build, create/update a Jira ticket, enable Automated Tests in gradle/maven build
Familiar with OSI stack and proper use of HTTP verbs etc.
Hive Tuning, Hive physical design (file formats, compression, partitioning, bucketing), Hive DSS Queries, Hive for Data Science, Hive for running R/Python (Streaming), Hive Transactions, Hive-HBase
Knows how to tune a job including parameters, more efficient API calls. Understands Spark SWL. Understands Spark Streaming and can transform a DStream.
Ability to write advanced UDFs, Serdes, input-loaders, log analysis, how the logical operators map to the lower, level physical implementation
Setup and leverage output from Ganglia, Nagios, Ambari, Cloudera Manager, etc.
Able to link Kerberos KDC to a backing LDAP or Active Directory authentication provider
Advanced understanding of access-driven key design, appropriate denomalization, use of co-locating records of differing schemas in a single table etc.
Understanding of best practices for Hive schemas. Denormalization, partitioning and bucketing, file formats.
Able to create, write to and read from Kafka topic. Understanding of key partitioning (just how it works), able to maintain an offset in the topic for consistent reading
Must be able to sit for long periods of time working on computers. Must be able to travel to client sites at least 50% of the time. Must be able to interact and communicate with the client in meetings. Must be able to write programming code in applicable languages. Must be able to write project documentation in English.
Bachelor's Degree or foreign equivalent in Computer Science or related technical field followed by six (6) years of progressively responsible professional experience programming in Java, Python or C/C . Experience with production software development lifecycle. Experience with Linux, SQL, relational database design and methods for efficiently retrieving data. Experience building complex and non-interactive systems (batch, distributed, etc.).
Master's Degree or foreign equivalent in Computer Science or related technical field. Four (4) years of experience programming in Java, Python or C/C . Experience with production software development lifecycle. Experience with Linux, SQL, relational database design and methods for efficiently retrieving data. Experience building complex and non-interactive systems (batch, distributed, etc.).
Employer will accept any suitable combination of education, training, or experience.
Primary Location: Europe, Middle East & Africa-Germany
Organization: TB Services International
Req ID: 168391