Site Reliability Engineer - Data Pipeline and Hadoop | Twitter, Inc. | San Francisco, CA
Site Reliability Engineer - Data Pipeline and Hadoop
Infrastructure Operations | San Francisco, CA
As a Site Reliability Engineer (SRE) in Twitter’s Data Pipeline and Hadoop team you will be working to improve the reliability and performance of the next-generation data analytics infrastructure at Twitter that processes billions of events daily which powers various products across Twitter. As part of this role you will get a chance to work on various open source technologies like Hadoop, Pig, HBase, etc. You will work shoulder-to-shoulder with our engineering teams to design, build and operate the next generation of data analytics infrastructure at Twitter, focusing on debugging, automation, availability and performance, and above all efficiency at ‘reach every user on the planet’ scale. If you passion lies in solving complex problems, and taking on new challenges, then we’d love to hear from you.
Responsibilities
- Work in engineering team to design, build, and maintain various Data Pipeline and Hadoop systems.
- Diagnose, and troubleshoot complex distributed systems and develop solutions that have a significant impact at our massive scale.
- Participate in building advanced tooling for testing, monitoring, administration, and operations of multiple clusters across data centers, primarily in Python and Java.
- Use Map-Reduce frameworks such as Pig and Scalding
- Work and collaborate across teams such Application services, Linux kernel, JVM and Capacity Planning, Hardware, Network, and Datacenter Operations to design next-gen Data pipeline and Hadoop platforms.
- Troubleshoot issues across the entire stack - hardware, software, application and network
- Take part in a 24x7 on-call rotation
- Participate in software and system performance analysis and tuning, service capacity planning and demand forecasting
Qualifications
- 2-5+ years of managing services in a distributed, internet-scale *nix environment
- Familiarity with any of the system management tools (Puppet, Chef, Capistrano, etc)
- Demonstrable knowledge of TCP/IP, Linux operating system internals, file systems, disk/storage technologies and storage protocols.
- Hands-on operational experience on managing JVM services and understanding of JVM internals.
- Practical knowledge of shell scripting and at least one scripting language (Python, Ruby, Perl).
- Basic familiarity with Java or C/C++ troubleshooting in a production environment.
- Experience with large datasets and Map-Reduce architectures like Hadoop
- Familiar with debugging tools such as JStack, JMap, JHat, gdb, JStat
- Ability to prioritize tasks and work independently
- Track record of practical problem solving, excellent communication, and documentation skills
- BS or MS degree in Computer Science or Engineering, or equivalent experience.
No comments:
Post a Comment