Sr. Infra Engineer, NetOps

We are looking for an impact-driven Sr. Infrastructure Engineer to define and build elegant, performant, and resilient systems for tomorrow's web.

Enthusiasm about the decentralized web and blockchains has brought an influx of people who want to use distributed systems, but don't know how to build the necessary infrastructure. We are building that infrastructure.To continue. that work, we're looking for people who thoroughly understand the principles of distributed systems and cryptography, and who will lean into the challenges of applying those principles in open-source code that will be deployed worldwide.

Distributed Systems Engineering at Protocol Labs

Distributed systems engineering lies at the center of many projects at Protocol Labs. The Infra team at Protocol Labs runs critical infrastructure that helps power networks like IPFS, Filecoin, libp2p, Drand, and other related projects. This requires rigorous engineering across protocol and infrastructure design, through all phases of implementation. All of this happens in an environment defined by curiosity, passion, and a love for open source. We encourage each other to think big, run experiments, and support each other in a blame-free environment.
Our infrastructure engineering team is growing, and you will have the opportunity to apply creative ways of solving complex challenges associated with building, securing, and operating large-scale infrastructure that is unique to the decentralized nature of Web 3.0

As an Sr. Infrastructure Engineer at Protocol Labs, you will…

• Design, develop, and maintain infrastructure in a mix of cloud-based and traditional environments to power large-scale, massively distributed, fault-tolerant services while ensuring the highest security standards.
• Work with standard tools to monitor and inspect the different deployments -- choosing and creating tooling to quickly assess health, evolution, and any adjustments needed.
• Work alongside a cross-functional team including software design & development, product management, and ecosystem engineers. Provide technical leadership, support, and best-practices to stakeholders across the PL Network (inside/outside the org).
• Support launch of new products & services by participating in system design, building and deploying software-defined infrastructure, capacity planning, and operational readiness.
• Operate and improve infrastructure for large-scale services such as the IPFS HTTP Gateways, IPFS Clusters, etc., by measuring and monitoring availability, latency, and overall system health to identify and mitigate risks.
• Incorporate monitoring, alerting, and observability to support services that allow us to maintain the highest standards of security, reliability and uptime.
• Scale systems sustainably through mechanisms like automation, and evolve systems by pushing for changes that improve reliability and developer velocity.

You may be a fit for this role if you have...

• Enjoy building infrastructure tools, and services that improve velocity for software development teams by enabling them to consume, and manage infrastructure in a self-service manner.
• You look for opportunities to automate, systematize, and document through APIs, runbooks, etc.
• Have experience deploying production-grade infrastructure in an automated, reliable, and portable manner using Continuous Integration & Continuous Deployment tools such as GitHub Actions, CircleCI, TravisCI, or similar.
• Comfortable with software-defined infrastructure & configuration management tools such as Terraform, Ansible, or similar.
• Experience with contemporary monitoring & metrics tools such as Prometheus, Grafana, InfluxDB, etc.
• Experience with container orchestration technologies such as Kubernetes, etc.
• Deep understanding and experience with core Internet protocols (BGP, IP, TCP, DNS, TLS, HTTP), data caching in networks, and Linux system administration.
• Have experience designing, administering, and securing cloud environments.
• Place high value in documentation, and sharing of knowledge through effective written and verbal communication with internal and external stakeholders.

Highly-valued bonus points include:

• Experience working remotely in a distributed team.
• Experience with Open Source Software.
• Being accountable for operational support based on an on-call rotational model.
• Familiarity with programming languages such as Golang, and the Docker ecosystem, (but you don’t necessarily need to be an expert).
• Experience with aggregated logging tools such as Filebeat, Logstash, Elasticsearch.
• Experience iterating and driving infrastructure projects with initiative and autonomy.