Never meet your heroes. Funny expression, on the one hand it’s about not meeting them so you wont get disappointed by the fact that they’re just humans; On the other it also shows that there’s nothing special about them either, so you can become one too.

For me this just happened after a recruiter asked me about a Senior Site Reliability Engineer position at a crypto exchange.

I always spoke very highly of them, specially their Security and Engineering departments. And I still do; But their recruitment-game is a bit on the lacking side.

Code Challenges are always a tricky topic.

I know many who simply dislike them and find them tedious - Others don’t mind or even love them.

I’m a huge fan of Puzzles so it’s no surprise that I lean more towards loving them.

But there’s a big catch, code challenges should always represent the level of seniority that’s being searched for.

This is a tale of the exact opposite; And ultimately how I’ve been disappointed by it.

Role & Challenges

Their role requirements where very straight forward and nothing too unexpected:

• 5+ years in a DevOps role (DevOps, SRE, etc)
• Extensive experience with Kubernetes
• Extensive experience with monitoring /alerting (primarily with Prometheus/Grafana) and knowledge of best practices in the area
• Experience benchmarking applications and identifying bottlenecks
• 1-3+ years experience with a programming language (primarily Golang)
• Experience with Terraform
• Extremely knowledgeable on security best practices
• Self starter

I’m happy to see the strong emphasis on Kubernetes and Security. Which fortifies my view on their competency.

As many in the business, they use Code Challenge to evaluate their candidates; They call it a DevOps Test.

First let’s roughly talk about how the Test looks like before going into the details.

The Test consists of 6 challenges; 3 are Mandatory and 3 are Optional with a total of 100 points and a pass mark of 35. Yes 35%, I wish my SAT’s were like that ;)

I won’t go into detail about the challenges, but I’ll give you a few examples with regards to the role requirements.

They ask you to create a simple application that will run on Kubernetes. Or at least they try to.

The actual challenge is to dockerize a known cryptocurrency and later wrap it into a StatefulSet.

But if you’re seeking for staff with Extensive experience with Kubernetes and then merely ask for a StatefulSet, I just can’t help but feeling a bit contradicted.

Personally I would’ve at least asked for a fully functional Helm template or something similar. And perhaps also ask for Clustering/HPA support as bonus points.

Observability, Traceability and Monitoring are very important as SRE to quickly identify hotspots before they become an incident.

Unfortunately, despite mentioning and requiring Extensive experience with monitoring /alerting, there wasn’t any challenge about integrating Monitoring into the above StatefulSet/Application.

I wish they would’ve asked for a simple prometheus scraper to feed custom metrics using the Prometheus Operator. Implementing HPA upon the custom metrics could’ve been a good bonus task.

Similar when it comes to Experience benchmarking applications and identifying bottlenecks; There wasn’t anything related to this either.

I would’ve expected a Challenge that asks for flamegraphs, memory profiling or routine optimizations… As most cryptocurrency wallets use JSON-RPC, it would’ve been a great candidate for loadbalancing and introspection through a service mesh to find bottlenecks.

What they do cover, to a degree, is having some programming experience as they ask you to come up with a text manipulation problem and solve it using the common grep, sed and awk tools - as well as rewriting it in a different language than shell-scripting.

I understand that you need to make sure the candidates know basic *NIX toolsets and the paradigms of a few languages but it’s just sad to have to ask for it in this way…

Arguably they covered the Security requirement by asking to pass Container Scans and doing a bogus IAM role in terraform but is that really Extremely knowledgeable? I’m not so sure…

But this is also hard to test, you could ask for a terraform recipe to create cross-account RBAC for different users and services and trust-relations between them; But how are you going to justify that to the candidate? Most just have their personal AWS Account and not 2-3 different ones to test it…

There is so much potential that could (should?) have been asked for with this level of seniority.

So what’s the Takeaway from this?

I guess that you as an employer should really consider when to use Code Challenges and more importantly match the seniority of the Challenges to the role they should cover.

Giving a Staff Engineer a challenge to write Hello World as Shell script might just turn them away from you.

These Challenges are a double-edged sword.

But I’ve also heard from many in the field that they’re told to dumb them down because recruitment cant find people able to solve them.

That’s hardly the Challenge’s fault and merely a sign of the inflationary market for Junior Engineers.

A company I’ve had the joy of getting to know had it all figured out right.

They sent me some logic based coding tests; You know which I mean, how many different coins can represent a set amount of change etc.

But then they also asked me to describe an algorithm to sort Phone numbers by carriers to obtain their cost per sent SMS.

Or describe what a Search Problem is and how A* or Dijkstra’s Algorithm work as well as mention where it’s commonly applied to.

This wasn’t even an engineering role, this was “just” for a Sr. Cloud Architect.

I fully enjoyed the half day being grilled by them, drawing wild things on whiteboards and discussing optimization strategies for various cloud setups on a extremely deep level.

The DevOps Challenge however left me in disbelief and wondering if I should even spend any more time on it.

And more importantly it left me wondering what their every day work is like if they ask for this sort of thing from a Senior level…

Final Words?

Not sure. I expected more from them but that’s exactly what Never Meet your Heroes is about, isn’t it?…

I still believe they are a great company and are doing a lot of good work in their field and having very skilled and qualified staff.

My opinion about them has not changed. But my perception has.

I see how they struggle with the same issues others have when it comes to recruiting and Code Challenges.

But there isn’t an easy solution and I dont claim to know any.

I’ve seen so many of my colleagues in the field being told that their requirements are too high and they need to adjust lower.

At the end they get staff but barely can use them for the intended job.

More and more Tech companies emerge and the amount of skilled and talented people is not increasing at the same rate.

Something that’s not being touched here at all was the whole topic of Softskills. That’s topic for a whole new post.

PS: I just got my hands on GitHub Copilot and it’s up to you to find out which of the sentences were written by it’s AI and which ones were written by me.