I am a Platform Engineer and it feels like your experience mirrors mine. Like you, our challenge is filtering out large volume but also filtering out LLM abusers. We're not opposed to people using LLMs, when appropriate. I find that candidates inappropriately use it to circumvent the process and that is a big deal for me (and our team). We try to do the right thing(TM) by the candidates by creating minimal interview workloads, asking highly relevant questions that aren't "gotchas", and updating their candidacy as soon as possible. It doesn't feel like many candidates are interested in returning the same courtesy. This kind of behavior means we have to lean harder into tapping our existing networks for sourcing "trust-worthy" candidates. That puts us at risk for creating additional blinders and also unfairly filters out "un-networked" candidates. For whatever it's worth, we are remote-first org so all of our interviews are done remotely.
One of the things I'm thinking about doing in the future is sharing the screen with diagrams and adding irrelevant annotations to it (while clearly indicating to the candidates that those are irrelevant) as a primitive adversarial AI technique. Perhaps on-site interviews is part of the solution.
When folks are engaging in mass circumventing of pervasive processes, it's because the process has broken 'typical' attempts to interact with it.
You're being penalized for doing right by candidates but it's likely that a lot of those candidates were penalized previously when they tried to interact the 'right' way with other folks hiring and adapted workarounds as a result.
It's a quintessential arms race. For what it's worth, I appreciate that you're trying hard to keep your hiring process broad and to mitigate your potential blind spots. That's refreshing to hear from a hiring manager.
Yep. Hiring managers are flooded with thousands of bullshit applications because job seekers are flooded with thousands of bullshit jobs, and/or unfairly filtered out of the funnel for real jobs. So now it’s a matter of sheer application volume for candidate employees more than ever, who after all are in a rather more desperate position than potential employers will ever be.
Besides the arms race with AI on both sides to filter/escape being filtered, the other problem is that it’s completely normal these days to use so called “hiring” more as cheap version of advertisement or a growth signal to investors rather than to indicate you are actually hiring.
I would hazard a guess that the average job-seeking application count for individuals has gone up not 2x, not 10x, but like 100x in many fields the last few years, and similarly for the time involved. And this happens without the economy as a whole even being in serious troubles. The only people that win here are the staffing platforms like indeed and linked-in, and the options in that space and in recruitment/staffing generally are decreasing as the industry consolidates with M&A. Brutal
I think there is a sort of just world fallacy employed here. It seems more like that there opportunists everywhere, and always have been. LLMs have amplified their destructive potential.
Fellow Platform Engineer here, and I can relate 100% with your comment.
We decided to stop announcing our engineering jobs and go back to mouth to mouth for sourcing candidates for now.
It's a move I didn't want to make as, like you said, it means a lot of less networked engineers will not know about it and all. but for now this was the only way we got rid of the constant stream of letters from AI.
Latest rebranding of “sysadmin”, which became “devops engineer” or SRE a decade ago. It’s the people who shove kubernetes, datadog, and CI/CD tools into every corner.
Platform Engineers are operationally focused software engineers who focus on enablement of other software engineering groups through building self-service tooling and create unified platform for app deployment.
The cultural focus is placed on enablement of teams through self service, whereas DevOps is more about reducing silos and SRE is more about doing infra through the software engineering lens with metrics (SLO/SLA/SLI).
Elitism is alive and well in this little nook. Equating platform engineering, SRE and sysadmin to the same thing.
Platforms are often large scale distributed systems, dealing with problems like ensuring 100000s of compute nodes are in a deployed and in consistent state. Millions of lines of code are written, peer reviewed and committed to solve this problem.
This mirrors an attitude I have frequently encountered from "traditional" or "mainstream" software engineers who devalue any work that isn't features, and don't want to have to work on problems like "make my feature appear on all deployments and work well" - it's just something sysadmins do amirite?
Remember - the vast majority of candidates who take the time to do right by your process get zero reward for their effort. You get a reward in the end, so it feels imbalanced. This is true for VERY good candidates, as well.
Precisely my problem. I only apply if I know I’m a good fit and have the required experience. I spent countless hours manually adjusting my resume and writing cover letter out of my heart. Just got the usual cold rejection from a no-reply address. I know do the same with ChatGPT. I also get rejections, but at least I waste little time and can therefore submit many more applications - so my odds are higher
In our remote interviews, I've started pasting the question into meeting chat that I've already fed into ChatGPT. Mainly because some candidates do actually do better with reading and thinking but it's also just pure bait to paste into their open ChatGPT window. Since I've already got input on my side, if they start reading off ChatGPT output, they get a strike, two strikes and interview is ended.
However, I do believe onsite interviews is best solution but finance obviously screams about cost.
I discovered a new tactic where you ask a vaguely worded question on a niche subject, such that any seemingly off the cuff comprehensive answer must be ChatGPT. Asking something outside the candidate’s declared experience or following up on experience with tech they spoke well to will also reveal discrepancies.
I'm pretty sure the temperature of even GPT4o-mini is not 0 so how would you know it is the something like you have. It would be hard to be reading an answer, it would feel awkward and probably obvious it itself. But I'm just saying that some people would have memorized answers to some standard questions (they apply to many places as you might know).
it is surprising there isn’t some
SaaS bullshit company that solves this problem. we have shit like Pearsons and whatnots when taking exams, I took few certification exams and it was like
- install this thing that takes over my machine
- 360 camera around to show my surroundings
- no phone/watch/…
One would think by now there’d be two Stanford grads with a SaaS shit taking care of this for $899/hr
Last interview I did it was obvious candidate was cheating. Gave him my cell and told him to call me, no speakerphone or bluetooth and hung up Teams meeting - never got a call :)
a company would run this… Not your company taking over candidate’s computer, an intermediary that candidate and potential employer are using.
candidates are already using Slack/Teams/Zoom/… now they get to use Pouixy or whatever BS name someone in SF comes up with. guarantee you this will be a thing in 2025, some stanfords are on the case
If you (the company) send me a company laptop to use for that shit, sure, we can interview that way. It is the same deal with Teams and Zoom. None of that shit is touching my personal devices, it is strictly limited to the work machine.
you might be slightly more receptive to this idea if you have, the company administering the exam needs to ensure no cheating is happening so app starts, all your other apps are shutdown, you get a call through the app to show your surroundings with the camera on your laptop etc before exam begins. at no point in time did I find any of it intrusive or strange, I wanted to get the exam done remotely, they need to ensure that I wasn’t cheating
I assume this "app" is not open source, correct? Is is compatible with Linux systems? Can it run on non-FHS distribution?
> all your other apps are shutdown
I admit I am curious about this bit. Does it just start killing all other processes belonging to the same user ID? Or of all users (since you could get "assist" from process owned by an another user)? At least PID 1 needs to survive the slaughter, but it can be used to run arbitrary code to assist with the cheating. So how does it tell what is "an app" it needs to stop?
How is it too expensive? It takes the same amount of time for the interview, and you presumably have a room available in the office to book for the occasional interview.
And it instantly filters out all the spam applicants and chat GPT cheaters.
too expensive mate… we live in year our Lord 2024 - no one is building 2000 buildings that will be vacant as everyone is working from home (or India…) :)
this requires a simple saas solution - someone’s working on this for sure already as it is already a big issue
I hear you, and yes old school solution sounds absurd, but I suspect interview cheating will be on par with game cheating. Even if you install kernel level cheat protection systems the game cheater's still find ways around them.
These guys already developed an invisible desktop app to help everyone cheat on remote interviews.
no question there will be cat&mouse here but even more incentive for some stanford grads to charge premium for “unbreakable quantum-proof interview experience” :)
I don't believe one needs a startup to solve that problem - there are already a bazillion certification paths for a bazillion tech stacks. The(?) problem is one of trust from the hiring org that the certs mean anything, and that's where the whole discussion devolves into one of (gatekeeping|but muh leetcode|our business problem is special|$other)
I meant a startup that provides onsite screening / verification of candidates for companies. Only pre-verifed candidates can apply to company jobs. If the candidate is not local, the company can use the test center to do a remote screen in an environment where candidate cannot cheat. Etc.
I just brought up certs because back in the day you could not take those test online due to cheating.
Now in the age of AI you can't do any type of testing remote, imo.
I half-way suspected that's what you were going for (testing-as-a-service) but my point still stands: it is a web-of-trust bootstrapping problem. For example, Otherbranch[1] exists, is a startup, and is trying to handle pre-screening candidates, but they seem to have very few companies that are currently using them. One would assume if they were solving a real problem then companies would be beating down their door to get real, verified, actual people and yet.
this ain’t about pre-screening at all, it is about solving a different kind of problem. if you have experienced it already, you haven’t interviewed anyone recently.
You are correct, I'm over here stuck on the applicant side, feeling like both sides of this transaction are suffering from the same lack of trust
I therefore fail to see how introducing another party that the hiring managers have to cede their trust to solves our mutual lack of trust
If your company (since your reply implies that you are at least "hiring manager adjacent") merely needs that testing center to start hiring people, I'm totally open to going on Monday and starting a company to provide that service. I even already have a 4k security camera system I can wire up the room to provide DVR access to your interview candidate's session
But my strong suspicion is that such a video camera enabled room for a fee is not, in fact, the obstacle to getting people hired
11 out of last 12 candidates interviewed read their answers from
chatgpt or the like. always same scenario, video call, interviewer never makes an eye contact and obviously is reading answers. last one I gave my cell and told him to call me, no speaker or bluetooth on the phone and hung up Teams meeting - mate never called back :)))
this is a pandemic already and tool is needed to establish that interviewer is not cheating. prior to today’s tools at interviewer’s disposal this was not really a thing - today it is a huge thing
One of the things I'm thinking about doing in the future is sharing the screen with diagrams and adding irrelevant annotations to it (while clearly indicating to the candidates that those are irrelevant) as a primitive adversarial AI technique. Perhaps on-site interviews is part of the solution.