This work is very exciting to me for a few reasons:
- HTML is an incredibly rich source of visually structured information, with a semi-structured representation. This is as opposed to PDFs, which are usually fed into models with a "flat" representation (words + bounding boxes). Intuitively, this offers the model a more direct way to learn about nested structure, over an almost unlimited source of unsupervised pre-training data.
- Many projects (e.g. Pix2Struct https://arxiv.org/pdf/2210.03347.pdf, also from Google) operate on pixels, which are expensive (both to render and process in the transformer). Operating on HTML directly means smaller, faster, more efficient models.
- (If open sourced) it will be the first (AFAIK) open pre-trained ready-to-go model for the RPA/automation space (there are several closed projects). They claim they plan to open source the dataset at least, which is very exciting.
I'm particularly excited to extend this and similar (https://arxiv.org/abs/2110.08518) for HTML question answering and web scraping.
Exciting/scary stuff! A sophisticated enough version could carry out any range of tasks that a typical computer user/browser could from just a few sentences with somewhat high chance of success.
we will overuse this tech, forgetting important processes that are perhaps wise to keep a "human backup" for redundancy. Then again, RPA is already a case where a "proper" rewrite of some multi-program pipeline is impossible.
This is a "classic" tension. Having worked in the (broader) RPA space for a while, I would say that the true north star of most processes is (a) rewriting the internal procedures to be transformations on data (not UIs) and (b) standardizing communication across companies.
There is a lot of momentum to solve (a) with no code, but it's slow because processes are impossibly complex. I think AI will accelerate this and could result in the "human backup" dystopia. On the other hand, AI can also be used to generate code, and I'm optimistic that technology like this can accelerate humans' ability to encode complex processes robustly (as transformations of data) and would 10 or 100x less work than no/low code.
> On the other hand, AI can also be used to generate code, and I'm optimistic that technology like this can accelerate humans' ability to encode complex processes robustly (as transformations of data) and would 10 or 100x less work than no/low code.
Ah right, lots of angles to consider! A hybrid system would certainly be interesting. Let the AI runtime generate and evaluate code to perform tasks (e.g. selenium/puppeteer in python/java). Upon failure, "escalate permissions" to enable DOM control, or full mouse/keyboard to complete the task (probably best not to let the thing open up a code-editor with M/KB controls though heh)
The model used by this research is T5 which is open sourced already. So I think once the dataset is released, we'll see the open version of pre-trained model very soon.
DocQuery (https://github.com/impira/docquery), a project I work on, allows you to do something similar, but search over semantic information in the PDF files (using a large language model that is pre-trained to query business documents).
For example:
$ docquery scan "What is the due date?" /my/invoices/
/my/invoices/Order1.pdf What is the due date?: 4/27/2022
/my/invoices/Order2.pdf What is the due date?: 9/26/2022
...
It's obviously a lot slower than "grepping", but very powerful.
Wow this is exactly what I've been looking for, thank you! I just wish with these transformer models it was possible to extract a structured set of what the model "knows" (for e.g. easy search indexing ). These natural language question systems are a little too fuzzy sometimes.
Can you tell me a bit more about your use case? A few things that come to mind:
- There are some ML/transformer-based methods for extracting a known schema (e.g. NER) or an unknown schema (e.g. relation extraction).
- We're going to add a feature to DocQuery called "templates" soon for some popular document types (e.g. invoices) + a document classifier which will automatically apply the template based on the doc type.
- Our commercial product (http://impira.com/) supports all of this + is a hosted solution (many of our customers use us to automate accounts payable, process insurance documents, etc.)
Your commercial product looks very cool, but my use case is in creating an offline-first local document storage system (data never reaches a cloud). I'd like to be enable users to search through all documents for relevant pieces of information.
The templates sound very cool - are they essentially just using a preset list of (natural language) queries tied to a particular document class? It seems like you're using a version of donut for your document classification?
> but my use case is in creating an offline-first local document storage system (data never reaches a cloud).
Makes sense -- this is why we OSS'd DocQuery :)
> The templates sound very cool - are they essentially just using a preset list of (natural language) queries tied to a particular document class? It seems like you're using a version of donut for your document classification?
Yes that's the plan. We've done extensive testing with other approaches (e.g. NER) and realized that the benefits of using use-case specific queries (customizability, accuracy, flexibility for many use cases) outweigh the tradeoffs (NER only needs one execution for all fields).
Currently, we support pre-trained Donut models for both querying and classification. You can play with it by adding the --classify flag to `docquery scan`. We're releasing some new stuff soon that should be faster and more accurate.
Sweet! I'll keep an eye on the repo. Thank you for open sourcing DocQuery. I agree with your reasoning: my current attempts to find an NER model that covers all my use cases have come up short
The unstoppable administrative engine that is the American Healthcare system produces hundreds of thousands of continuously updated documents like this with no standardized format/structure.
Manually extracting/normalizing this data into a querable format is an industry all its own.
What is the development date? -> June 20, 2017
What is the medicine? -> SPINRAZA® (nusinersen)
How many doses -> 5 doses
Did the patient meet the review criteria? -> Patient met initial review criteria.
Is the patient treated with Evrysdi? -> not
> to extract a structured set of what the model "knows"
To be fair, that's impossible in the general case, since the model can know things (ie be able to answer queries) without knowing that it knows them (ie being able to produce a list of anserable queries by any means significantly more efficient than trying every query and seeing which ones work).
As a reducto ad absurdum example, consider a 'model' consisting of a deniably encrypted key-value store, where it's outright cryptographically guaranteed that you can't effiently enumerate the queries. Neural networks aren't quite that bad, but (in the general-over-NNs case) they at least superficially appear to be pretty close. (They're definitely not reliably secure though; don't depend on that.)
This is so epic. I was just ruminating about this particular use-case. who are your typical customers. Supply chain or purchasing? Also I notice that you do text extraction from Invoices? Are you using something similar to CharGRID or its derivate BERTGRID?
Wish you and your team more success!
Thank you ultrasounder! Supply chain, construction, purchasing, insurance, financial services, and healthcare are our biggest verticals. Although we have customers doing just about anything you can imagine with documents!
For invoices, we have a pre-trained model (demo here: https://huggingface.co/spaces/impira/invoices) that is pretty good at most fields, but within our product, it will automatically learn about your formats as you upload documents and confirm/correct predictions. The pre-trained model is based on LayoutLM and the additional learning we do uses a generative model (GMMs) that can learn from as little as one example.
If your goal is to be a better coder long term, I’d suggest learning both. As many have pointed out, Go is quick to learn, and I think learning both will only take a small amount of extra time (compared to learning Rust).
They each present trade offs that make them a better tool under particular circumstances. While Rust exposes you to more sophisticated typesystem features, Go's M:N scheduler is an incredible piece of technology that is (IMO) unmatched by any other mainstream language.
Finally, regarding the garbage collector, if you learn both languages, you'll get to viscerally experience the tradeoffs of having the garbage collector (try writing the same program in each language). There are some projects where it makes sense to use one and others where you shouldn't. Trying out both is the best way to build up intuition for this kind of trade off.
> Go's M:N scheduler is an incredible piece of technology that is (IMO) unmatched by any other mainstream language.
I guess it depends how you define mainstream, but in Rust, Rayon has a work stealing scheduler too.
On the other hand, it has existed in Erlang for decades, and Elixir takes full advantage of it.
As for my 2c, I would say Rust is a better choice than Go in terms of the first language to learn. The main reason for this, is it is easy to embed in other languages. When I get to a problem that is too slow in a higher level language, eg. Python or Elixir, Rust is a great way to solve this problem. Just have a look at polars (https://www.pola.rs/).
It allows you to create many more language threads (go routines) than kernel threads, and unlike other languages properly hides this abstraction from you with a rope stack (so you don't need to create coroutines or use async/await syntax).
I'm not up to speed with the latest and greatest in Haskell or Elixir, so they very well may have something similar. Rust has great runtimes, like tokio, that do similar things, but without rope stacks and with painful async/await syntax.
A M:N Scheduler was neither revolutionary nor rare even when Go was launched. Even a mainstream like C# already had one (albeit based on continuations, until C# 5.0 came out). At the same time, some mainstream programming languages either had similar third-party M:N stackful schedulers when go came out (gevent in Python) or got them a little while after (Quasar in Java).
Go's scheduler was only somewhat unique in the combination of features it pursued:
1. M:N
2. Stackful (i.e. unoptimized memory usage for each task/goroutine)
3. But using very small stacks[1] so it's easier to create a very large number of goroutines.
4. Integrated with the GC
5. Colorless functions
6. Built-in
7. No access to native threads
8. Not configurable or customizable.
9. Run as Native code, without using a virtual machine.
Colored functions (Async/await or Kotlin's suspend) are a matter of taste. They're heavily criticized for the burden they add, but advocates prefer the extra type-safety they provide. If you want to be able to statistically analyze or data races (or prevent them completely, as Rust does), I don't think you can avoid them.
Speaking of Rust, Rust did start with an M:N, work-stealing scheduler based on stackful coroutines. This scheduler was eventually removed[2] from the standard library, since it was deemed a bad match for a systems language.
Go was originally marketed as a systems language, but it was really a language that was optimized for writing concurrent servers by large teams of programmers with varying experience[3]. Specifically it was designed for server software at Google[4], and to replace the main place C++ was used for in Google: I/O-bound server software. That's why Go made very radical choices:
- Go Native (we still need good performance, but not C++-level)
- Maximize concurrency
- Make concurrency easy
- Use GC (we need the language to be much easier than C++)
- Minimize GC pause times (we need reasonable performance at server workloads)
This meant that the Go M:N Scheduler was usually the best performing stackful scheduler for server loads for a while. Interpreted languages like Python, Node and Lua were slower and were either single-threaded or had a GIL. Erlang and Java used a VM, and weren't optimized for low latency GC. C and C++ had coroutines libraries, but since these languages were not garbage-collected, it was harder to optimize the coroutine stack size.
I think it created a wrong impression that the Go scheduler was revolutionary or best-in-class. It never was. The groundbreaking thing that Go did is to optimize the entire language for highly concurrent I/O-bound workloads. That, along with a sane (i.e. non-callback-based) asynchronous model and great PR from being a Google language helped Go popularize asynchronous programming.
But I wouldn't say it is unmatched by any mainstream language nowadays. Java has low-latency GC nowadays and it's working on it's own Go-like coroutine implementation (Project Loom). But all mainstream JVM languages (Kotlin, Scala and Clojure) already have their own M:N schedulers.
Rust, in the meantime, is strictly more performant than Go: It's using state-machine based stackless coroutines, which emulate the way that manual asynchronous implementations (like Nginx) are done in C. You can't get more efficient than that. Not to mention that Rust doesn't have a GC and features more aggressive compiler optimizations.
[1] IIRC, initial stack sizes has changed between 2k-8k during the language lifetime, and the stack resizing mechanism has changed as well.
It really depends on the goal. If one wants to end up with an elaborate hellow world, probably yes, one can learn both.
Learning Rust to make it applicable to mid/large-scale projects, and potentially for a professional transition (my case), is a one-of-a-kind commitment, in my opinion, hardly compatible with learning another language at the same time.
In that case, I'd suggest learning Go first, as a kind of "step up" to learning Rust. If you haven't already, I'd also suggest learning C.
I think there's some linearity to these tasks. Rust (and C++) are an amalgamation of many complex features that are present in simpler predecessors. Learning C for example will not only speed up your ability to learn Rust, but also leave you with a deeper understanding of the core principles.
> If you haven't already, I'd also suggest learning C.
Learning a new language is not something to take lightly. Should I take 6 months in order to get more solid foundations before moving to Rust? 1 year? N years?
My biggest hurdle has been ownership ("borrow checking", which is the common problem) in the broad sense (which includes: learning to design a whole program considering ownership). This is something that C/++ programmers are surely familiar with, so it absolutely helps, but I don't think it's efficient to learn C/++ in order to step up to Rust.
The more languages you learn, especially with reusable concepts, the less of a tax it is to learn a new one. As someone who has written C/C++ for over a decade, learning Rust felt very straightforward -- I just had to learn about the borrow checker, and it was fairly straightforward to reason about how it works based on my knowledge of unique pointers in C++.
I certainly don't think C++ is a prerequisite to learning Rust, but I do think your path to deeply understanding Rust will accelerate if you understand C (or specifically, understand pointers and heap allocations). But to each their own!
Learning new languages is certainly useful/important, but time is finite. The idea of learning a certain language as a bridge to another is unrealistic unless one has a _lot_ of time.
Just replied to another comment recommending C. Can you help me understand which core principles C would help to refine i.e. what am I leaving on the table if I go with a Rust or C++?
Nothing, these languages all expose the same underlying memory controls. But going with C may still be valuable, because rust and C++ has much better abstractions and especially with Rust, it can easily hide the pointer manipulation part from you, when that’s the reason one presumably learnt the language for.
> Just replied to another comment recommending C. Can you help me understand which core principles C would help to refine i.e. what am I leaving on the table if I go with a Rust or C++?
In the other thread you mention features. I suggest C because of its "simplicity", that is, its lack of features. It really is a thin abstraction on top of assembly. C's closeness to the hardware is edifying.
Moreover, since the goal is learning, History is quite relevant. Languages evolve in the context of their predecessors. You probably already know this for typescript since it is so close to JavaScript. But how do C++ and rust relate to C? And how does Go relate to C++ and C?
Languages always involve tradeoffs. For example, a language that checks memory bounds at runtime (Go) uses more CPU cycles than one that doesn't (C). A compiler that does checks is more complex than one that doesn't. Language features can help with safety (rust), but then the language takes more time to learn. Complexity can be hidden (Go garbage collection) to make up front learning easier, but this can make the language less flexible or more difficult to debug. Complexity can be exposed later but then the language learning curve is lengthened.
I'm not suggesting you go become a C pro, but learn enough to shoot yourself in the foot. It will help contextualize the features of these other languages and give your more mental tools for decision making in your own code.
Pointers :). C++ and Rust try to hide the concept away with References, which add a very helpful amount of type safety, but do not protect you from the fundamental idea of allocating memory and then later free-ing it.
It's not that you leave it on the table with Rust/C++ but more that you'll have a deeper understanding for what's going on if you understand how pointers work in C.
Thanks this is what I was looking for! Pointers are one of those things I’ve heard about for years but don’t know too much about. I know that in js objects are passed by copies of a reference to that object in memory. I have never felt that there was something missing here — only that it requires an understanding of which types in js are passed by value vs reference. What is the upside to pointers in C? For instance in js I can pass an object to a function that modifies the object’s properties. The only thing I can’t do (I think) is pass an object to a function, and then reassign that object’s reference to a new object literal such that the original object reference now references the new object literal. Is this gap in a language like js where pointers come in, or am I missing the forest for the trees? If this is where pointers become useful/beneficial to be knowledgeable on, can they really be that useful? Full disclaimer that I don’t know anything about pointers aside from implementing data structures in js e.g. a linked list node has a `this.next` property, but in other languages this seems to be called/implemented as a pointer from what I’ve seen.
To sum my thoughts on this up: can you help me understand why pointers would be useful for me to invest in learning on the level of C versus continuing to let them be abstracted away/not present in my language? What would that knowledge do for me? Will I be a better problem solver etc. thanks again!
It's not that there's an "advantage" to pointers, but rather it's the reality that they exist. Everything you're doing in JS also "uses" pointers under the hood, but the language and runtime abstract that away from you. C exposes you to pointers directly, so you have an opportunity to learn what they are and how they work.
It's important to learn how they work, because it'll give you a better idea of what's actually happening in the higher level languages you're using. That may not be important if you stick to JS or Go, but if you'd like to learn Rust then it's impossible to accurately think through tradeoffs like whether to use Box, Rc, Arc, etc. without understanding how pointers work. In other words, I'd only recommend learning Rust if you take the time to understand pointers, and the best way to understand pointers is to write C.
If you want to play with MDX without having to set it up, check out https://motif.land. It comes with a collaborative editing environment too, that is incredibly easy to use for authoring markdown (our non-technical folks use it too).
This is mind blowing stuff. I was skeptical that you could use the browser effectively as an intermediary to the file system, and that CRDTs would work reasonably well on files, but they seem to have overcome both obstacles. I’m curious, how do you envision this approach working with version control systems? What would it mean to “explore” a branch of a git repository, for example? Would that overwrite the global version of the file system?
We're in an exploratory phase on this. We have been experimenting with reading the .git folder to determine the current branch, and store this info alongside the CRDT in the client. Still early to say whether this is a fruitful approach or not.
At the very least, that does not work for rolling back transactions with the database/sql (https://pkg.go.dev/database/sql) package, although it may work for other cases. We've had numerous production bugs result from this.
You can't take that for granted, but it turns out in practice that memcached does not use atomic add, and so it happens to be straightforward to support multiplication with the same set of locks.
Now that's a compliment. "We couldn't use your code as a basis for interviews because it was too obvious how it worked and how to modify it." Definitely a condition worth striving for.
Author of the interview question here. I came up with the idea after working at the company for a couple months and realizing that the skill of "diving into an unfamiliar area of the code and quickly figuring it out" was very important for us. Database codebases are large and complex -- so much so that almost every new feature you work on for the first year or so feels a lot like doing the question.
Over the years of evaluating with it, we learned a lot about its quirks (both false positives and false negatives). At one point, someone got so frustrated that they threw their laptop on the floor. Happy to answer any questions about it!
I would pass the shit out of this question because it's right in my wheelhouse, but I don't consider myself a particularly exceptional programmer. I have a CS degree, but I'm positive I'd fail a FAANG interview w/o advance prep. I'm really comfortable at modifying existing code[1], gluing together existing solutions, and know enough C to get through this question.
But the OP says this:
> When you’re maintaining a large codebase, there are always going to be codepaths you don’t fully understand
Ugh... I hate modifying code I don't understand. That doesn't mean I need to understand the entire code base, but if I'm modifying a portion of the code base, I'm loath to touch it till I understand what it's doing.
Also, of all the work I do, I consider this the least expressive of my ability. It's such a mechanical thing to do and doesn't require a whole lot of thinking?
So I guess I can see how this skill—for a job at your company—is necessary but in no way sufficient.
I YOLO’d the Google interview two times. First time I failed. Second time I somehow passed. It can be done. Might have gotten a higher level with studying but I don’t think I would have ever studied.
After interviewing people for a while, I've learned how to suss out what people gain from practicing vs actual engineering instinct
I've passed people who "didn't meet the bar" because I could tell they just didn't practice, but exhibited 4 stars on every "Ultra Instinct" signal. Programming speed isn't important, correctness & habits are what are important. + or - 10 minutes to finish a problem doesn't really matter in the daily job
I've had two memorable interviews where timed coding was part of the interview, and I wowed the management team but did not get a call back due to taking about 10% too long on the coding. This is a good thing, in hindsight, considering the eventual fates of those businesses that hired like that.
It's counter-intuitive not to test for coding at an interview for a developer. But you just can't learn what you need to know, as a hiring manager, from a pass/fail timed test. This greatly informs my hiring process now.
I'm a former FB eng and a Xoogler and have been an interviewer at both places. The recruiter sets the level for your interviews. There's an option for the interviewer to recommend you for a higher level, but there's no incentive for the interviewer to do so and is almost never used. Getting higher rating from the interviews will only make the hiring committee's decision to hire you easier, and almost never affects your levelling.
I've done two onsites with Google in the past essentially YOLOing it (I only study my interview failures because I view otherwise as an inefficient use of my time) - first time did terrible, second time almost passed if I didn't completely bomb my very last session. The second time ended up not really mattering because two different teams in two different orgs for my current non-Google FAANG wanted to hire me after onsites done on back to back days (side note: that was almost 15 hours of interviewing in two consecutive days - that's a lot of time, I only was able to do it because I was funemployed at the time).
I actually appreciate it very much if a candidate didn't study & focus more on giving the best answers to their capability when I interview them - the questions I give them are usually questions that no amount of studying would have prepared them for, so already taking the mindset of trying to respond thoughtfully & earnestly to problems & situations that change on a whim puts them a step ahead.
I studied for the interview I wanted (by being a thoughtful software engineer in my day job) and not for the interview they offered (which would require me to either be a professional leetcoder or some algo/performance expert). If they didn’t want a good software engineer then they’d have to pass on me. I made it clear in every session that I was thinking through the problem, asked good questions, and when there were aspects I could write concrete code for I did. If you score by solution competence and performance I think I aced 2 of the coding sessions and did pretty mediocre in the other 2. My interviewers must have been willing to go a bit off of the default mode of operation as I managed to get an offer.
I don’t know if interview performance has anything to do with negotiating power, but I was able to get damn near the highest possible total compensation my level allows for without a competing offer.
I YOLO’d FAANG interviews twice (The interviews were in cities I wanted to visit, so why not)passed both times and even accepted the offer the second time. In the end I wasn't a very good fit for the company.
What happened if you don't mind me asking? (I might be in a similar situation lol. I just started a FANG on a team of all Indian ppl and I'm the only white dude and I can barely understand the speech much less the code)
I worked with games development most of my life and then went to web development in this change, for some reason I ended up in position a bit more senior than what I was prepared for in this domain. I thought the job would be fine, specially being my second successful interview with the company, I thought their processes meant that I was good enough for the job they offered me.
Is game development as tough regarding WLB as people say? It's always been one of my dreams to work in Game Dev but not sure if I should keep it as a hobby or give it a shot. Are there sane Game Dev co's with decent WLB?
I mostly worked on smaller studios and for the last 6 years of that career I was working in my own company, I only had WLB problems in my first 2 years, but I never worked in AAA
I got a job as a front end developer. We had a ton of streaming data and needed to index it in the front end.
The "right" solution would be to fix it in the back end so that we didn't fetch all that data when it wasn't needed, but that wasn't possible because of the horrific project/product management.
So I had to build binary search trees to index the data so we could work with it fast enough to have a reasonable user experience.
So yeah -- you will need some of this stuff. You're constantly going to be searching for things, reversing things, looking for patterns.
it won't be so clear and abstract, as a leetcode puzzle but the reason why I had to write that binary tree was because whoever came before had clearly never considered using one and was doing everything completely wrong. It was a disaster and made the codebase insane.
If they filtered in the hiring process for people who know the basics then things would have been a lot more performant, and they wouldn't have burned so much time working around the performance issues caused by his terrible solution.
I absolutely agree that hiring folks who are aware of the basics is important -- essentially, you want engineers who are aware of the space of possibilities.
In my view, what makes folks dislike typical coding interviews is that in the real world, what you need is a solid understanding of what algorithms exist and when to use them/what to look for, rather than the knowledge of how to build one on-the-fly.
To solve the issue you described, you don't need to know offhand how to implement a binary search tree on a whiteboard. You do need to know how to identify indexing as a bottleneck, and how to broadly think about a solution. You could then search for indexing strategies and, having studied them at some point in the past, you'd be able to pretty quickly refresh your memory and find the one that's a good fit for the problem at hand.
For this reason, I've always thought these exercises would be much better off as essentially "open book" rather than real-time whiteboarding problems -- because that reflects how engineers actually work. That's also what I've pushed for in my own workplaces, and we've had good success finding talented folks, and heard positive feedback about this aspect of the process.
I would add to this that while knowledge of computer science algorithms and data structures is _absolutely_ useful, being able to implement a binary search tree RIGHT NOW, in under an hour, is not necessary for... pretty much any job anywhere.
I've worked at 75% of FANGs. My method was basically just bone up on algorithms and data structures by going onto the Wikipedia list and trying to implement them on pen and paper. Practice thinking out loud, practice space management(no backspacing on pen and paper). Be honest if you've heard a question before. Know how to analyze runtime of your algorithms.
I chose to interview in python, even though I know other languages better, because it is fairly dense relative to, say java or c++ and easier to write by hand.
I can't reply to the sibling commenters but providing a contrarian opinion. I'm an interviewer and there's no way for me to tell who is just really good vs who has seen the exact question before. Telling the interviewer just gives them information that goes against you so I wouldn't recommend doing this.
I always prefer to say "I've seen a similar problem before, let's see if I can remember how to solve that" or "let's see if the same kind of approach works here".
This is a balanced approach. It's honest: you probably haven't heard the exact question word for word, even if you can't recognize the differences. And it's actually what they're selecting for in the first place. They're not expecting you to invent the concept of a binary tree, or whatever, but to know it exists and how to implement it and to recognize where it might be the right concept to apply.
I've never had any interviewer say "OK forget it, let me give you another question where you'll never have seen something similar".
I was thankfully asked (in the interview, not just assuming I’d been asked the Q before).
Question was: write code to determine if the stack grows up or down. I’d been writing computer games for several years and smashed it (after telling the interviewer that it would be necessarily technically rely on undefined behavior) and the interviewer somewhat dismissively said “you should have told me someone else asked you this question” “What are you talking about? This is just an easy question.”
A good answer would be "no, but when I was working on <game title> which is a game that has to run on many different consoles including <console>, we had an interesting but where <weird stuff happened> and the stack cookie I inserted to test if it's a buffer overflow remained pristine. After a lot of debugging, at the stage where I started suspecting it's a compiler bug and inspecting raw opcodes, I was looking at my debugger and noticed the stack pointer was lower than the stack cookie address and understood this CPU has a stack that grows the other way than I'm used to".
> “you should have told me someone else asked you this question”
At what point do we end up saying "my current employer 'asked' me this question, because it's part of my day to day job..."? At some point you have some experience in certain areas that you just 'know', and it's not some sneaky "oh I crammed leetcode for 3 weeks!" tactic.
Yes, it seems to me that if interviewers are going to be annoyed about this then they should stop using leetcode and/or using generic interview questions.
I got hired from that loop. Same guy challenged my failure to write ifdef include guards on my header file on the whiteboard. My answer of “oh, you’re right; I’ve configured emacs to automatically insert those, so I don’t have to think about it” seemed to more than satisfy him and we ended up being close as colleagues for several years.
It does in this case yes, how would you see it as an advantage? You have to pass X number of algo/data structure, your being honest will only make them find a different question. If you get the different question wrong guess what you are out. It is what it is.
If the hiring process is set to punish honesty, then maybe by being honest you avoid working at places with such a process and consequently with the people who passed it?
Note that I'm not saying that FAANGs are like that (I have no opinion on that).
I am not denying honesty won't hurt, and probably would help you. But again, the interviewer is going to have to get a replacement question...you have a reduced chance of solving it correctly.
I have a question that I've been asking for years that is somewhat domain-specific, but should be answerable by anyone who knows what a set/map are. About 20% of people get past "stage 1" within 5 minutes, usually people with finance/trading experience, and some will say "oh, I implemented something just like this a few years ago." Some people take 30 minutes and do ok, and some never finish. When someone tells me they have had this problem before, I tell them it should be really easy for them then, and we can move on to more interesting things!
For those that DO get past stage one, it is a boss with multiple health bars. We talk about what improvements could be made, then what improvements on THOSE improvements. We keep digging until we are out of ideas. The best candidates are the ones that stump me, or introduce new ideas. I present this problem as a pair problem solving challenge, so there is no one right answer, and has a lot of back and forth.
The whole leetcode approach to interviewing is basically
1. interviewers asking questions that they wouldn't be able to solve
2. interviewees pretending to solve on the spot things they've memorized
3. interviewers pretending to believe them
There is no way for you to tell if someone is really good (as is) vs who has grinded leetcode since the process (which assumes prior preps) a priori discards the first group.
Leetcode grinding is suggested as necessary by everyone, including recruiters (in-house or 3rd party), hiring managers, prospective colleagues. You can see comments from devs at FANGs even in this thread.
I understand that this is a common belief, but I don't agree that it is strictly true; it is contrary to my own experience, both as an applicant and as an interviewer.
Note that we are talking about a particular interview process which uses CP (Competitive Programming). This is common at FAG (Lets exclude Netflix since as I've heard they don't do it) and at companies that copycat the process. Of course, there are many other places that don't do it.
We are? What is "competitive programming", and where did it come into this thread? All I see upstream from your comment is a discussion of Google interviews and large tech company interviews generally. I don't recall anything particularly "competitive" when I interviewed at Google (or Facebook, or Microsoft, or...); they just had me solve problems, as usual.
That's the point, CP != CS (Computer Science). It is a separate discipline/subject with its own trivia knowledge & tricks. CP uses Computer Science the same way as e.g. Physics or Biology use Math. And CP problems are used in those interviews. https://en.wikipedia.org/wiki/Competitive_programming#Benefi...
I guess, it is okay telling your interviewer depending on your comfort level, as most never change the question regardless. That said (and as another commentator points out [0]), if an interviewer asks you call out questions you know of from before, then you most certainly should (unless you can sell the bluff...).
When you’re an interviewer and you’ve asked the same question enough (tens of time is plenty, really) you’ve seen it all and it’s really obvious when someone has seen the question before.
People who haven’t seen the question before always stumble somewhere. There’s something they didn’t notice at first and need to account for, their solution is not structured optimally for the follow ups, they iterate over some possible solutions while thinking out loud to eliminate them etc.
It’s honestly not that hard to tell when someone is pretending they haven’t seen it before.
As an interviewer I find it easy to catch candidates who are regurgitating a memorized answer, but catching people who know the answer in-and-out is really hard. I've had the exact same experiences on the interviewing side of the table as well.
I think interviewers tend to overestimate their ability to catch people who have seen the question before and miss on tons of candidates who are good at answering seen questions.
It's actually not all that hard to stumble on the question at predictable spots. My process for solving a question I've seen before and one that is new to me actually doesn't differ all that much: as several clarifying questions at the start to confirm I understand the problem, then write a quick-n'-dirty solution as fast as possible. In this first pass I usually don't pay too much attention to things like bounds, which you wouldn't memorize anyways even if you knew the solution. Then I run a pass to polish the solution up, present it, then think of edge cases and how I'd test the solution. The only real difference is that I very rarely pretend not to have seen a question (I have done it twice, both in situations where it was clear the interviewer was out of questions and running down a list of things from memory, and I just wanted to humor them). You'd think that I'd be more sure of myself when I actually know the answer, but when I don't actually know what I'm doing I will usually come to the answer as I ask those clarifying questions and start stating the facts that usually score "this person at least has the gist of the problem" points, like "oh this is a graph and we're doing something with distance, let's see if Dijkstra is the right thing to apply".
When a programmer goes through a question fast and jumps right into the solution, there is two possibilities. One is candidate already knows the question. Two is candidate is exceptional programmer. If the other part of the interview doesn't match up to this, then we will assume the first case.
I definitely place candidates who are being honest upfront above others. They will be reliable and trustworthy, which is important quality for programmers.
First principle that every interview guide teaches you is to take time to understand the questions. A candidate who has spent months preparing for a FAANG interview will highly unlikely rush into answering a question, even if they have seen the question before.
One reason for folks to jump right into the solution is the interview anxiety - which is both due to lack of practice and trying to compare oneself to folks preparing for months ahead of interviews.
As someone who has done a lot of interviewing from the interviewer side at my current FAANG, it's usually obvious when someone has seen the question before - in the debriefs, the candidates who were honest have it noted out loud to the debrief panel & it makes them look good due to integrity, and those who were dishonest also get it noted as a significant negative.
Perhaps I evaluate differently than a lot of interviewers, but I'm primarily interested in figuring out if a candidate has the traits/skills that I'm looking for in a potential coworker - for us, the traits matter more than the skills even.
I don't think I've ever been asked a question at a FAANG interview that isn't in one of the (legion of) "FAANG interview prep" books/sites that ex-FAANG engineers and interviewees constantly publish and update.
Your criteria here is likely unfairly dinging pretty much everyone who has done significant interview prep.
But significant interview prep hurts the primary job of the interviewer, which is to evaluate whether candidates would work well within the team/org - I don't mind if candidates do preparation or not, but the point isn't to try to find people who game the process, the point is to find out if a candidate is likely to be successful in the role. The problem is oftentimes it turns out that a lot of candidates who focus too heavily on preparation fail to demonstrate the qualities needed to be successful.
I don't give a particularly hard interview, candidates and interviewers who pair with me are all pretty happy with the session & almost always leave relaxed, which is usually targeted towards a certain set of leadership traits (and occasionally I'll do coding screens as well although I've been put on those less in general). My interview feedback also has generally been corroborated by other interviewers in debriefs as well, so it's not like it's in crazy land.
FAANG recruiters literally send the candidates emails that say things like "you should study Cracking the Code Interview beforehand". No, they aren't supposed to, and no, the incentives aren't set up correctly to prevent it (I've received these emails from Google, FB, and Amazon recruiters in the past).
We can't really go around penalizing behaviours that the recruiting side of the house is actively encouraging shrug
It's better to fake having not seen it than failing an unseen before question. No amount of honesty points will save you from failing a question.
Sure if you are good enough that you pass unseen questions regularly go with honesty. If you are not, better to fake it. If FAANGs have problems with candidates knowing the questions maybe they should think of different interviews?
My coding question I give (in the occasion I am called upon to ask one to candidates) typically is not particularly unique, but practical - if someone wants to study for it, it doesn't really give them a notable edge given all the possible branching questions. The most notable tell if people did see the question prior though is if they jump right into talking about the problem without asking qualifying questions, implying that they are quite familiar with the problem & don't need me to qualify it & don't think to verify scenarios with me.
For my non-coding problems, I just create it from scratch depending on the position/needs & spend a bit of time navigating the scenario myself and store the question in my notes.
As to failing a question, failing a single question isn't necessarily a deal breaker in itself - it's showing a pattern of not meeting the bar that is. I may rate someone a 2 out of 4 if they didn't go into sufficient depth in a particular question I asked, but I probably won't stay in the way of hiring them if they did ok otherwise and that failure was just an aberration. Loss of integrity is perception that is likely to sour people on any upside of hiring though, and overcoming that bar is incredibly difficult - if someone is clearly rehearsed on a particular question and is dishonest about it, they're probably not getting a 3 or 4.
I've failed enough interviews to know that failing a question is almost always a failing interview. There are enough candidates that will get the question right and one of them will be hired, not me. Honesty aside.To be clear I am talking about FAANGs or famous startups that get hundreds of candidates per position.
I'm a senior engineer at a FAANG who is close to reaching staff by promotion - I've conducted enough interviews over the past 5 years at my current company to at least understand how my org operates.
How the process for us works is if you are a strong enough candidate, we have no qualms giving multiple offers simultaneously and working out the reqs afterwards, even borrowing against a future one if we have to. How we evaluate is probably also a lot different & more thoughtful than a lot of candidates realize - we're discussing leadership traits, strengths & weaknesses, and skills in our debriefs and what we all observed about the candidate in our sessions. No candidate does perfect in any given session - even candidates who I have given 4s for have slipped up or had negatives observed.
I think you are an outlier then. What leadership traits do you discover when forcing candidates to do BFS on a binary tree or similar questions?
Why would you take someone who said he knows question A and then moved on to bomb question B when you have 30 candidates who solves question A? Ok no one does perfect but bombing a question seriously hinders your prospects. If you see a question you know or semi know you gotta be very silly to say it. People don't drill Leetcode to say oh sorry I saw this one already. Nope. In fact even remembering hundreds of questions and being able to solve
them under stress is hard enough.
Many FAANG candidates are just brilliant and don't really need much preparation to get accepted. But others are normal smart but spent months preparing by going through algorithm questions. It's quite certain they come up with 1-2 question they have seen before and this enables them to pass. If it wasn't the case no one would subscribe to Leetcode....
My org is generally anti-leetcode so I can’t speak for your experiences - the only data structure/algorithm questions you may encounter in an interview with my org is likely practical questions (i.e. problems we have had to solve on the job).
I’m usually not even asking a coding question in my session - I set up a practical/common problem beforehand and we explore the scenario together. I can assure you that many candidates don’t pass my session necessarily, even if they have proven in other sessions to be brilliant coders - I’m not looking for technical brilliance in most of the interviews I give, and neither are the hiring managers I work with. To me, focusing on the coding is most important on the technical screen, not the full panel - once you reach the full panel, your goal is to demonstrate technical leadership, which includes expertise in knowledge, coding competency, focus on UX, and some other areas for more senior roles (conflict management, responsibility, navigating different stakeholders for product/project decisions, etc.).
If all your focus is in is just the coding questions, you’ve likely already set yourself up for failure.
I don’t work at a FAANG but I hope to one day. But I do interview for Data Engineering roles from time to time for my and other teams. I think by saying you’ve seen this problem before it shows honesty. It shows character. Those are points that, at least for me, are positives and I like to hear it. I’ll still go ahead with the question just to make sure but if something typically takes 30 mins and you finish in 10 then I’ll move on to a different question to fill the gap.
If this is the first time, at the same company, it (probably) does not matter, that much.
When I was an interviewer for a (second) technical phone interview at Google, one candidate performed pretty bad at one question. Later that week, or maybe the following week, that very interview was in the pre-HC meeting and one of the other pHC members pointed out that I had repeated a question from the first phone interview. At which point I pointed out that I had not been provided a list of previously asked questions, the candidate had not highlighted it, and the candidate's response was still sub-par.
Not mentioning the repeat counts against character and responsibility.
Not performing better at a repeat question a week later counts against competence.
That was an easy "let's not bring this candidate on-site" decision.
> That doesn't mean I need to understand the entire code base, but if I'm modifying a portion of the code base, I'm loath to touch it till I understand what it's doing.
This is also me. I get flack for spending too long on what seem like "small" fixes but I can't be sure (especially in spaghetti code) until I've dug through it. On a couple occasions, different companies unfortunately, I caught flack for spending too long trying to understand critical finance code. As in, how we were billing every single customer. But it always yielded results, and a week later I'm in a meeting explaining how abc original implementation would have broken xyz, but the scowls never really go away. It's frustrating.
This is why diversity of thought is important in teams. Some people will be the “ship, ship, ship type who make sure that the team is meeting its goals. Some will be detail oriented, may frustrate the first type but will ensure the team is producing quality results. Some people like to be on the bleeding edge and bring you techniques to the team. Others like to clean up tech debt or bring incremental improvements to old solutions. All of these are valuable even if they at times frustrate each other.
These are generalisations. There are likely far more “types” and different people will act as different types in different situations.
Having a mix isn’t sufficient though. The team also needs to find ways to prioritise and manage conflict to get the best approach in place for any given situation.
I'm in the same camp, I don't even understand why programming is done under constant artificial time pressure, why are programmers constantly hounded and stressed for no particular reason? Nobody is really waiting for this task, it's an old well-known bug, or just a new additional feature but OMG HOW LONG, ARE WE DONE YET!??
Edit: same thing in this interview question, the whole point is super quick performance and working under time pressure, and making changes to code bases that you don't have time to understand, why?
You have a very good point there. Deadlines in the real world are often fairly artificial. So, why do we put people under extreme time pressure to perform in interviews?
It depends on the company. Especially in smaller companies, the success of the manager is very much tied to the success of the whole company, but yeah, in bigger enterprises it's often like you said.
I have found that sometimes (!) imposing an artificial timeline can bring the whole team together if done correctly. I've been in situations where projects weren't finished for a long time, everyone just cruised along, because there was no timeline and time to market was "when it is done". And there were always more important things to ship. This is bad for company and everyone involved... Just ship it, then iterate.
This is okay and to be frustrated at it, but it's important to recognize why it's happening.
FAANGs go through an incredible number of candidates with only a few slots (comparatively) and the point is to simply make the bar higher and higher. Just like a pragmatic engineering - you have a 'good enough' candidate set.
For me, the problem is when smaller startups copy the format - expecting candidates to jump through all the same hoops. If a FAANG has 1:100 position to interview candidates ratio, a startup will be lucky to have 1:5 (it's incredibly expensive/time-consuming for a startup to interview).
Running the same test likely means you're rejecting a lot of potentially good candidates who didn't want to go through a 'FAANG' interview.
If any of the candidates they are refusing are capable of doing the daily work at a FANG but can't pass the interview, they're artificially constraining their supply of candidates and increasing the cost they need to pay for developers for absolutely no reasons.
The type of work in a FANG is mostly not that dissimilar to other companies (the exception being teams working with machine learning, performance optimization, dealing with tricky scale). I understand hiring specialists for specialist jobs (and I still wouldn't test them on leetcode; ask domain specific questions).
> It's just ego driven over spending on developers.
It's worth considering that engineering interviews are largely conducted by other engineers, and engineering hiring bars are largely set by senior engineering staff. They have a pretty vested interest in maintaining their own status and (relative) scarcity. Constantly raising the technical interview bar keeps engineers a supply-constrained resource...
They artificially restricting the supply of viable candidates, but this gatekeeping happens whenever engineers want (sometimes subconciously) to limit the competition and protect their high wages.
> they're artificially constraining their supply of candidates and increasing the cost they need to pay for developers for absolutely no reasons.
The incorrect hidden assumption is that by decreasing your own supply the value goes up.
It's well known that turning down candidates or refusing to interview them sends the message that their skill are less valuable. It's HR management 101.
The same goes for "anti-poaching" agreement. If FAANGs don't hire from each other they reduce the number of high-paying employers willing to hire the average FAANG employee, effectively reducing average salaries.
FAANGs go through an incredible number of candidates with only a few slots (comparatively) and the point is to simply make the bar higher and higher.
This doesn't fit the "we can't find candidates, so we need more H1B's" narrative.
Supposedly, there are many open slots, unfilled, so purposefully failing people which can do the job, which meet the minimum requirements, should not be the outcome.
Instead of raising te bar, as you say, due to candidates o'plenty.
> This doesn't fit the "we can't find candidates, so we need more H1B's" narrative.
I disagree. In fact, this validates it.
On the basis of a world-wide talent pool vs a domestic. If you know there are stronger candidates elsewhere then you absolutely want to test away the domestic candidates.
Thinking about it from a hiring manager's perspective. You don't lower your standards just because a specific pool of candidates can't meet your criteria when you can widen the pool - as far as you are able.
If the H1B candidates were tested on a 'easier/weaker' test the I would 100% agree with you, but I'm assuming things equal here. And, ignoring any thing related to domestic vs foreigner workforce, "people taking our jobs" debate.
The bar is higher than needed to fill the positions (several people in the thread say they work in senior positions at a FAANG and wouldn't pass the interview). So good enough candidates are being rejected, so they shouldn't need special visas for people from abroad that are intended for real labour shortages, it would seem.
They wouldn’t pass the interview *without studying. And studying is expected for these interviews. That doesn’t mean the bar is to high. It just means the bar is an indirect metric not testing exact job experience/knowledge.
I think "necessary but not sufficient" is an accurate way of capturing our goals for the interview question. Some really great candidates felt just like you do about the question, and we learned over time how to present it to senior candidates in a way that did not (at least in my experience) trigger that reaction.
That said, there were also plenty of candidates who came to the question guns ablazing with the same attitude, and then failed miserably. I'm not saying that's you -- but rather that it worked out in practice to be a fairly good filter for whether people actually had (enough of) that skillset.
> but rather that it worked out in practice to be a fairly good filter for whether people actually had (enough of) that skillset.
How did you rate your negatives to judge the filter was correct? You know the "Positive" status, you know your true and false positives from the interview question (People you hired that performed well or didn't. Or said another way, the predicted value was positive while true value was positive or negative).
But did you know your true and false negatives (the people you didn't hire that would have been good or bad. Or said another way, the predicted value was negative but true value was positive or negative)? Did you know the candidates you rejected who would have been good at fulfilling the roll at your company (False Negatives)?
To be clear, I don't know the false negatives either and the interview question may be the best litmus test for the behavior desired, minimizing false negatives and maximizing true positives. But I just don't know for sure if the question did work out in practice to be a good filter of people with that skillset without a true negative dataset. The Precision is good, but the recall/sensitivity may not be. Then again, you probably know how the candidates you hired performed before you used the new question and after - which may be an indicator on false negatives.
Also, that does seem like a really fun interview question to answer for me. Kudos.
EDITED: Changed the question at the same time the response came from OP. Original post asked if they hired those who failed the test to know more about the negatives. They did.
In my team I need someone like you to deliver features, not someone that can leetcode. I don't remember the last time I had to check an algorithm in Wikipedia. Probably 2-3 years ago.
Most of the time it will be debugging some arcane crap or adding to an existing codebase, anyway.
Algorithms are funny this way. Most of the time you don't need them, but when you do (and if you don't, you can't tell it when), it is a world of difference.
Still, just knowing they exist and how to formulate the search is usually enough.
I don't know if I would have passed the exact question, but something like this is also in my wheelhouse. The other thing that I value very highly is consistency in any given code base. However things are already done, is how I'll keep doing them, unless there is a good reason to change. And to your point, that reason cannot be I don't understand the code. IMO, too often people want to make big changes w/o understanding (b/c understanding can be hard), and end up missing the reasons why the code was hard to begin with.
> This challenge is particularly well calibrated for an interview because there is only one correct answer: “change bool incr to int opcode” (or anything isomorphic to that). The codebase and problem statement together very clearly imply that there are currently two arithmetic opcodes, and your job is to extend that to three arithmetic opcodes. [...] Cloning even one of these functions is probably suboptimal, but I might spend twenty minutes before realizing that.
I just completed it sans writing tests (took ~30min), considered and discarded that approach, instead duplicating the code-paths (leaving incr/decr untouched and instead making mult_ versions of everything). I did reuse the delta_result_type, though.
Briefly my reasoning was that this would make the fork easier to maintain against hypothetical future upstream changes and keep the logic simpler, while guaranteeing that I didn't miss to implement any particularity - compiler would complain about missing constants or functions as long as I covered entrypoints.
A bit of rule-of-three thinking: If in the future there would be even more arithmetic functions added, maybe it would be prudent to refactor and generalize parts of it? But not at this point.
Curious to hear if you agree with OP that my approach was incorrect or with me that it's equally valid (:
It's equally valid — your reasoning exceeds our expectation for the question. Essentially we're trying to test for whether you can understand at least one correct implementation.
Whether or not your approach is "better" depends on factors that aren't part of the problem statement, so it would be a mistake to assess it as incorrect.
> instead duplicating the code-paths (leaving incr/decr untouched and instead making mult_ versions of everything)
If there already are two arithmetic opcodes, shouldn't both of them already be handled by the same code, only with an operation parameter? In that case all that should be necessary is the handling of that parameter in the place where the operation is actually performed, and you only add one more value of that parameter.
Having said that, I haven't looked and memcached yet how it actually implements these operations. I'll have to do that.
That's the problem: Since only incr and decr exist, a bool is enough to say which of these two operatios should be done. But when you add a third, the bool isn't enough. So you either change the opcode to an int, like the autor, or branch the mul instruction away above like above solution.
1. If you don't do arithmetic on it and it's not a primary key then it's not a number (eg an employee number might be "123456" but it's a string not a number); and
2. It's almost never a boolean; it's an enum.
I've lost count of the number of times I've had to change a boolean to an enum (some of which I created in the first place).
My favourite hack for this is when someone decides to add a third value to a boolean with:
Optional<Boolean> foo
Nope. You're wrong. It's even more hilarious when they add a fourth value:
Yea, I would prefer an enum over an int to choose the operation, but don’t forget that this is C, where enums _are_ ints. Oh well.
Also, that it turns out there is already an enum to extend for the binary protocol, so the blog author reused that instead of making a new one just for the this one function.
I would lean this way - I'm not a software developer by trade but I can cut and paste and look at compiler output.
Changing a bool to an int is the kind of thing that I would worry would have unknown side-effects somewhere, whereas adding new code paths is unlikely to explode the existing.
Upstream would be a consideration, and potentially seeing what kind of a patch they'd accept.
Never worked at a DB business, but feel confident that
> "diving into an unfamiliar area of the code and quickly figuring it out" was very important for us.
Universally applies to all software jobs.
What's I find interesting (based on my own personal history) is not only the ability to solve a task in an unfamiliar code base, but also do so without creating side effects (like say quadrupling the size of a binary since you incremented a poorly named constant that is reused as a size of a memory allocation by multiplying itself with itself).
I dunno, I've worked at a bunch of places where it seems people have gone "I'm not bothering to understand the old code, I'll just replace it with <framework/technology du jour>[1] and GOOD LUCK MAINTAINERS."
[1] Sometimes they write their own frameworks. These are almost always the worst.
I guess it seems like you kinda chose this question by chance, but for new similar types of questions, do you now do them yourself (or get a coworker to try) to get at least an n=1 sample for how long it ought to take? I like to give a 2x margin of time over my / a coworker's time, so anything that takes us over 30 minutes is right out or needs to be simplified when we only get an hour with the candidate. An experience with an intern at my last job who struggled with our large codebase (mostly in the way of being comfortable despite not having or being able to fit everything in working memory all the time) led me to conclude the same as you, i.e. it's an important signal (at least for our company's work) to be able to jump in and effectively work in a somewhat large existing codebase.
I'm amused at the comments suggesting this is too easy (or your own suggesting the same for redis), I think if I tried this I would have filtered almost everyone, at least if they only had an hour (minus time for soft resume questions to ease them into it, and questions they have at the end). So many programmers have never even used grep (and that's not necessarily something to hold against them at least at my last job where the "As hire Bs and Bs hire Cs and Cs hire Ds" transition had already long since occurred). I've made two attempts at the idea though by crafting my own problem in its little framework, the latter attempt I used most involves writing several lines of code (not even involving a loop) into an "INSERT CODE HERE" method body, either in Java or JavaScript, and even that was hard enough to filter some people and still provide a bit of score distribution among those who passed. Still, I think it's the right approach if you have a large or complicated existing codebase, and even in the confines of an hour seems like a better use of time approximating the gold standard "work sample test" than asking a classic algorithm question.
Yes definitely -- although it's pretty crazy how much not being under pressure affects your ability to complete the question in X minutes. I've learned over time that good interviews take a lot of experimentation, and the best thing you can do is test/refine questions in real interviews before you rely on them as a signal to make a hiring call on a candidate.
I wouldn't necessarily consider these questions a substitute to an algorithms question, but rather a way of obtaining a different and important signal. An algorithms question may be a valuable signal too, depending on the nature of the role.
This seems so much more relevant than what FAANG companies are asking apparently (never did an interview with them), ie undergrad algorithms and data structures problems, but in a really tricky way. I wonder why your approach to interview questions isn't the norm.
At the time we debuted the question, it was actually a recruiting advantage for us. People would interview with us and other "big tech" firms, and get excited about the opportunity because we asked this question.
I was asked this question when I interviewed in 2020 and finished in about 35 minutes without any C experience. I can confirm that this question made me very interested in working for MemSQL/Singlestore: I loved the experience. The next step of the interview was to implement a pretty large concurrent program. I was given conflicting/very little information from the recruiter on how to complete the assignment, which I thought was odd given that in my estimation this was far too large for an interview question. I submitted my work after a couple hours of effort in an incomplete state because I decided I didn't really want to work for the company anymore. I never heard back. Really soured me on the company.
I definitely appreciate the intent of it, and think measuring along that axis is incredibly important (and rare). But as someone who continually struggles more to dive into some codebases than my peers, I can say that this task is far too easy and wouldn't help you gauge my ability to do that at all. Something that requires implementing a new feature that isn't a near-mirror of an existing one (ideally one that requires a bit of thought at the design level) would probably be a much better measure, but I don't know what that might be for something like memcached.
I think you'd be surprised how many candidates you'd be able to filter out using a very easy task, despite the great credentials touted in their application.
That said, I don't think this is a VERY easy task, per se. And even if it were, there's achieving the stated objective, and then there's you taking the ability to demonstrate some additional skills. I'm in the Java world, and whenever I requested candidates doing exercises, them building a solution with no unit tests, or at least not voluntarily acknowledging that omission, would be a red flag.
And then I'd ask: let's say this wasn't an exercise, what things would you add before shipping this to production, which would be a great conversation starter around (again) testing (second chance), observability, logging, and so on.
That's a great point! We didn't actually think of this question as much of a "measure" (and I think the blog post alludes to this as well with a "Fizz Buzz" analogy). We thought of it more like a filter — if you couldn't pass, then you _may_ not be adept at navigating through a database-y codebase at the pace that was necessary (at that time) for the work we did.
We had other questions (which I won't spoil) that tested more complex system-design skills, without having to implement them within an existing codebase. We felt this provided clear independent signals (ability to "hack" within a codebase and ability to reason through complex system design) vs. conflating the two.
As the team grew, and we needed folks who were much better at one or the other, this helped us round out the team with folks who excelled at either skill.
It depends on the project, but it actually can take 2 or more hours.
It can start with getting it to compile in the first place. I've spent days on building C code bases, getting the dependencies there, everything in place, then you wait, then there is some error, then you google, then you don't find anything, then you debug a makefile of one of the dependencies, etc etc. Some projects don't compile with packages from apt-get at all. Take most Rust projects for example, unless you are using a rolling release distro, your rustc version is too old.
Then you need to look at the quality of the code base. Sure maybe it's just a simple else if chain and one of them has an add and sets result = a + b; inside. Then you can copy it, modify it for multiplication, done. Trivial!
But maybe it's done via a plugin system and spread over 5 different components, and there's actually two plugins you have to add, one for the repl, one for the internal handling, and there is non trivial communication between them. So you copy the 2 thousand lines of plugin code spread over 8 files and there's a bug. How do you debug it?
You also need to be able to start the thing. Maybe it's not meant to run on developer's laptops but instead is deployed in the cloud and you have to use some unfamiliar tooling to access it?
Now, apparently in this instance the project was easy to compile, modular enough (and also not too modular) for the position needing the change to be identified in quick time, and easy enough to run and test that you could also explain your solution in one hour. But this is not guaranteed.
Any interviewer who designed a question with those kinds of pitfalls in it has done a bad job. This question is good partly because the memcached codebase is itself good. It’s not so abstract that the candidate needs to chew through a whole class hierarchy or plugin system in order to make changes, and yet it is complicated enough that the will have to make important choices, particularly with how to handle a new type of operation.
> But maybe it's done via a plugin system and spread over 5 different components, and there's actually two plugins you have to add, one for the repl, one for the internal handling, and there is non trivial communication between them. So you copy the 2 thousand lines of plugin code spread over 8 files and there's a bug. How do you debug it?
In enterprise Java, there is almost certainly a crazy amount of indirection, both static and compile-time.
A trivial task like "Write an API to retrieve a certain set of records from the database, and return them to the client caller" can involve:
1. Figuring out how the framework calls your API, and where the parameters are in the call. This is specified in some config (yaml/xml/whatever) file somewhere that the framework reads so that it can validate the parameters before it passes them to your code.
2. Figuring out the config-file syntax for specifying the validation criteria for each parameter and for the return value.
3. Determining if the specified list of columns is already encapsulated in an existing class. If it is, you're luck, just instantiate that class with the correct criteria and read the fields into the instance you will return to the framework.
4. If no existing class satisfies your needs (99 times out of a 100, there won't be), you realise by looking at similar existing API code that you need to create a application-logic level class that contains fields with all the values you require.
5. You do #5 above; then you read the existing API callchain again, and see that all other application-logic level classes don't talk to the DB directly anyway. They go through a DB-level class. This is because at some point in the future someone may want to swap out the underlying DB with another DB.
6. The DB-level class is tied to a specific ORM (say, MyBatis). It uses classes generated by MyBatis. You write your DB-level class to use a MyBatis-generated DAOClass.
7. You then dig into the details of the ORM, figure out which config-file (xml/yaml/whatever) to modify, how to write the template that contains the specific parameterised SQL statement (if using SQL), how to specify to the ORM that each column be matched to a specific Java type, and how to convert it if it is some other type.
8. You Aren't Done Yet! For each of those classes you created (the API handler, the Application-level class, the Dao class and maybe the generated classes from the ORM, you need to write unit-tests! The unit tests, in this simple example, will easily be twice as large as the actual new code written/generated.
9. You're done, for now, until integration testing in a pipeline fails. But that's a different issue that will always exist regardless of language or development methodology, so that one gets a pass.
And, believe it or not, the above is actually a simplified version of how it actually gets done. For example, there's going to be more lines of code mocking an instance in the tests than there are lines of code implementing the class itself.
Also, the framework calls your API, but your API probably doesn't directly instantiate the Application-level class - it hands it off to a 'call_handler' type of class that performs a threaded/sleep invocation[1] of an actual 'call' method in your API class.
Also, anything your class may need (instances of other classes that perform non-local stuff, like recording metrics, logging, lookups, etc) will be injected at runtime, and so you cannot simply declare an instance of the (for example) LookUpAddress class - you have to request an instance of that class from some sort of dependency mapper or injector, which will be provided to you when the framework calls into you.
This is normal. There are various reason for every single step above, and large companies (FAANGs, for example) will adhere to every single step (and more, like using classBuilders) because it checks all the boxes except velocity.
A startup, OTOH, had better forgo all of the above; create a monolith with no runtime injection or modification and skip all the unit tests (use more beefed up integration tests, for example). A startup can't afford to spend a day adding a single simple API, when, in the same time, they might be adding 25 new simple APIs.
There's no point in all of the rituals if the startup finds out 5 months in that the product doesn't have market-fit. It's better to determine market-fit in the first month or two.
Having paying customers is more important than having dependency injection, or the ability to switch databases.
[1] When not using Java, the 'call_handler' will use some sort of async/await call to do a proper async request.
The blog post is where that duration came from; I think both the asker and the question author missed the first part of the line from it I quoted.
If you search the rest of the comments here for "three hour" there's quite a few people jumping on that rather than just accepting it as "eh, I don't remember so I'll just throw something out there".
This was a fun distraction to start the morning with, and I like that it is testing a set of directly-useful-on-the-job skills. It feels well calibrated in terms of difficulty.
Personally I feel this question ranks low on a quality scale. Too many moving parts and too many variables.
There’s a million different reasons for why they could bomb it and a million different ways they could succeed at it and none of those reasons might be evident to the reviewer.
I don't know. You seem, like others in the thread, focused on "implement 'mult' as age times parameter value", but the goal of the exercise seems much more around: what's your process of going into a code base, asking the right questions, making some good observations by yourself, and not getting stuck. It'd be interesting to hear from the original interviewer if they even cared about how well the solution ended up working, or if the candidate's ability to demonstrate understanding about the (presumably unfamiliar) code base was more important to them.
You hit the nail on the head. We explicitly did not care about how "well" the solution was written (some solutions were very surgical, others involved a lot of copy/pasta, some folks wrote very detailed comments, etc.), although we did point out and ask you to fix solutions that were incorrect. For example, some candidates implemented "mult" as "add" in a loop, with the row lock acquired and released in each iteration. This solution is vulnerable to a data race if another user concurrently updates the value.
It seems a simple way of building the "mult" solution as a composition of the existing primitive(s), here "add". But agreed, that I would not find the tradeof acceptable either.
Yes - so the safe thing to do is to put the lock outside the loop. Anyone experienced in multithreading will instinctively know this is the correct alternative for any codebase (and it performs better too).
I agree, there's a million things that can go wrong just in installing, setting up and compiling the project. I see losing a lot of good candidates here because they happened to have the wrong version of some library installed..
This type of question definitely hinges on having a pre–prepared build environment. The author of the blog even noted that getting the build environment set up was not part of the test.
False positives: we hired a few folks who could prototype changes but struggled to ship them. This question wasn't solely to blame for it, but it wasn't enough of a signal on someone's ability to _thoroughly_ work within a large codebase.
False negatives: senior candidates who were very used to a particular programming environment (e.g. at Microsoft) and didn't have side projects that kept them up-to-speed with the basics of editing code on the command line over SSH. We did a lot of work over time to setup alternative environments for these candidates.
I'll add to your false negative case. I only write code over ssh in mostly emacs (and sometimes vim).
A couple years ago I was asked to interview via CoderPad and while I appreciate their attempt at having emacs and vim bindings, the fact that they're not actually accurate was worse than using Google Docs.
That is, it's actively worse to be almost your editor of choice than I think it would be to obviously be a "clumsy editor". In the former case, the interviewer perceives you as poor at writing code when you're fighting with a slightly drunk editor, while in the latter (like at a whiteboard) the interviewer adjusts for the environment.
Edit: the interviewers let me retry via projecting my screen while using emacs locally in a terminal. I hope CoderPad has improved it's key bindings since then, but I hope interviewers test out these editing environments before assuming people would be 100% proficient.
For my part, when someone's looking over my shoulder, I get too nervous to rely on basically any editor functionality except cut/copy/paste, undo, and save. I start second-guessing everything. So you may as well just give me Notepad, because I'm gonna look like an idiot in any editor when someone's watching and I'm not already quite comfortable with that person (and the inherent scrutiny/judgement aspect of interviews makes that impossible, in that context).
That's very valid. I was actually only thinking of having someone metaphorically looking over my shoulder, like in a remote interview, but literally having someone in the same room watching is even worse.
To me, purely as far as the using-my-editor part of it goes, it's a lot like having practiced an instrument to an OK level of skill, but only ever playing it alone, then being asked to perform for others, who will make decisions about my future in part based on that performance. So I fall back on playing "Mary Had a Little Lamb" because I'm too worried about screwing up in some stupid-looking way if I attempt something fancier, but that also makes me look like I'm bad at it.
Where in the interview process was this question used? Were there other programming and/or system design questions before and after? Was this a senior-level candidate question?
We experimented quite a bit. Since it was relatively inexpensive to administer (almost anyone could ramp up to conducting it), we were able to test it in a variety of scenarios (even for intern candidates). It ended up being _after_ the initial phone screen, but early into the on-site. There were several other questions that tested other skills (system design, algorithms knowledge, reasoning about concurrency, etc.). Every engineering candidate did the question, regardless of seniority, at least during my tenure.
I recently got this question for an engineering manager position. I got it done in an hour and 20 minute with all test cases presented passing. But I don't particularly think this is a good question for senior level candidates since implementing is mostly copy pasting existing functions and renaming them and adding a new memcached VERB.
I'd guess senior level candidates should solve it without copying and pasting, perhaps make sure test cases are present for the behaviour before modifications are made, and then generalize the existing functions. Or, state why adding more functionality on the server side for every type of arithmetic operation would be unnecessary complexity, and instead add support for any type of atomic arithmetic operation with a transaction/UDF approach.
I'm curious about how you'd evaluate this author's solution. Where it does rank on the rubric you used compared with some of the other ways people approached the problem?
The rubric was more of a "Y/N" for this question. But we'd refer to the author's solution as "great". Messier solutions would involve a lot of copy/pasta and occasionally inefficient or incorrect implementations that tried to repeatedly add.
I can't remember exactly, but I probably administered the question for Arthur when he interviewed (he may remember). I do (fondly) remember working with him.
The author of TFA mentions these solutions usually acknowledge the need for atomicity, and try to achieve it by reusing memcached's locking primitives. I understand the failure is that either they don't get them right or use up all their time without a complete solution.
To be honest: it's how I would have failed this question.
Empathy and kindness is the top, fundamental value I hold professionally, but I am struggling to understand why you would re-assess after a single incident. If a large fraction of candidates showed signs of extreme stress, sure. One candidate, though, isn't enough to determine anything meaningful, in my opinion. As Mack "Bumpy" Freeholder famously said "one thrown laptop is on them, but two thrown laptops is on me!"
Yes it did, at first. It was one of the first times we presented the question, so we didn't have a great signal on its efficacy just yet. However, other red flags emerged during the interview process, and we decided to try the question out a few more times, with a lot more success. And the rest is history.
How did the logistics of this question work out? Did the interviewees work on the problem live, with you (or the interviewer) in the room acting as a sounding board or as some other resource?
We gave candidates a heads up before the interview that they'd be working on a live coding question, and that they should familiarize themselves with editing code over SSH. Some candidates felt uncomfortable with this, and we were able to make alternate arrangements (e.g. setting up VisualStudio locally on a Windows machine).
During the interview, the interviewee worked on the problem live, and the interviewer stayed silent, except for questions. If they didn't come up for air after 30 minutes or so, I'd check in and see if they were stuck, offer help, etc.
Do you let candidates noodle, on their own, for those three hours?
Worst part of programmer interviews are coding as performance art.
I don't code "out loud". I kind of get lost inside my own head. I can explain myself after I figure something out, not during. If I'm supposed to talk, then I'm thinking about talking, not programming.
Obviously: I've never liked pair programming. Rubber ducking has never worked for me.
I remember being taught pair programming in school and thinking "This is awful". I just can't see how it's supposed to actually work.
Rubbing ducking works, but only has a debugging measure. Explaining what your code is actually doing on a line-by-line basis can sus out the root cause of a bug.
I suppose that's the only time pair programming could work for me; by explaining what each line of code does, another programmer could stop me and tell me "No, that's not right".
I hate to say this, but I would have failed the type 2's as well, in just copying and pasting and changing pluses to multiplies.
assuming signed int16 (for simplicity of discussion), 32767 + 1 is far easier to error handle (a simple check of the max_int value) than 10000 * 4, unless you are just blindly allowing the range of values and don't care about overflow/underflow.
I absolutely agree with the premise but for the same reason it's a valuable skill it's also really hard to time box. I'm assuming you picked this particular task but you had already scoped it out and knew memcached didn't have any particular bear traps that a decent programmer would step in and lose 2 days?
I had never looked at memcached's source code prior to thinking of the question, but we did try it out before (and measure the time) before administering it for the first time.
Love questions like this since they're way more practical than some Leet Code algorithm I haven't seen since my sophomore year algorithms text book.
Looks pretty fun too actually! Nice work and I hope you got some good engineers out of it. Any other retired interview questions you're particularly fond of?
How much searching did you need to do before you found a good codebase to work on? Was memcached the only one you used or did you have a larger set of questions that you picked from?
I got super lucky and came up with the question an hour before I first administered the interview. Memcached was the first codebase I went for, because someone had asked me a question involving it a few months ago, and it was conceptually a "simpler" version of MemSQL.
Later on, we tried to diversify the question to other codebases (because it leaked, although less publicly). I remember trying it out on Redis, which was too easy because the codebase is so nice. We never found a great alternative while I was there (through 2016). Although, the team may have found something since.
At my current company, Impira, we have a similar question on a different codebase :)
It was a lot harder (even at MemSQL) to repeat the success of this question. At Impira, it took us about a month of trial and error over 5-6 similar codebases to find a good (and appropriately relevant) question.
You problem setup seems to assume that your interviewee is compeltely unfamiliar with memcached. So how do you expect someone to be able to answer this question if you haven't told them how to add commands to memcached?
[UPDATE] Ah, I see you actually want them to go hack on the memcached code base. That is not at all clear from the problem description. The language of the problem setup is pretty elementary. I think anyone who is actually capable of doing this task would find the description pretty condescending.
The question wasn't administered with the "script" laid out in the blog post. I understand why if you were just reading that, you might get confused. In practice, we had a script that walked the candidate through how to use the memcached client, explained that the command didn't exist, pointed them to the codebase, and instructed them to add the command within the codebase, so that you can issue it through the client.
Ok, I guess you’re thinking of your shell where you can “add a command” by dropping a shell script anywhere on the disk? Or your text editor, where you can add commands by editing some config file? I guess that’s fair. Note however that each of these programs has bent over backwards to enable easy extensibility. The average program does not. This is more like adding a command to sed or grep, which have no extensibility (aside from editing their source code).
This interview will filter for people who happen to already be familiar with the memcached code base, in which the interviewer will believe he has found his genius superhero.
Luckily I did not have to deal with that during my tenure (I left in 2016). However, we had experimented with other changes to memcached as well as similar changes to other codebases. There is no shortage of OSS projects to draw from. However, calibrating something that's doable in about an hour is really tough.
So you hire people who skip over the depth of the locking model and just copy and paste the addition code, but you fail the people who spend time to deeply understand the locking model but can't produce a solution in three hours?
Interesting approach, I bet your codebase is a thing of beauty...
What frustrate me about these interviews is one need to talk on phone and code at the same time. Who the fuck do that in real life.I am so fucked up with this thinking out loud shit.
It's why despite constantly being asked to interview at FANG I don't, as I just don't feel like dealing with that kind of stress. I prefer to stagnante to being obsolete where I am.
I saved up enough that I paid off my house and I opened a bar. I just don't have the heart for the /r/iamverysmart crowd, that always paints themselves into the same corners because "neat and new", or dealing with more political crap where the customer is never a thought in their equation
Don't worry, there are a lot of jobs that don't do this kind of high pressure interviews. Especially outside of web development there are huge markets of people who are really happy to get their issues solved, and don't really care if it takes one or two days.
I think you might be reading something that isn’t there. Look for comments by ankrgyl where he says that some candidates liked to talk about what they were doing as they did it, and others didn’t, and that it wasn’t really important to them one way or the other. Also, I think he said that these interviews were done in person, and that this replaced a question where coding on a whiteboard was expected.
I would probably also pass this question, but I would reject this company for this type of cliffhanger drama which tells me it's an unhealthy working environment.
Maybe he means the part1/part2 split? Perhaps he misunderstood that this is a device of the blog author, who wants to give his audience a chance to try out the question without spoilers, and that it wasn’t part of the interview itself?
At the time I developed the question, I was employed by MemSQL as a full time software engineer. Eventually, I became the VP Eng. I spent 5.5 years there. It was an incredible experience.
This work is very exciting to me for a few reasons:
- HTML is an incredibly rich source of visually structured information, with a semi-structured representation. This is as opposed to PDFs, which are usually fed into models with a "flat" representation (words + bounding boxes). Intuitively, this offers the model a more direct way to learn about nested structure, over an almost unlimited source of unsupervised pre-training data.
- Many projects (e.g. Pix2Struct https://arxiv.org/pdf/2210.03347.pdf, also from Google) operate on pixels, which are expensive (both to render and process in the transformer). Operating on HTML directly means smaller, faster, more efficient models.
- (If open sourced) it will be the first (AFAIK) open pre-trained ready-to-go model for the RPA/automation space (there are several closed projects). They claim they plan to open source the dataset at least, which is very exciting.
I'm particularly excited to extend this and similar (https://arxiv.org/abs/2110.08518) for HTML question answering and web scraping.
Disclaimer: I'm the CEO of Impira, which creates OSS (https://github.com/impira/docquery) and proprietary (http://impira.com/) tools for analyzing business documents. I am not affiliated with this project.