Maybe this lisp syntax intro is a good place to ask veterans a related question.
I find s-expressions quite beautiful and simplistic way to express both logic and data as a tree. On the other hand, top-level evaluation rules are often glossed over, and seem to mess up this elegance. I feel like there is a set of braces missing at top level, for s-expression concept to be complete. I don't really know enough about specific rules or reasoning why they need to be introduced. My mental model of top-level execution is an implicit `(do ...)` block. In rest of clojure code the `do` is a special form that is frowned upon because it is used only when side-effects are necessary.
As it is, the 'module' concept is strictly tied to a file, which feels unnecessary. If top-level was enclosed in braces and evaluated with same rules as the rest of s-expressions, maybe we could move away from concept of files and, for example, store our code in a graph database. This would allow for IDEs that can collapse and expand code tree as needed.
Rich Hickey uses expression 'place oriented programming' as derogatory term (PLOP) for relying on exact memory addresses as places for our data. It would seem to me that same term would apply to our coding practices - we tie code implementation to specific locations inside files on disk. This seems like accidental complexity that introduces mental burden. If location of implementation could be divorced from specific location inside specific source file, we could concentrate on implementation that is always in context where code is actually used.
Is there any decent discussion of such concepts, or some other language that explores such direction? I'm lacking both terminology and in-depth knowledge of code evaluation to express this in more clear way.
In Common Lisp, and similar dialects, the module concept is certainly not tied to a file. In fact, there is no module concept. Packages work at the symbol level. Any file can use symbols from any package.
The concept of a compilation unit is closely tied to that of a file. That's not even remotely the same thing as a module.
Even so, Common Lisp has with-compilation-unit which allows multiple files to be compiled together. They still emit separate files, but for the purpose of diagnosis (warnings) they are considered one unit. In concrete terms, complaining about undefined variables and functions is deferred until the entire compilation unit is processed.
Top-level form processing is deliberately contrived in order to make certain practical situations conform to obvious expectations.
For instance, suppose that we have a top-level form:
(progn (defmacro foo ...) (foo))
this cannot work if the entire form is macro-expanded first, and then evaluated, because then (foo) is undefined.
The eval-when control over the top-level is also very important in the face of compilation.
http://lighttable.com/ might be interesting for you if you've not seen it before... It's looking at some of those concepts. (I've not used it personally, but heard good things)
Definitely avoid lighttable. Unfortunately it is in a zombie like state with tons of open bugs, outdated dependencies and creators who abandoned the project. If you want a simple experience in an "electron" editor, I highly recommend VSCode + the Calva plugin.
The file system is a storage medium that is ubiquitous and requires no install. It's an implementation detail, (and convenient for piggybacking on the Java classpath construction), but semantically Clojure can be represented as a graph where terminal nodes are forms and internal nodes are namespaces. I bet you could save all that to a db if you want, and load it all into a running repl again as well.
You can put all your `ns` forms in one file, one after the other, and if you really wanted that top level s-expression, you could make a macro for that as well (check out the `ns` definition). Still you're stuck with the single file where everything lives, unless you want to move into a graph-db, but why? You'll have to bootstrap from code as text stored somewhere (even in your memory) at some point.
> IDEs that can collapse and expand code tree as needed.
This is "only" a representation problem, because the semantics are already there, you don't have to change the storage medium to buy this behavior.
Thanks for shedding some light on technical side. I'm still conflicted about the mental model.
We should organize our code into deeply nested tree (with exception of cyclical references for recursions), but at top level we use linear and procedural model. In each s-expression there is typically 3 to 5 elements, but at top level there's usually more than dozen definitions. It seems jarring especially when contrasted to elegance of s-expressions.
> This is "only" a representation problem
What if I want to write new function? I still have to redirect it to specific location in specific file.
> We should organize our code into deeply nested tree (with exception of cyclical references for recursions), but at top level we use linear and procedural model
A filesystem with nested directories and files is a tree. The contents of a file are read in linear fashion, for the same reason you'd read an s-expression character by character--because it's simple and consistent.
> In each s-expression there is typically 3 to 5 elements, but at top level there's usually more than dozen definitions. It seems jarring especially when contrasted to elegance of s-expressions
There is no law for the "branching factor" of code. It depends on your style, the problem, etc. It seems like a somewhat superficial elegance you're chasing.
> What if I want to write new function? I still have to redirect it to specific location in specific file.
I'm trying to picture what your ideal alternative is, and it doesn't seem much different. Codebase backed by a graph representation that explicitly encodes dependencies in code loading order? When you add a new sexp you add a new edge and node? That is essentially what we have now, `require`s `refer`s and `declare`s as needed, and the convention that you may depend on everything introduced above your current location in the file.
I agree, hashing content of syntax sub-tree is great idea. Also I like the idea of personalizing the interpreter, like output "I found and typechecked these definitions". Now when things go wrong you have somebody to be angry at.
> As it is, the 'module' concept is strictly tied to a file, which feels unnecessary.
In the case of Clojure specifically; I'm not sure I've ever seen a concept of a 'module'. Clojure has a concept of a 'lib' [0] which is an artifact left over from being implemented on Java. The internal concept to Clojure for dividing up code is the namespace [1] which has no concept or link to files at all.
There are two things going on here. First is the text files of source code on your computer that the build systems or what have you are interacting with. They are organised into files because storing code on a filesystem demands it. The second is once the code is parsed and moves into RAM those concepts fall away fairly quickly and you should be left mainly manipulating namespaces which are only linked to files if programmers want them to be.
If you develop a build system that loads code from a database of some sort then it probably wouldn't have a concept of files, but I don't think anyone has come up with an impressive model for what the advantage is to that. You'd have to reimplement Git and create an Emacs extension to interface with the database; so there would need to be a big benefit to justify the cost.
> My mental model of top-level execution is an implicit `(do ...)` block. In rest of clojure code the `do` is a special form that is frowned upon because it is used only when side-effects are necessary.
Clojure isn't about ideological purity. If it seems that way it is only because Rich Hickey believes strongly that ideologically disciplined code is easier to debug. (do [...]) clauses are only a problem if they are being used in preference to an alternate way that is easier to test and debug.
Here are benefits that I think might justify the cost.
- Easier code browsing: What is the body of this function? What are some optional arguments not used here? Where else is this function called from?
- When teaching, ability to chose depth of visible code means more focused learning. For example, start from overview algorithm with function code collapsed, and then expand one by one to dive into each.
- Reduced mental overhead due by eliminating concept of 'physical' location of code. No refactoring to extract common code into separate module, just reference same subtree from multiple locations.
- Persistent code structures would allow for specializing a function to work differently in one place, while other callers still point to older implementation (until you decide that new implementation is superior).
- A more strict tree-based approach to code would bring in many more visualization and manipulation techniques. It would be sort of seamless blend between text-based and visual programming.
Regarding Git capabilities, I feel that Datomic model would provide much of same benefits. It would even allow for complete history of development without manual effort of checking in. Of course, talk is cheap and these are just random ideas.
I'm not really complaining about Clojure. Its success is mostly due to pragmatic choices. I'm more curious has anyone put serious thought into purely tree-based lisp, and decided that it is somehow worse than top-level + s-expressions?
Are you suggesting an image based model like smalltalk?
>Easier code browsing
At the cost of throwing away all the standard tools and starting from scratch. No files means no grep, no emacs, no nothing until a replacement, plugin, or work alike is built.
I absolutely agree and think the same thing about Python - I dislike the module=file relation because I often want to split a module into multiple files and I also dream about storing code in a graph database often.
There's nothing stopping you from splitting a module across multiple files in Python
Yes, it will result in sub-modules
If you hate addressing code in the sub-modules via the sub-module path you can import them in the __init__.py of the parent module
Personally I dislike languages which add another layer of (IMHO worthless) complexity by decoupling the owning module of the code from the file path, so you have now have to resolve that before you can understand it.
Well, that actually needs to be transparent to be useful for the way I want to use it. I can't even understand why would one need sub-modules so far. The actual reason is to have different members (i.e. every method) or sets of members of a same class to be implemented in separate files. I just hate to have long code files and to have code above and below a function declaration.
Partial classes in C# are an example of how can that be implemented the way I like it. The only problem with the C# way is it adds an additional indentation level. I would prefer the module (or even the class) to be declared in the beginning of the file without contributing to indentation.
There is no such limitation in Clojure, though it is that way by convention. You can put all namespaces in one file, if that's how you want to roll. You can also split a module into multiple files.
Files become namespaces (clojure term for modules) in the Clojure default behaviour, but namespaces can also be manipulated from code and don't need to have associated files. See https://clojure.org/reference/namespaces - you can create and remove namespaces, introspect them, work with ns aliases, etc. So namespaces are not strictly tied to files in Clojure.
re: ""As it is, the 'module' concept is strictly tied to a file, which feels unnecessary.""
In Common Lisp you get to decide which package each source file contributes to. For utilities that are (or will be) used for multiple projects, I use a different package for each library and for small libraries it is likely that all the code will be in one file (plus a project and 'asd' file, details for which aren't important here).
For non-library projects, like a complete application, I usually use many source files that all contribute to the same package.
The point is that a Lisp like Common Lisp is almost infinitely flexible, but it really helps if you have a standard way that you like to set up your projects.
I used to, back in the 1980s, use a hardware Lisp Machine where all my work was in one environment. I feel like I have something similar now by maintaining a large mono git repo for all of my work and research in Common Lisp. The Quicklisp package manager keeps my own packages and all 3rd party packages available to me in a transparent way.
I love my own working environment, but if you go down the Lisp path, you should get an environment set up that is right for you.
In Common Lisp, you get to decide which package each individual token of source code contributes to.
If a package abc exists, we can write abc::def. Now the abc::def symbol exists, if it did not before: the symbol named "DEF" in the package named "ABC". If we write (defun abc::def ...) we are binding that symbol to a function. Informally speaking, we are defining a function def in pakage abc.
Yes, much of this space is fairly well trod, though completely agree the threads are hard to find. I'll try to point some out:
1. Clojure, top level forms, trees, and evaluation
Agree in part about top level evaluation rules in Clojure. It does seem, for instance, like the ns form should enclose the rest of the forms that comprise that namespace, rather than essentially doing something that is unusual for Clojure- silently changing what seems to be a global context.
When one digs a little deeper, however, there is a logic to those semantics. The main reason comes from the problem that the compiler faces in having to reconcile the use of a program thing- a symbol or name or variable or whatever-one-calls-the-named-elements that are used in a program- with the defining of that thing.
There are basically two approaches to this problem. The compiler can read all the source code, find all the definitions, then reread all the source code, and match the uses to those definitions- and only then inform the programmer if there is some problem where a use doesn't match or doesn't have a definition. Even with modern computers, for large programs, this is too expensive and time consuming.
What many compilers do instead is to require that any used names are defined "first". Clojure does this- it reads files from top to bottom, and it requires that any used names are defined earlier in the file.
This notion of earlier- this notion that things defined in a program have an ordering to them, not just in their execution but also in their composition- this is deep and pervasive, and puts the lie in the idea that a program is just a big tree.
One branch off this tree, so to speak, where the ordering in a file doesn't correspond to the compositional or execution ordering, is in the functional programming concept of monads.
2. Alternatives to file storage for program modules
It is a pretty old idea that files are a poor way of storing source code (sorry). There is a long train of work that Wikipedia summarizes poorly with almost no references under Source Code In Database: https://en.wikipedia.org/wiki/Source_Code_in_Database. The idea here is that persisting code in a data structure and providing more "intuitive" tools for editing is better than requiring humans to work in files.
(Microsoft even tried in the 1990s to roll out a version of Windows that used a database for pervasive structured storage, rather than a file system. This was a failure, and a lot has been written about it- google WinFS).
Plain text files have a lot of underappreciated ergonomic properties. Their use doesn't keep tools from utilizing clever data structures to assist in the management and authoring of code in files. The SCID work has ultimately found its way into the cool incremental helpers and structural editors that most IDEs use now (Cursive/Paredit for Clojure: https://cursive-ide.com/userguide/paredit.html)
Forgoing the text editing paradigm altogether takes you into the world of visual programming editors, which also have a long history. An influential player in the space from the early rise of personal computers was a product called ProGraph. This technique is also now pervasive in tools like Scratch, but also in big data where flow graphs for processing immense streams of data are often constructed using visual tools, for instance, Nifi.
3. Literate programming
Another thread is Literate Programming, originally invented by Don Knuth. The idea is that the most important consumers of a program are other humans, not the computer, so one should author using tools that create both an artifact that a human can read, with both prose and code interspersed- as well as the code itself for a compiler to consume. But the combined prose/code artifact is a better way for communicating to other humans about the semantics of a program, than just the code.
This is a particular endearing thread, and the tool called Marginalia in the Clojure world provides something of the experience that Knuth intended.
4. Version control and program semantics
Yet another relevant thread is in version control, where the inability of files to keep the history of a programming authoring process is addressed. Early in Clojure's life Rich Hickey created a tool called Codeq: https://github.com/Datomic/codeq that loaded a git repo- essentially a graph of changes to program files- into a Datomic database- where Datomic can be seen as a graph db.
There has been some more recent work to be able to run semantic queries on those graphs. This is immensely useful for looking for patterns of code that may have security problems. A company doing a lot of work in this space is called Source(d).
Another set of tools for mining version control comes from a company called Empear, started by a programmer named Adam Tornhill. His work- originally in Clojure- looks at things like patterns of paired changes across files. Cases where the same sections of code in the same files are changed in the same commits demonstrate high "coupling" and are a "code smell."
==
All of this is really about building and maintaining a semantic model from the syntactic artifacts, which is what I read you as being ultimately interested in. There's a lot more, but that's all I have time for now. Hope that's helpful.
I somehow missed this post and I'm very lucky to have checked again. This is extremely helpful and taught me more than all my efforts to dig up relevant ideas.
You're very kind. I have spent a fair amount of time in this space and am happy to be able to share. My email is in my profile, happy to continue the conversation if helpful. Cheers!
Clojure is the most productive(and beautiful) language in existence(personal preference) as long as you don't have to leave its world.
I stop using it because the lack of Clojure libraries pushed me to use Java libraries, and that was a life-sucking experience. Java is fine as language but you cannot say the same about the APIs of many of its libraries. I also started sensing that new open source libraries are not being created in Java which worsens the story for Clojure.
The great thing about Clojure is that it's not tied to the JVM. JVM is just one of the hosts.
So I mostly do frontend development at the time and I know Clojure since before. Because of those two variables, I've chosen to use ClojureScript so now I 99% of my professional time write only ClojureScript, with the remaining time being 1% JS when needed. Otherwise the only time I see JS is when I have to understand how a library works.
So I agree with you on the Java side, I'm not a Java developer and I want to stay far away from it, but I love Clojure and I need to be close to it, any other language feels inefficient and slows me down. But there are other hosts you can use, if needed.
And since there is always someone who jumps in saying "Common Lisp is a true lisp while Clojure is not", read https://en.wikipedia.org/wiki/No_true_Scotsman so we can have a real discussion about the difference ;)
"No True Scotsman" is about moving the goalposts regarding to the quality or suitability of something. We wouldn't reasonably say that a Dane isn't a True Scotsman, right?
A piece of crap unsuitable for production use could be a Lisp. Lisp 1.5 is Lisp; I wouldn't use it today. Lisp or not is not about quality, but semantics. Clojure lacks or changes numerous semantics which define Lisp. That's not statement about quality, just otherness.
Clojure doesn't call itself a Lisp; though the main website claims it is a Lisp dialect. Dialects are defined by mutual intelligibility. An Osaka man speaks a dialect which is understood in Tokyo. The mutual intelligibility between Lisp dialects and Clojure does not extend very far beyond (+ 2 2).
The other way you have to leave the Clojure ecosystem is when you have to communicate across the network or persist to disk. Writing everything fits in memory data analysis in Clojure is some of the cleanest and most fun programming I've ever done.
But once you want to spread between multiple computers or operate on data larger than what comfortably fits in memory, you end up coding just like you would in C#.
I started working on some versions of persistent data structures that serialize to a KV store, which lets you send a hash map to another machine, have save that machine add a key, then send it back. It seemed to work pretty well, but there's a ~10x performance penalty over Clojure's inbuilt data types (which are themselves not lighting the world on fire speed wise).
That's a pretty harsh price to pay - you can use 100 machines to simulate single address space Clojure on a 10x machine. Awesome, but likely impractical. If you need the persistence between program runs or would like to run incremental computation on large data I could see it working well, though.
When I was using HoneySQL[0] speaking to PostgreSQL over JDBC to deal with persisted data, I did not feel non-idiomatic. When I used Kafka and Onyx[1] to process large dataflows, I did not feel non-idiomatic. Granted my working set was only a few 10s of TB, but there wasn't much pain involved. I tended to use Nippy[1] for serialization.
I didn't use special data structures (except when the problem called for it), with a clear separation using nippy serialization (in place of pr-str/read-string) at the edges of my app.
But did you really get much of the benefit of using Clojure? If you store your data in an SQL database for instance, that layer provides the equivalent of what Clojure's STM can do within the process. You open a transaction, run some queries, perform some logic, then update the database. Clojure's persistent data structures don't give you extra leverage here - you could just as well write the logic with mutable datastructures in C# (that you throw away afterwards).
This is no accident, as Clojure's STM was built around the premise of emulating the MVCC provided by databases to provide ACI (minus the D) guarantees to program against. If you're already getting those guarantees elsewhere, doubling up does no good.
Similarly, code that operates on streams is generally not where you get bogged down by state and multithreading in something like C#.
I say this as somebody that believes Clojure is a powerful addition to the toolbox. I even use refs in production code...
I say you still do get a benefit from using because of the repl-driven development process. I miss that instantaneous feedback and the feeling that you are in dialogue with your program whenever I develop in other languages.
Add to that the fact that the language shepards you into using immutable datastructures and writing pure code, I still think Clojure is a significant advantage when writing distributed applications.
Yes, we leveraged Clojure a lot. PostgreSQL has far more advanced indexing and querying capabilities, particularly around too big to fit into RAM data, than what I'd want to waste time on building myself (if my team could even do it). The D in ACID is pretty important btw, and being able to offload the replication/backups/failover to AWS (using RDS) was a huge win operationally.
Clojure's HoneySQL is the best way to programatically interact with a database I have found, and it's not particularly close. It provides nice data structures to work with (that can even be schemed or speced) and combined with clojure makes writing complicated logic a breeze. It is easily extensible to new SQL functions, clauses, and operators, even vendor-specific stuff (we used a lot of PostgreSQL specific features around function-based indexing). LINQ doesn't compare to the flexibility here provided by data structures and a potentially just in time SQL compiler. No ORM does. As an example, the user might be submitting ajax requests to my backend, filtering some data by adding more filters or other clauses to my query. What I store in the session or in the database is fundamentally just (pr-str my-query). When the user comes back and wants to continue modifying it, I (read-string my-query-str) from my store. Then I just keep assoc'ing, filtering, and manipulating the data. If they decide this is a nice query, I just save it. It can become an alert, a report, etc. The only machinery is basic clojure data structure manipulation, basic print/reading, and the power of compiling data structures to SQL.
As far as operating on streaming data via Kafka (raw) or with Kafka + Onyx, I suspect you've never actually done this in Clojure. It's a breeze, quick to write and performant. I've yet to meet someone who has worked with Onyx who wasn't blown away by how simple it is to use and how quickly you can evolve very complicated data flow graphs. It provides reasonably low latency, customizable batching, and I can still write straight-forward and simple Clojure. You don't really deal with state or multithreading when dealing with Onyx (Onyx handles that for you and you can declare the parallelism behavior you're looking for). When I consumed Kafka directly, I didn't have many problems that get bogged down by state. We'd keep some local state, but the system was designed to either run in batch to consume some entire time interval or to be able to be kill -9'd at any point, in which case the data was designed with idempotence in mind (such that we can handle/detect processing data more than once).
I selected and championed Clojure because it was a dynamic language (very similar to python or ruby) that had access to battle-tested JVM libraries. I selected it over python and ruby, because it made choices that resulted in simpler programs that have far better locality when I or a co-worker must understand and modify a program in the future. The emphasis on immutable data and functional programming allows nice straight line programming where all I need to know to reason about the code is in the function arguments. There aren't as many surprises. The reason I've stayed with Clojure is the community focus on simplicity and the stability of my code. The code I wrote 8-9 years ago works just fine today. That's not a huge brag (cough Fortran cough Common Lisp), but it's refreshing when I'm looking at how quickly evolving languages like Rust and Python.
So this got 3 downvotes, and I don't care about the karma, but I'd be quite curious to hear about opposing experiences scaling up Clojure programs since there seems to be some disagreement.
If you started to struggle with fitting everything in one node due to CPU or memory constraints, but still felt like you were writing Clojure in a half way idiomatic style when moving to a cluster, I'd love to hear from you.
~10X a performance penalty doesn't seem that bad. In C a pointer derefference will beat a web service call by more than 10X. Also why a handjammed KV store? EDN and other(faster) serialization libraries have been around for quite a while.
I used nippy originally, then later Java serialization libraries once it became clear optimization was required. I tried using both Redis and RocksDB. The happy case had all but the last layer of nodes sitting in cache in memory (as long as your memory is more than 1/32 of the size of your disk).
I think I could do better starting from scratch given what I've learned.
Clojure is a pragmatic language, and I think if you just reject the immense value of being on the JVM and try to treat it as something pure and unsullied then you're missing a lot of its motivation.
there are other vms that would provide the same level of value, though, for clojure. the fact that it runs on V8 via ClojureScript is proof of that.
personally, I believe they should have used BEAM originally and stayed as far away from the jvm as they could, but that is my own opinion, and everyone has their own opinions.
I don't think there's a world where Rich Hickey would have had it running on BEAM. From what I've read, a large part of why he originally chose the JVM is because companies he wrote software for would be okay with something that runs on the JVM, but not something like Common Lisp. And surely not BEAM.
At its core, Clojure attempts to be practical. Running on the JVM was a practical decision.
Oh, I understand the reasoning behind the choice. Doesn't change my mind in the slightest that BEAM would have been a better underlying VM for Clojure.
Yeah good point I find myself often having to use a Clojure shim over a not great Java library out there. It's painful at first but it can be really great once you've tamed that beast.
If we're talking about Lisps, I've found Racket to be considerably more expressive, and incidentally, more beautiful as well. If we're talking about beauty of any language, I think many would agree that APL takes the cake.
I don't see anything profound here, except perhaps that the language will gladly accept the user's blatant type error and just explode at runtime. For the most part this article is fetishising over basic syntax.
Surely the compelling parts of Clojure go somewhat deeper than basic syntax. And if this is another tired discussion about legibility of syntax between languages, I have to wonder why people are still so fascinated by this?
Between the following two representations of the same function…
(defn f [x] (+ (* 3 x) 1))
…and…
f x = 3 * x + 1
…I see very little difference in legibility. Sure, one is visually noisier (and incidentally, the noisier, less clean(!!!) notation is Robert C. Martin's favourite) but if you can afford to navel-gaze this much on syntax I'd question why you don't have anything more important to do. Of course I recognise that I must also not have much better to do since I read the article and I'm now commenting on it.
It's just weird to me that one would fetishise syntax, and then describe a language sporting a less visually-noisy syntax as "The Dark Path".
Edit: Apparently I need to state for the record that I spent a few years writing Clojure professionally. I'm not unfamiliar — I just haven't drowned in kool-aid.
[borderline trolling, but...] In general, the syntax doesn't matter much. Most programming language discussions focus on the syntax, but in the long term and in larger applications that's not the important part.
From my personal opinion, based on using Clojure for about ten years, and maintaining a SaaS business based on it for the last four: what really matters are many little features taken together. The sequence abstractions and functions, great data structures and their interop, defined serialization for data structures, core.async, transducers, spec — each of those can be picked apart as "nothing profound" by itself. And yet together they form a tool which removes most friction from programming and lets me focus on the problem domain.
I've programmed Lisp code for several years and personally I do find Lisp code harder to read, especially when doing arithmetic.
The direct benefit of Lisp syntax is that everything is consistent. Arithmetic operators have the same syntax as regular functions. Because all Lisp code has a consistent structure, that makes it easy to do some really neat things. For example, it isn't hard to write a program that takes Lisp code as input and processes it in some way.
This is the reason why you see so many people write a toy Lisp interpreter. It's really easy to write code to evaluate Lisp code because all Lisp code has an identical structure.
As a Lisp programmer, this benefit emerges in the form of macros. Macros run at compile time, take some Lisp code as input, and spit out new Lisp code as output. The new lisp code is then substituted for the old one.You can think of macros as a very small user-defined compiler. These let you build all sorts of language constructs that you couldn't build in other programming languages. For example, you can add pattern matching to a Lisp with macros. The macro would take a list of patterns and corresponding expressions as input and spit out some code that checks for the patterns and runs the corresponding expressions if the pattern matches.
> These let you build all sorts of language constructs that you couldn't build in other programming languages.
Many languages (e.g. Rust[0]) let you write procedural macros, even if their grammar is much more complex than that of a Lisp.
Is there some technical advantage to Lisp macros over macro processors in other languages (ignoring string-based ones like the CPP), or do all the benefits lie in the massively improved ergonomics of writing macros?
(Note that Rust also has simpler hygienic macros, but they're not expressive enough for many things.)
You've pretty much covered it; lisps are better at macros than any language with complicated syntax.
Most of the flow control in a lisp (for, while, if, etc, etc) can be implemented with macros. It isn't often a program needs a new form of flow control but when you need it it is there as an option and a lisp will do it better than the competition. If Rust can do threading macros then it probably has enough of the power that it isn't missing out. Still going to be a bit cumbersome.
Lisps give you the option that flow control can be implemented in a library. None of the old-school popular languages manage that.
I've been playing around with Clojure, having a lot of fun, and enjoying the community around the language. I love it, I absolutely love it, and I will come around to saying some glowing things about it at the end of this comment.
But first...
When I saw Bob Martin writing about it, it was a bit of a, "Damn.. Spaceballs!" moment. And the post itself didn't exactly light me on fire. So I don't know if this is the greatest introduction to the language.
And to be honest, the syntax is hard to read. It imposes more structure on everything you do: certain kinds of operations require more nesting, more context. Nothing just sits on the page with a standard layout and structure: it's always governed by these parantheses that are running around all willy nilly. And that's just harder to read.
BUT. There is one huge, huge, huge, huge win from it:
Every single thing in the language works the same way. Everything! It's all just expressions.
That might seem like no big deal. But as a programmer, it means that if I don't understand why a piece of my program isn't working, all I have to do is put my cursor right after the expression I want to understand and type C-c C-e to evaluate that subexpression.
So if
(defn f [x] (+ (* 3 x) 1))
...isn't working as I expect, I don't have to run (f 5). I can bind x to 5 and evaluate (* 3 x). And the same applies to any larger program I write.
There are other things that I love about Clojure, but this has certainly been a breath of fresh air coming from other languages that have more ceremony. It is liberating to be able to isolate and assess a portion of code without having to refactor it into a standalone method. I can move much faster, and have a lot more fun.
Quite agree on your syntax comments. The morphology of the syntax itself (as opposed to its consistency, homoiconicity, etc) is uninteresting. Who even notices syntax after a while using a programming language? I think I could edit any of the languages I know in any of a number of surface representations - syntax as-is, some kind of graphical AS tree, graphical nested bubbles, etc etc.
I honestly think the syntax fetish is just because people want to write something (we all like the sound of our own voices/keyboards), and if they don't know anything about a language, emoting on surface syntax is about all they've got.
Woah strawman. There's no fetishization going on here, it's a micro-tutorial.
It's the opposite of profound - and decidedly so.
You're focusing on something called syntax and legibility, not this article -- which is why you see no diff. between the two expressions.
Of course on a superficial level the difference is nominal.
Why the first one is more powerful than the second goes much deeper. The first expression is both a function definition and a first-class list. The latter is pure syntax.
The consequences of this go deep - but if you're looking for something profound at the surface of things then carry on shaving your yaks.
I think you've missed the point; this is a blog post by Uncle Bob.
Everything he says is profound, right?
(No, it's not, but to be fair the parent comment also about people assuming just because someone famous says something, it is necessarily also profound).
Also, unrelated...
> but if you're looking for something profound at the surface of things then carry on shaving your yaks.
Doesn't 'yak shaving' refer to the same thing as 'going down a rabbit hole', 'pulling a piece of yarn', etc?
Ie. Looking at something that looks simple and then finding it much deeper and more complicated than you expected?
Isn't that... exactly what you're talking about with the syntax?
Like, you're literally telling the parent to keep pulling at the syntax comparisons, like oh hey, bet you'll find nothing interesting here... then literally also telling them that it is a rabbit hole when you look at how functions are lists?
That makes absolutely no sense to me when you use it in that context.
I wouldn't give much importance to his Clojure articles, they are all shallow and superficial. My only conclusion is that he writes them primarily for marketing purposes.
That in and of itself isn't particularly useful or meaningful. There are languages with even less syntax than Clojure, but that doesn't necessarily make them more practical or ergonomic.
It matters if you care about consistency. After learning the simplicity of lisp, all other use of special symbols for infix syntax strikes me as arbitrary and fiddly. Arbitrary choices should be left to library (function and macro) developers, I appreciate that the core language is simple, extensible, and consistent.
And yet anything more substantial than a single-line function clearly demonstrates the breadth of syntax required in Haskell comparatively, e.g. https://gist.github.com/honza/5897359
Those two snippets are not equivalent. If you were trying to make an argument in good faith, you would have read the code properly and seen that for yourself, and/or read the comments below the snippets which point this out and link to a further explanation.
I don't think it's that it's "criticism of the church" that people get touchy about. It's just that people tend to focus on the syntax in isolation and not acknowledge the tradeoffs that are involved with different syntaxes.
A lot of lispers are used to people complaining about the parens and syntax but not acknowledging the consistency, the macros, the language-design-as-a-library, the structural editing, and the easy repl integration story that you get in return. And those are the features that lispers are really excited about, and it's frustrating when people don't acknowledge that you give up all of those benefits when you choose algol-like syntax.
Now, I'm not saying that you're one of those people who doesn't understand that, but if you've met lispers who "get touchy" about it that's probably why.
And then those same Lispers will whole sale write off procedural Algol like languages as being "lesser", despite everything that that has been accomplished with languages like C.
I'm certain most of us know who the author of this article, Robert "Uncle Bob" Martin, is, but for those that don't:
Robert Martin is one of the original signers of the Agile Manifesto, which is the outline upon which a great deal of modern software development is founded, and helped set the tone for the Software Craftsmanship movement. Martin has a large amount of talks (that are, admittedly, all quite similar to one another) and blog posts (on the site linked) where he discusses his views as to where the industry should go/should continue to go, with a heavy emphasis on professionalism in programmers and programming teams.
There are a hardcore group of developers that love functional languages. Whenever I look it just looks pointlessly complicated.
Maybe its because I'm not smart enough? Given that 99% of software is not written in these languages perhaps you have to have a special gene to figure this stuff out. Or maybe its just people trying to be different or trying out things on the fringe.
I'm not sure what the answer is or if I should try harder.
> Whenever I look it just looks pointlessly complicated.
I would say the exact same thing about the imperative style, and I have lived in both the Imperative/OO world and the Declarative/FP world.
> Given that 99% of software is not written in these languages
citation/elaboration required, and not just because SQL and similar tools are declarative like another comment mentioned: a great majority of imperative languages have begun to include/adopt functional programming tools and ideas (map, reduce/fold, lambdas)... perhaps it's because they are extremely useful concepts.
I find it the opposite. Lisp is much easier to write than, say Go. I think the problem is if you're used to the complexity of writing a Spring Boot app (Java) this simplicity of writing in Clojure will confuse you: you think it can't be that simple when in fact it is.
I used to freak out when I'd find the library in Clojure I wanted to use but it had no documentation. Eventually I just started looking at the implementation and I'd see it's like the entire library is 35 lines of code! Or like 102 lines of code! You don't need documentation for something that small.
I've never seen this before in other languages. If you think it's too complicated, you might just be overthinking it.
The need for documentation for a library is not linked to the amount of code it has. In fact, it’s rather sad that the programmer didn’t bother to document such a small thing. It most certainly wouldn’t have been a big task.
Could you share the 35 line library? I’m curious. I recently asked the Common Lisp subreddit why so many of their libraries lack extensive documentation and most agreed it is a wart on the language.
In this case, I think it's meaningful. If you take https://github.com/scotthaleen/clojure-spring-cloud as a reasonable example of what that would look like, it's (subjectively) ugly as sin. Because you're operating outside idiomatic clojure, there's a lot of incidental complexity and "why would you do it that way?" moments that you wouldn't get with any of the clojure-native, idiomatic web libraries. Yes, you could wrap a lot of it behind more idiomatic facades, but that's still added work you just don't want to have to do.
Clojure doesn't really have frameworks, though. Every once in a while a framework-like-thing emerges but it quickly decomposes into a few libraries. The level of composition tends to be functions not objects, so there aren't any "taxonomy traps" associated with monolithic things in the OO world.
It took me a long time to get used to functional programming.
For me a big problem was how dense the code can be. I was relying on the verbosity of imperative languages to act as a sort of whitespace.
Another thing is pattern recognition when reading code. I had been using imperative languages for so long that I could mentally gloss over chunks based upon the shape of the code. But functional languages tend to have different shapes from imperative ones so it all looked unfamiliar to me and I got overwhelmed.
Eventually I was able to get passed this stuff and functional languages make a lot more sense to me now. Now I prefer them despite working in Java for my day job.
Whenever I look it just looks pointlessly complicated
Well naturally if all you do is 'look'. Is that how you learnt whatever programming language or style you use now? You just glanced at some code and the necessary cognitive structures leapt with ease into your mind?
perhaps you have to have a special gene to figure this stuff out
Well I'm learning Clojure right now. It doesn't seem complicated to me (though it is unfamiliar enough that I have to think quite hard about how to do things, which strikes me as a good thing). And, for reference, I'd generally place myself in the lower 30% or so of developers in terms of natural ability.
It sounds to me like you're just not very curious about what functional approaches have to offer. Which should be perfectly fine. But your comment sounds like a complaint.
I went thru a magnetic pole swap, if you will. I went from growing up as a kid toying in C , later professional development in c++ and java. In college, had a small taste of functional style and found it difficult for my brain to grok. Later in life I took up clojure, and things started to click. Despite years and years imperative development, my personal preference is for the functional style.
I do find languages like Scala and Haskell slightly appealing, but I find many of the deep concepts and rigid type are orthogonal to the problem I am dealing with at hand. I realize for some working thru the type system helps them solve the problem at hand, but for me, it often seems to get in my way.
Not sure what languages you are referring to with the 99% comment, but declarative programming is extremely well established, and it marries well with functional programming. See: SQL, Excel, regular expressions, HTML. Functional programming is a subset of declarative programming, and there are a wealth of useful applications.
Given that C++, Java, C#, VB.NET, JavaScript now have functional programming. features, and Smalltalk always had LINQ like capabilities, there is plenty of software written in functional like languages.
And then there is the whole lot of Scala, Kotlin, Swift, TypeScript, before we even delve that Haskell and OCaml are used by Jane Street, Intel, Microsoft and Facebook, just to throw a couple of names around.
I have nothing against Clojure, but I agree with Donald Knuth who said: "Programs are meant to be read by humans and only incidentally for computers to execute." [1].
I could be wrong, but I don't find Clojure to be "simple" for humans to read (or write). Here is an example from the blog post:
So let’s write a simple one. Let’s write the factorial function.
(defn fac [x] (if (= x 1) 1 (* x (fac (dec x)))))
Note the "simple one" here. I don't know for others, but this is not simple for me as a human to read, understand and reason about. For instance:
if (= x 1)
As a human, I read this like "if equals x one", which does not translate to my natural language where I would say "if x equals one". So I need an additional mental effort to do the translation. This is not the case in other languages where "if x equals one" would be written like "if x = 1" or "if (x == 1)". If the gap between the natural language and programming language is big, it is difficult for a human to use that programming language. And for Clojure, this gap is big IMO.
(I (wanted) (to learn ((some)) Clojure in the past) but [quickly] ([(realized it was (not) for me)])) [2].
Yes of course, that's incorrect and provocative in purpose. The point is not about the syntax, but about expressiveness: A programming language should make it easy for a human to express (write) his intentions in code and for other humans to interpret (read) those intensions in a natural way. In other words, the gap between the developer's natural language and his/her programming language should be minimal (and this is regardless of the developer's natural language, be it English or whatever).
In other words, the gap between the developer's natural language and his/her programming language should be minimal
I'm not sure that follows from the original claim at all.
Mathematics has plenty of problems in when it comes to obscure terminology and regrettable notations, yet still vast numbers of people prefer the conciseness and precision of mathematics to using natural language when they want to discuss concepts in science, engineering, programming, etc.
At some point, if you're going to work in a technical field, you're going to need terminology and notation to match. Newbies to the field need to learn those before they can understand what is being said "naturally". The alternative is to attempt to dumb everything down to the point where newbies can understand it straight away but, assuming it is even possible, this risks losing a lot of effectiveness in communications between everyone who does have more experience.
Writing code is different from writing mathematical equations. Developers (humans) prefer meaningful variable names, method names, expressions, etc. Comparing this to maths where conciseness is key is not the best analogy IMO. And it is not about experience (btw, I have no problem being one of these newbies who "need to learn those before they can understand what is being said "naturally""), it is really about expressiveness to reduce the mental effort to match things (what I have in mind and what I see in code). That's why there are many attempts [1] to create programming languages that are as close as possible to people's natural languages.
My team has been hiring and training coop students for years, and it takes on average a week or two for students to become productive with Clojure. Furthermore, students with less programming experience actually have an easier time picking it up. So, I completely reject the argument that Clojure syntax is somehow less natural, it's simply different from syntax people are used to. Most people who doesn't have an existing bias pick it without any problems.
Developers (humans) prefer meaningful variable names, method names, expressions, etc.
Well, isn't that part of the premise we're debating?
Mathematical papers and books tend to have the advantages of dealing with only a handful of concepts at any given time, and of presenting their mathematical content in relatively small doses, both of which allow concise notation to remain both unambiguous and accessible enough to be useful. Anything other than the smallest programs probably do not enjoy the same advantages, so it makes sense that programmers tend to use longer names for entities that are relevant over a larger area. However, this argument still allows for short names to be used in programs as long as the scope is also small, and it says nothing at all about the relevance of natural language for representing programming language constructs.
We have some experience with programming languages that do try to read very much as natural human language. COBOL is probably the most famous example, and in that language even basic mathematical concepts like comparisons can be written out in words. We don't write much COBOL today, and it's hard to believe that the verbosity isn't a contributing factor in that.
No, you are deviating from the main point: natural expressiveness. So let me back up: If you think "if (= x 1)" is natural to humans, then try to ask someone in the street "is equals this that?" and let me know about their reaction. My point is that the natural way of asking such questions is:
* "Is this equals to that?" and not "is equals this that?"
* "Is this bigger than that?" and not "is bigger this that?"
And this is regardless of the language (be it English or whatever, see https://news.ycombinator.com/item?id=22811168). I'm not going to debate this to death, so feel free to disagree. Again, I have nothing against Clojure, I'm just trying to argue in a constructive way.
> If the gap between the natural language and programming language is big, it is difficult for a human to use that programming language. And for Clojure, this gap is big IMO.
English is not all natural languages. Verb-subject-object is the 3rd most common ordering in human languages[1]. Infix is not the only mathematical notation either[2]. I think we all benefit from mind-expansion from learning new languages.
Did I mention English? That still applies to any language. If the developer's natural language is Russian, the mental effort to translate what the developer wants to express in Russian in his programming language should be minimal.
> my natural language where I would say "if x equals one".
You gave an english quotation as an example of your natural language.
My only point is that there is no objective "natural" ordering, what is natural to each person depends on their starting point. Because this is somewhat arbitrary and provincial, it is good to be exposed to alternatives.
Thank you. This is really the impression I have right now after reading all these replies (and those on twitter as well).. Even though I'm trying to argue in a constructive way from the beginning.
I find s-expressions quite beautiful and simplistic way to express both logic and data as a tree. On the other hand, top-level evaluation rules are often glossed over, and seem to mess up this elegance. I feel like there is a set of braces missing at top level, for s-expression concept to be complete. I don't really know enough about specific rules or reasoning why they need to be introduced. My mental model of top-level execution is an implicit `(do ...)` block. In rest of clojure code the `do` is a special form that is frowned upon because it is used only when side-effects are necessary.
As it is, the 'module' concept is strictly tied to a file, which feels unnecessary. If top-level was enclosed in braces and evaluated with same rules as the rest of s-expressions, maybe we could move away from concept of files and, for example, store our code in a graph database. This would allow for IDEs that can collapse and expand code tree as needed.
Rich Hickey uses expression 'place oriented programming' as derogatory term (PLOP) for relying on exact memory addresses as places for our data. It would seem to me that same term would apply to our coding practices - we tie code implementation to specific locations inside files on disk. This seems like accidental complexity that introduces mental burden. If location of implementation could be divorced from specific location inside specific source file, we could concentrate on implementation that is always in context where code is actually used.
Is there any decent discussion of such concepts, or some other language that explores such direction? I'm lacking both terminology and in-depth knowledge of code evaluation to express this in more clear way.