So there's some boilerplate that needs to happen. With macros, you write the macro to insert the boilerplate, and then you never think about it again. You don't write it, you don't read it, it's not in the way. Without macros, you have to write the boilerplate every time you write the code. You have to read the boilerplate every time you read the code.

Here's the version with macros:

    (transaction (do-stuff)
                 (do-stuff-if-commit-successful)
                 (do-stuff-if-rollback))

Here's the version without macros:

    (transaction (lambda () (do-stuff))
                 (lambda () (do-stuff-if-commit-successful))
                 (lambda () (do-stuff-if-rollback)))

Which do you find more readable? Notice how there's no boilerplate in the macro version.

> This appears to be the same as your other example. Java has lambda expressions (since Java 8) that could do this.

What are the odds that Java 8 has provided everything you could want in out of Java? Macros let you add things in a better way than you could otherwise get. Look back to my prior example of Java 5's expanded for. You see how useful that was? If Java had had macros, people wouldn't have had to suffer through nine years without the improved for loop.

@ zck:

> What are the odds that Java 8 has provided everything you could want in out of Java? Macros let you add things in a better way than you could otherwise get. Look back to my prior example of Java 5's expanded for.

Again, please understand we non-Lispers find this argument utterly unconvincing :) Java in particular is a terrible example: it's a language that for many years remained in the dark ages, and now it's finally getting modern features retrofitted into it, while at the same time attempting to keep some sort of backwards compatibility, and the whole process is very painful.

Let's all agree to stop talking about Java. Let's assume we all agree working in a language that until very recently didn't have lambdas is painful. And that requires a horrific amount of boilerplate.

Let me re-throw your question back at you: what exactly can Lisp do, which has practical implications, that a modern language with lambdas, closures and lazy evaluation cannot accomplish in elegant ways? If you mention Java again, you lose :P

You seem to be concerned by the use of "Java". It's a fine example, and one I think I've explained why very well. It also seems petulant to declare Java off-limits.

But let me try again. For _any given language_, there are things you may want that the language doesn't provide. Macros let you do that in an elegant way. The example I gave above -- based off LanceH's -- of transactions is a way where the macro-based solution is more elegant than non-macro solutions.

Here's another example. Arc, like any language, has a built-in way of setting variables. It's called `assign`, and can be used as follows:

    arc> (assign a 3)
    3
    arc> a
    3

But no one uses it. Instead, people use =. = is a macro that's provided with the language. Because it's a macro, that means that if it wasn't provided, you could write it yourself.^1

What's the benefit of = ? It lets you set values of more than just variables:

    arc> (= my-table (table))
    #hash()
    arc> (my-table 'key) ;;look up the value
    nil
    arc> (= (my-table 'key) 'value)
    value
    arc> (my-table 'key)
    value

Note that in the second prompt, we're attempting to look up the value of 'key in the hashtable my-table, and we see that there isn't one there. We then set it in the third prompt, and look it up again in the final.

This works on the concept of "places". The "place" we try to set in `(= (my-table 'key) 'value)` is the association of 'key inside my-table.

Why is this beneficial? If you know how to get a value out of a data structure, you can now set it. This code is extremely clean and understandable compared to a non-macro version. It exhibits the principle of least surprise, and it's obvious how to set other data structures in an elegant way.

You can't do this without macros.

[1] If your objection here is "but it comes with the language", you're missing the point.

I disagree your example of Java is fine. It's not petulant to declare it off-limits, just as it's not petulant to declare COBOL off-limits. We are discussing features and extensibility of finer languages than Java.

> The example I gave above -- based off LanceH's -- of transactions is a way where the macro-based solution is more elegant than non-macro solutions.

Here we disagree. In your example of transactions, you didn't shown that macro-based solutions are more elegant than non-macro based solutions; you merely showed that Java before version 8 wasn't very good (and when someone replied "but we can do better with Java 8!" you basically replied "ok, but how do you know Java 8 is enough for some other unspecified problem?"). Your answer is unconvincing, especially since non-macro-based solutions to your example in other languages, such as Scala, are equally elegant to Lisp's, because Scala has support for first-class functions and closures. Let me preempt a "but how do you know that's enough for Scala?"... I don't know. Show me why it's not enough!

Re: your second example with assign and the = macro. I admit I don't understand it yet; I'll have to think some more about it.

edit: let me go back to this assertion:

> For _any given language_, there are things you may want that the language doesn't provide. Macros let you do that in an elegant way.

I find this problematic, for two reasons. First, we've acknowledged macros cannot solve everything; after all, there are new releases and multiple implementations of Lisp languages. What someone else said: "if it's not in the 'substrate', macros can't do it". Second, that macros let you do some (admittedly cool!) things doesn't automatically show that these same things cannot be accomplished in reasonably elegant ways in other languages. One thing doesn't imply the other!

>In your example of transactions, you didn't shown that macro-based solutions are more elegant than non-macro based solutions; you merely showed that Java before version 8 wasn't very good...

I think it wasn't clear what I was referring to. I was talking about not Java, but a Lisp-style solution. Here it is with macros:

    (transaction (do-stuff)
                 (do-stuff-if-commit-successful)
                 (do-stuff-if-rollback))

Here's the version without macros:

    (transaction (lambda () (do-stuff))
                 (lambda () (do-stuff-if-commit-successful))
                 (lambda () (do-stuff-if-rollback)))

The macro-based solution -- which doesn't mention Java -- is more elegant. You don't need the lambdas if you use macros (why? Well, you don't always want to rollback, right?)

>...when someone replied "but we can do better with Java 8!" you basically replied "ok, but how do you know Java 8 is enough for some other unspecified problem?"

That's the point -- a language with macros is extensible in a way that languages without macros aren't. So unless you believe that your language happens to be perfect, having macros would make the language more powerful.

>Re: your second example with assign and the = macro. I admit I don't understand it yet; I'll have to think some more about it.

Feel free to contact me if there's anything else I can explain -- I'm probably going to forget to check this thread soon.

>...we've acknowledged macros cannot solve everything; after all, there are new releases and multiple implementations of Lisp languages.

I'm not sure why new Lisp releases show that macros are not useful. You could write anything in assembly, but other languages are still released. And design is important -- which sets of functions, and macros should be provided with a language? Does having a release of Scala that includes functions mean that there's no need for user-defined functions?

>Second, that macros let you do some (admittedly cool!) things doesn't automatically show that these same things cannot be accomplished in reasonably elegant ways in other languages. One thing doesn't imply the other!

It doesn't mean that, no. However, I don't see elegant ways to do this kind of thing in other ways. If you can show me some, I'd be interested.

The problem that I have with lisp macros is that the elegance you gain at the syntax level is effectively a tradeoff with pragmatism when other people read and use the code. Java was developed with parts of C++/C as inspiration and parts of that language were left out. Particularly, operator overloading was left out (which can be viewed as a very restricted example of modifying the language), presumably because it's not an immediate thought when viewing the code that the operator isn't doing what you expect. While it's undeniable that macros make the syntax nice to look at, the same argument that lisp becomes a new language as you write your program means that every separate codebase has a lot more reading to understand because you have to go through all the macros. New releases in most languages introduce new features (and standardize functions, fix bugs, etc), as far as I can see, new releases in lisp enforce a standard (common) base set to decrease the amount of work required in learning new codebases.

Yes, macros can make code very hard to read, if designed badly. Of course, so can functions, variable names, and program flow.

But Lisp with macros is very different from C++ with operator overloading. With C++ operator overloading, you only know if a given line has something you don't understand (that is, an overloaded operator) by looking at every other file in your project. With Lisp macros, you know that you're dealing with something new because you don't recognize the first token in the s-expression. You might not know it's a _macro_ rather than just a _function_, but you know it's something you need to investigate.

Basically, in Rumsfeld's terminology, an overloaded operator is an unknown unknown, but a Lisp macro is a known unknown. A macro's behavior may be confusing, but its existence isn't. And that's a very big difference.

Agreed, I'm still unconvinced for the same reasons. In Scala or Haskell (or any language which supports first-class functions and lazy evaluation, I guess) the "transaction" example is easily done.