Most people wouldn’t regard Emacs Lisp (from now on “Elisp”) as a “proper” programming language. As to what constitutes a “proper” language, one could certainly have a longer conversation, but my feeling is that those who are conscious of the language dismiss it due to the atypically strong tie to Emacs the editor. Despite attempts to the contrary, the two have never been fully uncoupled – and I don’t mind that¹.

What I want to highlight here are two macros relating to pattern matching. Inherently, pattern matching is a “higher level” construct, leaning in towards a declarative style that even a language like Lisp lacks support for in its core. And while it would not be necessary to use macros to implement this, it certainly makes it feel more organic².

Symbolic Regular Expressions

Regular Expressions have a peculiar reputation among programmers. It is at least my perception that they are venerated by some, disdained by others. The latter is perhaps epitomised by statements such as the following by Jamie Zawinski:

Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems.

I happen to disagree, though to be fair some that know me might claim that I am a kind of regular expression enthusiast. If you ask me, one of the reasons for this reputation is the fact that in conventional programming languages, especially C, regular expressions are encoded in form of strings, which both restrict the syntax and introduce additional complexity when interacting with the remaining language.

Consider constructing a regular expression that depends on the contents of a variable in scope. A simple string concatenation risks the danger of injecting unintentional meta-characters that modify the meaning of the resulting expression.

To take another example, regular expressions usually lack the ability to define custom constructs, as one would expect to be able to define a function in other languages. The consequence is the complexity cannot be easily abstracted, and keeping an overview becomes more difficult than necessary.

While Emacs also uses strings for regular expressions, even differing in choice of meta-characters from BRE, ERE or PCRE³. But I want to claim that most of the time, this doesn’t matter much as one can use the rx macro, that “compiles” a s-expression into a Elisp-compatible regular expression. Here are a few examples of macro-expansion:

(rx ".c" eos)
;;=> "\\.c\\'"

(rx "{{" (group (*? nonl)) "}}") 
;;=>"{{\\(.*?\\)}}" 

(rx (any ?a ?b ?c)
    (= 3 (literal foo))
    (or alpha digit))
;;=> (concat "[a-c]\\(?:" (regexp-quote foo) "\\)\\{3\\}[[:digit:][:alpha:]]")

(rx-let ((name (: upper (or (* lower) "."))))
  (rx name (1+ space) name))
;;=> "[[:upper:]]\\(?:[[:lower:]]*\\|\\.\\)[[:space:]]+[[:upper:]]\\(?:[[:lower:]]*\\|\\.\\)"

Note that, since the body of the macro just consists of s-expressions, we are flexible in formatting. We can add comments or break lines to improve legibility. The last two examples also demonstrate how

1. … the (literal foo) expression can safely and easily inject the contents of a runtime variable foo, causing rx not to compile directly to a string but a function call that will evaluate to a string at runtime.
2. … as string literals always represent literal matches and symbols the meta-language, we can expand the meta-language with constructs such as rx-let or rx-define (analogously to how we can bind the value of a variable using let or defvar).

While we could imagine something like this being implemented inside a language like Java or C, it is arguably not as reliable/convenient.

Structural Pattern Matching

The other macro I want to highlight is pcase. At first, it is a specialised version of cond or a more powerful variation on Lisp’s case⁴. Specialised, in the sense that we use pattern-matching to abbreviate a series of equality matches, that might include destructing complex forms and binding values, and powerful in exactly the same sense.

Stefan Monnier developed the pcase macro independently of Racket’s match, though to my understanding was later inspired by it. What these two have in common, that pattern matching in static languages in the ML lack, is that the pattern matching is extensible, which we will return back to later on.

Here is an example of how pcase looks like from the Elisp manual:

(pcase (get-return-code x)
  ;; string
  ((and (pred stringp) msg)
   (message "%s" msg))
  ;; symbol
  ('success       (message "Done!"))
  ('would-block   (message "Sorry, can't do it now"))
  ('read-only     (message "The schmilblick is read-only"))
  ('access-denied (message "You do not have the needed rights"))
  ;; default
  (code           (message "Unknown return code %S" code)))

As you can see, we can match against specific literals such as symbol, would-block, …, bind the value as code in the final branch that serves as a fallback and construct complex conditions that apply the considered value against a predicate. We can take a look at the macro-expanded code to see that the resulting code is more or less exactly what we would have traditionally written out with cond (at least after byte-compilation):

(let* ((val (get-return-code x)))
  (cond ((stringp val) (let ((msg val)) (message "%s" msg)))
        ((eq val 'success) (let nil (message "Done!")))
        ((eq val 'would-block)
         (let nil (message "Sorry, can't do it now")))
        ((eq val 'read-only)
         (let nil (message "The schmilblick is read-only")))
        ((eq val 'access-denied)
         (let nil (message "You do not have the needed rights")))
        (t (let ((code val)) (message "Unknown return code %S" code)))))

Other fancy features include the ability to destruct S-expressions using quasi-quoting. Here is an example from the ELPA build server:

(pcase-exhaustive spec
  (`(,name :url ,url . ,rest)
   ;; ... modify rest ...
   `(,name :url ,url ,@rest))
  (`(,_ :core ,_ . ,_) nil))

The unquoted parts (,) turn into bindings, while everything else is matched syntactically. So if spec would be bound to something like (foo :url "https://..."), then inside the branch, name would resolve to foo, url to "https://..." and rest to nil. This might be unexpected to some, but is easy to follow with a basic understanding of lists in Lisp. Of course, all of these bindings can again include arbitrary pcase conditions.

As mentioned above, the real neat thing about all this is that this is all malleable and extensible, as the following example demonstrates.

Combining the Above

A pcase condition that I haven’t discussed yet, mainly as it was something I have overlooked for a while myself, is the rx form! The example I want to demonstrate is from rcirc and is an excerpt from the rcirc-authenticate function that consults a variable to check if the client should send some user-configured authentication data to the server when initialising a connection. The examples we will be considering deal with the “NickServ” authentication mechanism, where the password can either be stored in Emacs’ auth-source library or directly in the authentication configuration.

(pcase-dolist (`(,(rx (regexp rcirc-server)) . ,ai) rcirc-authinfo)
  (pcase ai
    (`(nickserv ,(rx (regexp rcirc-nick)) :auth-source . ,(or `(,nickserv) '()))
     (if-let* ((auth (auth-source-search
                      :host rcirc-server
                      :port (nth 1 rcirc-connection-info)
                      :user (nth 2 rcirc-connection-info)))
               (password (auth-info-password (car auth))))
         (rcirc-send-privmsg
          (rcirc-buffer-process)
          (or nickserv "NickServ")
          (concat "IDENTIFY " password))
       (rcirc-print
        (rcirc-buffer-process) rcirc-nick "ERROR" nil
        "No auth-source entry found for `nickserv' authentication")))
    (`(nickserv ,(rx (regexp rcirc-nick)) ,password . ,(or `(,nickserv) '()))
     (rcirc-send-privmsg
      (rcirc-buffer-process)
      (or nickserv "NickServ")
      (concat "IDENTIFY " password)))
    ;; ...
    ))

To get it out of the way, pcase-dolist is one of a number of pcase-enhanced macros that extend the usual binding with pcase-like pattern matching. Others include pcase-let, pcase-lambda and pcase-exhaustive. To take a simpler example, we can see here how easy it becomes to destructure an expression and get the data we are actually interested in without intermediate binders; but also how it can go wrong in subtle ways

(pcase-let ((`(,head . ,_) '("car" . "cdr")))
  head)
;;=> "car"

;; no error when pattern fails to match, see bug#71503:
(pcase-let ((`(,head . ,_) '["car" "cdr"])) 
  head)
;;=> nil

But that is not the interesting part I wanted to focus on; that is the following condition of the first pcase branch:

`(nickserv ,(rx (regexp rcirc-nick)) :auth-source . ,(or `(,nickserv) '()))

This combines a number of checks and bindings on the value of ai in one line:

1. We are dealing with a list that has at least three elements.

2. The first value of that list is unconditionally bound to the variable nickserv.

3. New and interesting: The second value will only satisfy the branch, if the second value of the list is a string that matches the regular expression produced by the rx expression. The rx expression itself is just a trivial injection, as (regexp rcirc-nick) expands to an expression that will evaluate rcirc-nick at runtime, without quoting it (compare this to the literal example above).

   Note that we couldn’t just use rcirc-nick directly here, because in that case we would be comparing the second value directly with rcirc-nick for equality, which is a much finer condition.

4. The third value must be eq to the symbol :auth-source.

5. The list may or may not include a fourth value. The (or ...) condition allows us to describe two different branches inside the same condition: Either the remaining list is a singleton value, where we will bind nickserv to the only value of that list or it is an empty list (in which case nickserv would remain nil).

One can imagine that a more imperative approach would be much more verbose and deal with various details (such as the different acceptable lengths of the list) in ways that might not make the intention clear to the reader. One has to be fair and admit that this condition also only makes sense if one is familiar with the language of pcase and rx, but I contend that by re-using parts of the host-language in natural ways (such as or or .), it is an enriching experience.

The second branch is similar to the first, with the exception that the third value must not be exactly :auth-source, but some other value. We know that is must be some other value, as the first branch did not match and pcase processes the branches in order. One could argue that these two branches could actually be merged together:

(pcase-dolist (`(,(rx (regexp rcirc-server)) . ,ai) rcirc-authinfo)
  (pcase ai
    (`(nickserv ,(rx (regexp rcirc-nick)) ,password . ,(or `(,nickserv) '())) ;untested!
     (let ((password ((if-let* (((eq password :auth-source))
                (auth (auth-source-search
                       :host rcirc-server
                       :port (nth 1 rcirc-connection-info)
                       :user (nth 2 rcirc-connection-info))))
              (auth-info-password (car auth))
            password))))
       (unless (stringp password)
     (error "Invalid NickServ password: %S" password))
       (rcirc-send-privmsg
    (rcirc-buffer-process)
    (or nickserv "NickServ")
    (concat "IDENTIFY " password))))
    ;; ...
    ))

… but perhaps this is too verbose?

Returning back to the general idea, the rx condition allows us to match strings against patterns. Considering that one frequent use-case of regular expressions is also to destruct a string with a simple encoding of some data, it is not surprising the pcase extends rx conditions “let”-expressions that allow us to symbolically bind a submatch to a variable inside a branch. Take this example, that parses a simple command language and naively translates it into symbol manipulations inside the Lisp environment:

(pcase command ;<- some string
  ((rx bol "set"
       (+ space) (let var (+ (not space)))
       (+ space) (let val (+ alnum)))
   (set (intern var) val))
  ((rx bol "get" (+ space) (let var (+ (not space))))
   (symbol-value (intern var)))
  ((rx bol "check" (+ space) (let var (+ (not space))))
   (boundp (intern var))))

The (let var (+ (not space))) introduces the var binding, and binds it to the first subexpression match — in this case a substring consisting of no spaces. In most languages, this would involve dealing with subexpression numbers, occasionally named subexpressions that I would have to declare within a match group inside the regular expression, and then reference later on. I am not aware of any type system or linter that checks for mistakes that could arise form this weak coupling (e.g. a typo between the group name and the reference later on). SNOBOL might be an exception, but conditional value assignment is not exactly the same.

What fascinates me the most about all this is that these languages were created independently, and could be merged in an elegant way due to their respective potential for extensibility⁵: The definitions for rx support in pcase is in fact implemented in rx.el — just M-x find-library rx RET and search for pcase-defmacro to see how it was done.

One criticism, which is due to the nature of macros, is that the step from what we have to being able to use pcase to search inside of buffers, meaning that the rx expression would map to a re-search-forward is non-trivial:

(pcase (current-buffer) ;<- some buffer
  ((rx bol "set"
       (+ space) (let var (+ (not space)))
       (+ space) (let val (+ alnum)))
   (set (intern var) val))
  ((rx bol "get" (+ space) (let var (+ (not space))))
   (symbol-value (intern var)))
  ((rx bol "check" (+ space) (let var (+ (not space))))
   (boundp (intern var))))

The reason here is that we cannot know the type of the matched expression at macro-expand time. So unless we can re-use the static type information from Emacs 30, the macro-expanded code would have to generate both the string-matching and the buffer-searching code. The choice which to use, would probably have to occur at runtime. Here is a diff against Emacs’ master branch that one could take as the basis for this feature:

diff --git a/lisp/emacs-lisp/rx.el b/lisp/emacs-lisp/rx.el
index c512d42cd15..69e625d971d 100644
--- a/lisp/emacs-lisp/rx.el
+++ b/lisp/emacs-lisp/rx.el
@@ -1693,7 +1693,7 @@ rx
 (defun rx--pcase-expand (regexps)
   (let* ((rx--pcase-vars nil)
          (regexp (rx--to-expr (rx--pcase-transform (cons 'seq regexps)))))
-    `(and (pred stringp)
+    `(and (or (pred stringp) (pred bufferp))
           ,(pcase (length rx--pcase-vars)
             (0
              ;; No variables bound: a single predicate suffices.
@@ -1703,9 +1703,16 @@ rx--pcase-expand
              ;; is the submatch value, 0 on failure.  We can't use nil
              ;; for failure because it is a valid submatch value.
              `(app (lambda (s)
-                     (if (string-match ,regexp s)
-                         (match-string 1 s)
-                       0))
+                     (cond
+                      ((bufferp s)
+                       (if (re-search-forward ,regexp nil t)
+                           (match-string 1)
+                         0))
+                      ((stringp s)
+                       (if (string-match ,regexp s)
+                           (match-string 1 s)
+                         0))
+                      ((error "Cannot match %S" s))))
                    (and ,(car rx--pcase-vars) (pred (not numberp)))))
             (nvars
              ;; Pack the submatches into a dotted list which is then
@@ -1714,11 +1721,20 @@ rx--pcase-expand
              ;; A dotted list is used to reduce the number of conses
              ;; to create and take apart.
              `(app (lambda (s)
-                     (and (string-match ,regexp s)
-                          ,(rx--reduce-right
-                            (lambda (a b) `(cons ,a ,b))
-                            (mapcar (lambda (i) `(match-string ,i s))
-                                    (number-sequence 1 nvars)))))
+                     (cond
+                      ((bufferp s)
+                       (and (re-search-forward ,regexp nil t)
+                            ,(rx--reduce-right
+                              (lambda (a b) `(cons ,a ,b))
+                              (mapcar (lambda (i) `(match-string ,i))
+                                      (number-sequence 1 nvars)))))
+                      ((stringp s)
+                       (and (string-match ,regexp s)
+                            ,(rx--reduce-right
+                              (lambda (a b) `(cons ,a ,b))
+                              (mapcar (lambda (i) `(match-string ,i s))
+                                      (number-sequence 1 nvars)))))
+                      ((error "Cannot match %S" s))))
                    ,(list '\`
                           (rx--reduce-right
                            #'cons

Is it ironic that I ended up using cond?